Home automation is one of the key focuses of my project, as it's one of the areas where privacy and security matter the most but also an area where many consumer devices skimp on. It's also unusual to see Kubernetes handling this; most k8s demos tend to be a bit more focused on webservers or enterprise software.

For my lab I'll be using a Z-Wave USB dongle to communicate with Z-Wave smart devices. The stick is plugged into a Raspberry Pi, which will talk via MQTT to Home Assistant running on Kubernetes, an application which provides an easy to use web dashboard. The setup is summed up by this diagram:

Setting up the Raspberry pi

1. Set up the Raspberry Pi with an OS of your choice. In my case I'm using Ubuntu Server for Raspberry Pi. Set the server to use a static IP, and optionally you can also set up a DNS resolver entry for the Pi in your router.

2. Install Docker on the Pi. You can do so easily with the following script (Ubuntu specific, for other distros see the docker website):

# Remove old docker if it exists
sudo apt-get remove docker docker-engine docker.io containerd runc
# Update dependencies
sudo apt update
sudo apt install -y apt-transport-https ca-certificates curl gnupg software-properties-common
# Docker repo key
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
# Install Docker latest runtime
sudo apt update
sudo apt install -y docker-ce docker-ce-cli docker-compose-plugin
# Add current user to docker group (so you can use docker without sudo)
sudo groupadd docker
sudo usermod -aG docker $USER

3. Plug in your zwave usb dongle.

4. Find the id of the serial device you just plugged in. You can list all serial devices using ls /dev/serial/by-id/.

5. Create a docker-compose file for zwavejs2mqtt:

cat <<EOF > docker-compose.yml
version: '3.7'
services:
  zwavejs2mqtt:
    container_name: zwavejs2mqtt
    image: zwavejs/zwavejs2mqtt:latest
    restart: always
    tty: true
    stop_signal: SIGINT
    environment:
      - SESSION_SECRET=mysupersecretkey
      - ZWAVEJS_EXTERNAL_CONFIG=/usr/src/app/store/.config-db
      # Uncomment if you want logs time and dates to match your timezone instead of UTC
      # Available at https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
      # - TZ=America/Los_Angeles
    networks:
      - zwave
    devices:
      # Do not use /dev/ttyUSBX serial devices, as those mappings can change over time.
      # Instead, use the /dev/serial/by-id/X serial device for your Z-Wave stick.
      - '/dev/serial/by-id/insert_stick_reference_here:/dev/zwave'
    volumes:
      - zwave-config:/usr/src/app/store
    ports:
      - '8091:8091' # port for web interface
      - '3000:3000' # port for Z-Wave JS websocket server
      - '1883:1883' # mqtt
networks:
  zwave:
volumes:
  zwave-config:
    name: zwave-config
EOF

6. Start the container:

docker compose up

Deploying home assistant

Over on the jumpbox machine, add the k8s-at-home helm repository (if you haven't done so already).

helm repo add k8s-at-home https://k8s-at-home.com/charts/

Create a values file:

cat <<EOF > home-assistant-values.yaml
env:
  TZ: America/Los Angeles
service:
  main:
    type: LoadBalancer
persistence:
  config:
    enabled: true
EOF

Then install:

helm install home-assistant k8s-at-home/home-assistant -f home-assistant-values.
yaml

Installing Mosquitto MQTT

Create a values file:

cat <<EOF > mqtt-values.yaml
env:
  TZ: America/Los Angeles
service:
  main:
    type: LoadBalancer

Install the helm chart.

helm install mosquitto k8s-at-home/mosquitto

Now get the external IP address of the LoadBalancer service:

kubectl get svc mosquitto
NAME        TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)          AGE
mosquitto   LoadBalancer   100.69.203.75   192.168.30.213   1883:30582/TCP   62d

The mqtt broker will be running at externalIP:1883.

Setting up zwavejstomqtt

Open the web ui at <raspberrypi ip>:8091. Configure the following:

1. Set serial port to /dev/zwave under Settings → Zwave → Serial Port (this is where docker mounts the USB stick).
2. Create network security keys under Settings → Zwave → Security Keys (S0 Legacy, S2 Unauthenticated, S2 Authenticated, and S2 Access Control). You can use the generate button (two arrows in each text box) to generate them for you.
3. Under Settings → MQTT, add the MQTT information from earlier. For Host url use the LoadBalancer external IP, and for port use 1883.
4. Under Settings → Home Assistant, enable MQTT Discovery.

Adding a device to zwavejstomqtt

Under the "Smart Start" tab, click on the + icon in the bottom right. If your Zwave device has a QR code on the side you can use the add using QR code option. Click on the option, grant the website camera access, and scan the QR code with your computer's webcam (or scan the info with your phones webcam and type it in manually). If your device does not have a QR code you can also add it manually using the DSK.

On the Control Panel page you should see a second device (with the first being the USB stick itself). If you click the device you should be able to control it, as well as query its data if it presents any. For example, for the smart outlet I have configured I can look at the historical power usage.

Adding MQTT to Home Assistant

Under Settings → Devices and Services → Add Integration (bottom right button), search for MQTT. Add the integration. Configure with the IP and port of the MQTT broker. Finally, go back to the dashboard. The added device should be visible.

Conclusion

Bit of a long article, but we configured a number services across different machines. Hopefully this helps you on your Kubernetes edge journey!

Observability is a function that allows developers and operators to identify problems where and when they happen across multi-node systems. Proper instrumentation enables you to aggregate metrics, traces, logs and events from a distributed system and correlate them across various application components and services, identifying complex interactions between elements and allowing you to troubleshoot performance issues, improve management, and optimize cloud native infrastructure and applications.

This article covers running a logging solution hosted on, and supporting Kubernetes.

ELK is an acronym that describes a popular “stack” of open source components used to implement an n-tier logging solution with search, analytics and user interface. E=elasticsearch, L=Logstash, and K=Kibana. With Kubernetes it is often popular to swap out the L=Logstash component for f=fluentbit (aka EfK stack).

This substitution is popular because fluentbit is comparatively very light on resource demands (important because log collection elements run on every Kubernetes cluster node) and because it has a set of configurable input plugins that support gathering logs from Kubernetes itself plus the containers Kubernetes is hosting.

Install ElasticSearch and Kibana using a helm chart

The best practice is to use seven pods in the Elasticsearch cluster:

• Three master pods for managing the cluster.
• Two data pods for storing data and processing queries.
• Two client (or coordinating) pods for directing traffic. The “official” helm chart from elastic involves a multi-step process to do a horizontally scaled configuration like this so we are using the helm chart from bitnami instead.

Add the Bitnami helm repo if you don't already have it from a previous service installation:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update

This example assumes you have a storage class pre-defined, named silver, and have a service load balancer in place. Adjust the steps as needed.

# create namespace
kubectl create ns kube-logging

# install
helm install elasticsearch bitnami/elasticsearch -n kube-logging --set global.storageClass=silver,global.kibanaEnabled=true,service.type=LoadBalancer,kibana.service.type=LoadBalancer,master.replicas=3,coordinating.service.type=LoadBalancer

Install Fluentbit using a Carvel package

To feed logs from Kubernetes to the Elasticsearch, Kibana combo - fluentbit will be used. This will cover an install using the Carvel based package option built into Tanzu Community Edition. If you are on a Kubernetes distribution that does not support Carvel, then a fluent bit install using helm with similar settings should work.

First a yaml file is composed to set inputs, outputs, parsers and filters appropriate for Kubernetes log collection.

cat << EOF > fluent-bit-data-values.yaml
namespace: "kube-logging"
fluent_bit:
  config:
    service: |
      [Service]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020
    outputs: |
     [OUTPUT]
       Name            es
       Match           *
       Host            elasticsearch
       Port            9200
       Logstash_Format On
       Replace_Dots    On
       Retry_Limit     False
       Buffer_Size     64KB
       Suppress_Type_Name On

    inputs: |
      [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            cri
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

      [INPUT]
        Name                systemd
        Tag                 kube_systemd.*
        Path                /var/log/journal
        DB                  /var/log/flb_kube_systemd.db
        Systemd_Filter      _SYSTEMD_UNIT=kubelet.service
        Systemd_Filter      _SYSTEMD_UNIT=containerd.service
        Read_From_Tail      On
        Strip_Underscores   On

    filters: |
      [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On

    parsers: |
      # see https://github.com/fluent/fluent-bit/blob/v1.7.5/conf/parsers.conf
      [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

      [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

      [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

      [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

      [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

      [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

      [PARSER]
        Name        docker-daemon
        Format      regex
        Regex       time="(?<time>[^ ]*)" level=(?<level>[^ ]*) msg="(?<msg>[^ ].*)"
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

      [PARSER]
        # http://rubular.com/r/tjUt3Awgg4
        Name cri
        Format regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z

      [PARSER]
        Name        logfmt
        Format      logfmt

      [PARSER]
        Name        syslog-rfc5424
        Format      regex
        Regex       ^\<(?<pri>[0-9]{1,5})\>1 (?<time>[^ ]+) (?<host>[^ ]+) (?<ident>[^ ]+) (?<pid>[-0-9]+) (?<msgid>[^ ]+) (?<extradata>(\[(.*)\]|-)) (?<message>.+)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

      [PARSER]
        Name        syslog-rfc3164-local
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S
        Time_Keep   On

      [PARSER]
        Name        syslog-rfc3164
        Format      regex
        Regex       /^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/
        Time_Key    time
        Time_Format %b %d %H:%M:%S
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

      [PARSER]
        Name    kube-custom
        Format  regex
        Regex   (?<tag>[^.]+)?\.?(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$

    streams: ""
    plugins: ""

  daemonset:
    resources: { }
    podAnnotations: { }
    podLabels: { }
EOF

tanzu package install fluent-bit --package-name fluent-bit.community.tanzu.vmware.com --version 1.7.5 --namespace kube-logging --values-file fluent-bit-data-values.yaml

Examine logs using the Kibana UI

Wait for all pods in the kube-logging namespace to start.

1. Use kubectl get svc -n kube-logging to get the external load balanced ip of the kibana ui.
2. Open http://<kibana ip>:5601  in a browser
3. Dismiss the integration popup, click on the menu on the left, then discover
4. Create a data view
5. Set index pattern to  logstash-* Note that the trailing asterisk will be added automatically
6. Choose @timestamp from dropdown.
7. Choose discover and select “last 15 minutes”. You should see log entries. Refer to the ElasticSearch/Kibana online docs or videos for more details on how this can be used

Harbor is a container registry, i.e. a system which manages container images which can then be used by Kubernetes. It is a self-hosted alternative to a cloud based service like Docker Hub.

Why do we want to use Harbor (vs a cloud registry like Docker Hub)?

A few concrete reasons stick out:

• If your cluster restarts (say due to a power outage or hardware failure) then on reboot Kubernetes may try to pull many images at once from the internet, which could clog up your internet connection. Your download speed from a local hard disk is probably faster then your internet connection, especially at home.
• Cloud registries are known to be vectors for malware. Docker Hub integrates with scanning tools to allow you to first pull images, scan them for malware and vulnerabilities, and then choose whether or not to deploy.
• As with any cloud based service, it is subject to change. While Docker Hub is free, it is rate limited, and it could be not free tomorrow.

Installation

Before we can even think about installing Harbor there's a small sized list of dependencies we need to go through (an American small size, that is).

Command line tools

Install imgpkg:

wget https://carvel.dev/install.sh -O install.sh

# Inspect install.sh before running...
sudo bash install.sh
imgpkg version

Install yq:

wget https://github.com/mikefarah/yq/releases/download/${VERSION}/yq_linux_amd64 -O yq && chmod +x yq && sudo mv yq /usr/bin/yq

Kubernetes Packages

Install cert-manager:

# Get package versions
tanzu package available list cert-manager.community.tanzu.vmware.com
# Install
tanzu package install cert-manager \
   --package-name cert-manager.community.tanzu.vmware.com \
   --version ${CERT_MANAGER_PACKAGE_VERSION}

Install Contour (note: to install Contour your cluster must have a service LoadBalancer. See post 2 for more info:

# Get package versions
tanzu package available list contour.community.tanzu.vmware.com
# Install
tanzu package install contour \
   --package-name contour.community.tanzu.vmware.com \
   --version ${CONTOUR_PACKAGE_VERSION}

Harbor

Creating Certificates for Harbor

In order to issue certificates cert-manager is going to use a DNS challenge. This challenge creates temporary DNS records to prove your ownership of your domain to Let's Encrypt. Since this challenge runs each time the certificate renews (every 60 days), cert-manager requires an automatic way to create DNS records for our domain, e.g. by providing cert-manager with an API key or other credentials by which it can access the DNS provider. This will vary a bit depending on the DNS provider for your domain. I'll be covering Cloudflare, see the cert-manager docs for other providers.

On the Cloudflare web dashboard navigate to User Profile → API Tokens → API Tokens. Create an API token with the following settings:

• Permissions: Zone - DNS - Edit, Zone - Zone - Read
• Resources: Include - All Zones

Next create a Kubernetes secret to store your API token:

cat <<EOF > cloudflare-api-dns-token-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-dns-token-secret
  namespace: cert-manager
type: Opaque
stringData:
  api-token: YOUR_API_TOKEN
EOF

kubectl apply -f cloudflare-api-dns-token-secret.yaml

Create a ClusterIssuer resource. This will issue certificates.

cat <<EOF > letsencrypt-prod.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
  namespace: cert-manager
spec:
  acme:
    email: youremail@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: cloudflare-api-dns-token-secret
            key: api-token
EOF

kubectl apply -f letsencrypt-prod.yaml

Finally create a Certificate resource for Harbor to use:

cat <<EOF > harbor-cert.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: harbor-external-cert
  namespace: harbor
  labels:
    app: harbor
spec:
  dnsNames:
  - harbor.private.tcehomelab.com
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: letsencrypt-prod
  secretName: harbor-external-cert-tls
  usages:
  - digital signature
  - key encipherment

If everything went well kubectl get certificates -n harbor should show the following:

kubectl get certificates -n harbor
NAME                   READY   SECRET                           
harbor-external-cert   True    harbor.private.tcehomelab.com-tls   7m

It may take a few minutes to become ready. You can run kubectl describe certificate -n harbor harbor-external-cert to see Certificate details. If you have waited a few minutes and the certificate is still not ready, follow the troubleshooting steps listed here.

Installing Harbor

We are finally ready to install harbor.

Download the values file and password-generation scripts.

# harbor-values.yaml
wget https://raw.githubusercontent.com/vmware-tanzu/community-edition/main/addons/packages/harbor/2.4.2/bundle/config/values.yaml -O harbor-values.yaml
# generate-passwords.sh
wget https://raw.githubusercontent.com/vmware-tanzu/community-edition/main/addons/packages/harbor/2.4.2/bundle/config/scripts/generate-passwords.sh -O generate-passwords.sh

Run the password generation script on the values file. This will modify the file to include the passwords and keys that Harbor needs.

bash generate-passwords.sh harbor-values.yaml

Open harbor-values.yaml in an editor of your choice. Change the hostname to the FQDN where you plan to host Harbor and should match the DNS name of the Certificate created earlier (in my case harbor.private.tcehomelab.com). Set the  tlsCertificateSecretName to harbor-external-cert-tls. Save the file.

Run the following to create a copy of the values file with comments removed (it uses the yq tool we installed earlier):

cp harbor-values.yaml harbor-values-clean.yaml && yq -i eval '... comments=""' harbor-values-clean.yaml

Install harbor:

tanzu package install harbor -p harbor.community.tanzu.vmware.com -v 2.4.2 -f harbor-values-no-comment.yaml

Finally, create a Kubernetes ingress resource:

cat <<EOF > harbor-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: harbor
  labels:
    app: harbor
  namespace: harbor
spec:
  rules:
  - host: harbor.private.tcehomelab.com
    http:
      paths:
      - backend:
          service:
            name: harbor-portal
            port:
              number: 443
        path: "/"
        pathType: ImplementationSpecific
EOF

kubectl apply -f harbor-ingress.yaml

If you run kubectl get ing -n harbor you should see an output similar to the following:

kubectl get ing -n harbor        
NAME     CLASS    HOSTS                           ADDRESS          PORTS   AGE
harbor   <none>   harbor.private.tcehomelab.com   192.168.30.212   80      10m

Add a DNS entry on your local network DNS (e.g. pfSense) that points the harbor host to the IP address of the ingress.

Finally you should be able to go the Harbor webpage at whatever your hostname is. You can login with the username admin and password from harbor-values.yaml (harborAdminPassword). See the usage section of the TCE harbor docs or the official Harbor docs for more info, such as instructions for uploading an image to Harbor.

In the last post we deployed Ghost, a content management system for a blog. In this article we'll be discussing ways to expose our blog to the internet.

The traditional way to expose something running on a home network to the internet would be to punch a hole through the home firewall to allow the outside internet access to the specific server on the home network. However, this comes with a few drawbacks:

• Your home's IP address is exposed to anyone who knows the web address of your blog
• Any allowed access to the internal network from the outside world could be a potential vulnerability. It is safer and simpler to have a standard firewall (which denies all inbound traffic) than one that allows some traffic through.

Further, in order to HTTPS secure the blog we need to obtain trusted certificates from a 3rd party certificate authority.

An alternative method is to use a tunnel service, such as Cloudflare's Argo tunnels. Here, a program that runs on your local network (cloudflare daemon) makes a connection with Cloudflare's servers. When users connect to the site, they are connecting to Cloudflare's servers, which then redirects the connection to the home network. Your home IP is not exposed, no firewall holes are required, and Cloudflare handles issuing HTTPS certificates.

Setting up Ghost

We're going to uninstall ghost.

helm uninstall ghost

This won't delete everything; we need to delete the PersistentVolumeClaim created by Ghost as well.

kubectl get pvc                  
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-ghost-mysql-0   Bound    pvc-eff97035-0ffb-4415-8c64-088239c88e10   8Gi        RWO            default        46h
kubectl delete pvc data-ghost-mysql-0
persistentvolumeclaim "data-ghost-mysql-0" deleted

Next we'll be creating a values file for the Ghost install:

cat <<EOF > ghost-values.yaml
service:
  type: ClusterIP
ghostHost: ghost.acgandhi.com
ghostEmail: youremail@example.com
ghostPassword: mustbemorethan10chars
EOF

And then finally (re)installing ghost:

helm install ghost bitnami/ghost -f ghost-values.yaml

You can run kubectl get po,svc to see which pods and services are created by Ghost. Unlike last time the Ghost service is of type ClusterIP instead of LoadBalancer. If you want to test the installation, you can use kubectl port-forward, and then access the website at jumpboxVMIP:8080.

kubectl port-forward svc/ghost 8080:80 --address 0.0.0.0

Installing Cloudflare Tunnel

If you don't have one already, you will need a domain name. It can be from any registrar, such as Google Domains, Namecheap, GoDaddy, Hover, etc. Follow Cloudflare's instructions to add your site and change your domain name to use Cloudflare as its authoritative DNS.

Next we have to install the cloudflared on our jumpbox. For more information look at Cloudflare's documentation.

# download binary 
wget https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 -O cloudflared
# allow execution
chmod +x cloudflared
# move into path
sudo mv cloudflared /usr/local/bin/

Afterwards you can login, and then create the tunnel resource.

cloudflared tunnel login

cloudflared tunnel create ghostdemo
Tunnel credentials written to /home/tceadmin/.cloudflared/7e87f3da-29da-4a99-a4f0-ca7eafffec9d.json. cloudflared chose this file based on where your origin certificate was found. Keep this file secret. To revoke these credentials, delete the tunnel.

Created tunnel ghostdemo with id 7e87f3da-29da-4a99-a4f0-ca7eafffec9d

Next you'll upload the tunnel credentials created to Kubernetes.

kubectl create secret generic tunnel-credentials \
--from-file=credentials.json=/home/tceadmin/.cloudflared/7e87f3da-29da-4a99-a4f0-ca7eafffec9d.json

Then, we're going to create two DNS records: one for the blog itself, and another for Cloudflared's hello_world test service. You can use the cloudflared CLI or manually add CNAME records targeting <tunnelid>.cfargotunnel.com. I ran into issues with the manual record creation and so I recommend the command line approach (it's easier, too!).

The command syntax is cloudflared tunnel route dns <tunnel> <hostname>. So, in my case I ran the following two commands:

cloudflared tunnel route dns ghostdemo ghostdemo.acgandhi.com
cloudflared tunnel route dns ghostdemo helloworld.acgandhi.com

If we look on the Cloudflare web console we can see that both of these records are created:

Finally we're ready to deploy cloudflared on Kubernetes. Copy the following text into a file named cloudflared.yaml.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudflared
spec:
  selector:
    matchLabels:
      app: cloudflared
  replicas: 2 # You could also consider elastic scaling for this deployment
  template:
    metadata:
      labels:
        app: cloudflared
    spec:
      containers:
      - name: cloudflared
        image: cloudflare/cloudflared:2022.3.0
        args:
        - tunnel
        # Points cloudflared to the config file, which configures what
        # cloudflared will actually do. This file is created by a ConfigMap
        # below.
        - --config
        - /etc/cloudflared/config/config.yaml
        - run
        livenessProbe:
          httpGet:
            # Cloudflared has a /ready endpoint which returns 200 if and only if
            # it has an active connection to the edge.
            path: /ready
            port: 2000
          failureThreshold: 1
          initialDelaySeconds: 10
          periodSeconds: 10
        volumeMounts:
        - name: config
          mountPath: /etc/cloudflared/config
          readOnly: true
        # Each tunnel has an associated "credentials file" which authorizes machines
        # to run the tunnel. cloudflared will read this file from its local filesystem,
        # and it'll be stored in a k8s secret.
        - name: creds
          mountPath: /etc/cloudflared/creds
          readOnly: true
      volumes:
      - name: creds
        secret:
          # By default, the credentials file will be created under ~/.cloudflared/<tunnel ID>.json
          # when you run `cloudflared tunnel create`. You can move it into a secret by using:
          # ```sh
          # kubectl create secret generic tunnel-credentials \
          # --from-file=credentials.json=/Users/yourusername/.cloudflared/<tunnel ID>.json
          # ```
          secretName: tunnel-credentials
      # Create a config.yaml file from the ConfigMap below.
      - name: config
        configMap:
          name: cloudflared
          items:
          - key: config.yaml
            path: config.yaml
---
# This ConfigMap is just a way to define the cloudflared config.yaml file in k8s.
# It's useful to define it in k8s, rather than as a stand-alone .yaml file, because
# this lets you use various k8s templating solutions (e.g. Helm charts) to
# parameterize your config, instead of just using string literals.
apiVersion: v1
kind: ConfigMap
metadata:
  name: cloudflared
data:
  config.yaml: |
    # Name of the tunnel you want to run
    tunnel: light-api
    credentials-file: /etc/cloudflared/creds/credentials.json
    # Serves the metrics server under /metrics and the readiness server under /ready
    metrics: 0.0.0.0:2000
    # Autoupdates applied in a k8s pod will be lost when the pod is removed or restarted, so
    # autoupdate doesn't make sense in Kubernetes. However, outside of Kubernetes, we strongly
    # recommend using autoupdate.
    no-autoupdate: true
    # The `ingress` block tells cloudflared which local service to route incoming
    # requests to. For more about ingress rules, see
    # https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ingress
    #
    # The first rule proxies traffic to the ghost installation
    - hostname: ghostdemo.acgandhi.com
      service: http://ghost.default.svc.cluster.local:80
    #
    # This rule sends traffic to the built-in hello-world HTTP server. This can help debug connectivity
    # issues. If helloworld.example.com resolves and tunnel.example.com does not, then the problem is
    # in the connection from cloudflared to your local service, not from the internet to cloudflared.
    - hostname: helloworld.acgandhi.com
      service: hello_world
    #
    # This rule matches any traffic which didn't match a previous rule, and responds with HTTP 404.
    - service: http_status:404

All the way at the bottom of the file the configuration for the tunnel is defined. Each of the entries in the ingress section of the configuration define connections between a domain name to another a local URL. In this case, they connect to Kubernetes services, using the Kubernetes DNS name of that service. The format for the DNS name is servicename.namespace.svc.cluster.local, for more info see the Kubernetes docs. Change the hostname to your domain name (the same one you created a DNS record for in the last step).

Apply the yaml.

kubectl apply -f cloudflared.yaml

Finally, if you go to the hostname in your web browser (in my case ghostdemo.acgandhi.com) you should be able to see the site. You can also go to the helloworld status page (in my case helloworld.acgandhi.com) and see the cloudflare helloworld service. The blog's admin dashboard is available at ghostdemo.acgandhi.com/ghost.

Tanzu creates a Management Cluster, which is used to deploy one or more Workload Clusters which will actually run our applications.

To create a workload cluster follow the TCE docs. Make sure your kubectl config is set to the workload cluster (you can check this using kubectl config current-context ).

Installing kube-vip

Kubernetes runs workloads inside of pods. In order for those to be reachable outside of Kubernetes, they need to be exposed to the outside world using a service. Services come in 3 flavors: ClusterIP (the default, which only exposes a pod on the k8s network), NodePort (which exposes pods on a single IP using a range of ports), and LoadBalancer (which chooses from a range of IPs to assign to a pod). TCE does not come with a LoadBalancer implementation, allowing the user to choose one to use. We will be installing kube-vip (MetalLB is another choice).

To install kube-vip, we will be using a Carvel package created by fellow VMware employee Scott Rosenberg. You can follow Steps 1-5 in the excellent blog post by another VMware employee, William Lam.

If you want to be sure that your installation was successful, you could continue following William's tutorial and install the yelb demo application. Once you are done, you can uninstall yelb using kubectl delete all -n yelb (deletes everything in the yelb namepace) and kubectl delete ns yelb (deletes the namespace itself).

Hosting a Blog

As a part of the jumpbox installation helm was installed. Like Carvel, helm is a package manager for Kubernetes, which allows packages to be defined using charts, which can be installed with the helm install command. Charts also define a set of values, variables which can be set by the user to customize the install.

In order to install Ghost we first need to add Bitnami's chart repository to helm.

helm repo add bitnami https://charts.bitnami.com/bitnami

We can then install ghost with the following command:

helm install ghost bitnami/ghost

Running kubectl get all you can get some of the kubernetes resources created by Ghost (note that unlike the name suggests "get all" does not show every Kubernetes resource, and excludes many e.g. ingresses and persistent volumes).

You can see that service/ghost is of type LoadBalancer has an external IP. This IP is assigned to it by kube-vip out of the pool of IPs specified in the kube-vip values file.

kubectl get all         
NAME                READY   STATUS    RESTARTS   AGE
pod/ghost-mysql-0   1/1     Running   0          4m50s

NAME                           TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)        AGE
service/ghost                  LoadBalancer   100.71.156.29   192.168.30.240   80:30201/TCP   4m51s
service/ghost-mysql            ClusterIP      100.70.89.188   <none>           3306/TCP       4m51s
service/ghost-mysql-headless   ClusterIP      None            <none>           3306/TCP       4m51s
service/kubernetes             ClusterIP      100.64.0.1      <none>           443/TCP        5h32m

NAME                           READY   AGE
statefulset.apps/ghost-mysql   1/1     4m51s

The eagle-eyed among you may notice the only pod is one for the database. That can't be right—where's the actual website?

As a part of the installation we needed to specify the hostname of the website. The problem is we don't know the IP address the Load Balancer will get until after the deployment.

No problem. We can simply get the address and other info, and then redeploy.

export APP_HOST=$(kubectl get svc --namespace default ghost --template "{{ range (index .status.loadBalancer.ingress 0) }}{{ . }}{{ end }}")
export GHOST_PASSWORD=$(kubectl get secret --namespace "default" ghost -o jsonpath="{.data.ghost-password}" | base64 -d)
export MYSQL_ROOT_PASSWORD=$(kubectl get secret --namespace "default" ghost-mysql -o jsonpath="{.data.mysql-root-password}" | base64 -d)
export MYSQL_PASSWORD=$(kubectl get secret --namespace "default" ghost-mysql -o jsonpath="{.data.mysql-password}" | base64 -d)
helm upgrade --namespace default ghost bitnami/ghost --set service.type=LoadBalancer,ghostHost=$APP_HOST,ghostPassword=$GHOST_PASSWORD,mysql.auth.rootPassword=$MYSQL_ROOT_PASSWORD,mysql.auth.password=$MYSQL_PASSWORD

Now when we run kubectl get all we get the following output:

kubectl get all         
NAME                         READY   STATUS              RESTARTS   AGE
pod/ghost-75dd96d9f9-gc7dv   0/1     ContainerCreating   0          22s
pod/ghost-mysql-0            1/1     Running             0          5m35s

NAME                           TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)        AGE
service/ghost                  LoadBalancer   100.71.156.29   192.168.30.240   80:30201/TCP   5m36s
service/ghost-mysql            ClusterIP      100.70.89.188   <none>           3306/TCP       5m36s
service/ghost-mysql-headless   ClusterIP      None            <none>           3306/TCP       5m36s
service/kubernetes             ClusterIP      100.64.0.1      <none>           443/TCP        5h33m

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ghost   0/1     1            0           22s

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/ghost-75dd96d9f9   1         1         0       22s

NAME                           READY   AGE
statefulset.apps/ghost-mysql   1/1     5m35s
    

Once the ghost pod is created and ready you can go to the LoadBalancer external IP in your web browser, and you should see the ghost homepage.

In our next post we'll discuss exposing the blog to the internet, including support for HTTPS.

We have a laundry list of services that we need to run, and we need a way to organize the running of them all. One of the hottest new technologies in the field for orchestrating applications at scale is Kubernetes (or k8s as the cool kids call it). Standing up full, managed, multi-node, high availability Kubernetes is often considered a daunting (or perhaps more accurately, dreaded) task. Having been designed to run workloads at Google scale, Kubernetes is quite complex. Luckily for us, VMware’s Tanzu platform makes managing Kubernetes quite straightforward. Tanzu Community Edition (TCE) is the free, open source version of Tanzu that still has all of the core features we will need in our homelab deployment.

Deployment process overview

The deployment process for TCE uses a bootstrap machine which runs the Tanzu CLI.

Here’s a high-level overview of the process:

1. tanzu management-cluster create… is run on the bootstrap machine
2. Tanzu creates a bootstrap Kubernetes cluster on the bootstrap machine, using kind.
3. Providers are installed on the bootstrap cluster. These providers communicate with vSphere to provision VMs for the management cluster.
4. Management cluster VMs are created.

Creating the bootstrap machine

Typically the bootstrap machine is your local computer. This requires certain software (like docker, kubectl, and the tanzu cli) to be installed on your machine. Instead of this, our bootstrap machine will be a VM on vSphere. We’ll be using a cloud-init image of Ubuntu Server as its basis. This image will be customized with a yaml file, which will install all of the necessary dependencies and the Tanzu CLI for you.

Follow the instructions by Steve Wong here to set up the bootstrap machine.

Note: if you don’t have VMware PowerCLI installed on your machine, you can just upload the OVAs to vSphere via the vCenter web ui.

Deploying TCE: vSphere prerequisites

Before deploying Tanzu, the following requirements must be met:

vSphere

• vSphere version: vSphere 6.7u3 or later, VMware Cloud on AWS, or Azure VMware Solution account
• One of the following editions: vCenter Standard, vSphere Standard, vCenter Essentials, vSphere Essential, or vSphere Essentials Plus
• A folder in which TCE can place its VMs. To create this, go to the VMs and Templates tab on vCenter, right click on your datacenter, and under New Folder select New VM and Template Folder. Give it a name (I called mine TCEVMFolder).

• Optional (but recommended): A resource pool in which TCE can place VMs. In the Hosts and Clusters tab, right click on the cluster in which you want to deploy TCE, then select New Resource pool. Give it a name (I called mine RpTce), and leave the other settings at their default values.

• Check to make sure the datastore you are planning to use has sufficient storage for the control plane and worker node VMs.

Network

• A DHCP server to connect the cluster node VMs that Tanzu Community Edition deploys. The node VMs must be able to connect to vSphere.

• A set of available static virtual IP (VIP) addresses for the clusters that you create, one for each management and workload cluster. Each cluster requires a static IP address that is used to access the cluster’s Kubernetes control plane.

  • Make sure that these IP addresses are not in the DHCP range, but are in the same subnet as the DHCP range.

• Make sure that:

  • Traffic is allowed out to vCenter Server from the network on which clusters will run.
  • Traffic allowed between your local bootstrap machine and port 6443 of all VMs in the clusters you create. Port 6443 is where the Kubernetes API is exposed. In our case, the bootstrap machine will itself be a VM on vSphere.
  • Traffic allowed between port 443 of all VMs in the clusters you create and vCenter Server. Port 443 is where the vCenter Server API is exposed.

• The Network Time Protocol (NTP) service running on all hosts, and the hosts running on UTC.

Creating a template from the node OVA

During the jumpbox step you should’ve uploaded the node image from customer connect to vSphere. Right click on your resource pool for TCE (or if you don’t have one, on the cluster), then select New Virtual Machine. For creation type, select Deploy From Template, then select next. Select the node image OVA you uploaded. Review the details of the template, and accept the license agreement. Select the datastore on which you want TCE to place it’s VMs. Here you can also change the disk format to “Thin Provision” to use less space. Then, select the network on which you want TCE to place its VMs (this does not have to be the same network as the jumpbox). Finally, click on finish.

Once the VM has finished being created do not power it on. Right click the VM, and under template select “Convert to Template.” If you go to the VMs and Templates page you should see the template in the folder you selected.

You are now ready to deploy vSphere.  In your bootstrap/jumpbox VM as the tceadmin user, execute the following command:

tanzu management-cluster create --ui --bind <VM ip here>:8080 --browser none

This will start up the web ui, which you can access on your local machine with the port and IP you specified.

On the web ui, select vCenter. There are 8 parts to configuring the deployment:

1. Enter the credentials for vCenter. Recommendation: use the TCE user account created as a part of jumpbox installation instructions.
2. Select the number and type of nodes for the management cluster. Recommendation: development (single node) cluster of size medium or large for the control plane, and size large for the worker node. Choose smaller instances if your homelab is more resource constrained. Management cluster name can be whatever you want (in my case “manclusname”). Control plane IP should be an available static IP on the same subnet as the rest of the kubernetes network.
3. Leave as default (blank), unless using VMware NSX.
4. Leave as default (blank)
5. If you click on VM folder a dropdown should appear with all of the folders on vSphere. Select the one you created earlier for TCE (in my case TCEVMFolder). Select the datastore you want TCE to place the management cluster VMs on. Select the resource pool you created earlier for TCE (in my case, TceRp).
6. Select the network that the management cluster VMs will use. The control plane IP chosen earlier should be contained in this network. Cluster service and pod CIDRs can be left as default (unless you have existing networks that conflict with either).
7. Optional: enter details for OIDC or LDAP. Click on the “Enable Identity Management Settings” toggle if you do not wish to.
8. Select the template that was created earlier from the node OVA.

Once you click on Review Configuration, all of the settings that you entered will show up. At the bottom you can click on Deploy Management Cluster, which will start the deployment process.
Optional (alternative method): Copy the command shown in the CLI command equivalent box.  Stop running the kickstart UI (using ctrl+c). Then paste this command onto the terminal of the jumpbox VM. Here you can customize options of the command, including log verbosity (-v 0-9) and the timeout (--timeout or -t, default 30m, can be increased for slower internet connections).

Part way through the cluster creation process you should see 2 VMs be created on vSphere, one for the control plane and another for the worker node. Once the installation process is completed, you can run kubectl get all -A to see the pods, services, daemonsets, deployments, and replicasets created by Tanzu, running on the management cluster.

You are now ready to deploy your first workload.

There were a few pieces of software that fell into the category of core infrastructure, i.e. software that will act as a foundation for the other software (including Kubernetes) in the system . This included a log server (a centralized location to collect logs from all over the network), an authentication/LDAP server (which will allow us to have shared usernames and passwords throughout the network), and a certificate authority. As a result, rather than running them as workloads on Kubernetes, they were run as VMs on vSphere.

Note: technically speaking, all of the software in this post is optional, but recommended. You can setup Kubernetes without LDAP and centralized logging.

FreeIPA: a certificate authority and LDAP server

Note: The authors are not entirely happy with FreeIPA and may update this section to use a different LDAP server in the future.

FreeIPA is a piece of software that integrates both a certificate authority and a LDAP server, and also has a convenient web interface to manage it all.

Unfortunately, since it’s developed by Red Hat it only really works on their distros of Linux (such as RHEL, Cent OS, and Fedora). To host FreeIPA, I first created a Fedora Linux VM on vCenter (I used Fedora 36 Server). After connecting to this VM, I followed the steps of “method one” in this article to configure a static IP address for the VM. Once this is done, you should see the static IP address be displayed as the instance IP in vCenter. On the pfSense, I added a DNS record that points to the VM by going to Services → DNS resolver, then adding an entry under “Host Overrides”.

Afterwards I ran the following commands to install FreeIPA:

dnf install freeipa-server

Then, to configure:

ipa-server-install --ca-subject="CN=TCE Homelab Root CA, O=PRIVATE.TCEHOMELAB.COM"

Follow the onscreen prompts to configure the installation. The hostname you set for the server should be the one you configured in the DNS resolver of your router.

Afterwards, you should be able to go to the domain for the FreeIPA server (in my case auth.private.tcehomelab.com). There you will be greeted with a browser login box. Ignore this box, it doesn’t work. Wait 10-15 seconds for the page to load, then press the cancel button (you may need to press cancel more than once, I usually just spam it). There you will be greeted with the actual login page, where you can enter the credentials you chose during the installation process.  

In order to get FreeIPA to play nice with vSphere a number of schema modifications are needed. Follow this blog post to configure FreeIPA for vSphere and add it as an authentication source.

Syslog-ng: core logging server

It should be noted at this time that this is the logging server for the rest of core infrastructure. Eventually, we will set up a much more capable logging stack on Kubernetes. However, if there is a problem with Kubernetes (or for that matter any of our other core infrastructure), then we need a place to look at the logs for it that isn’t on Kubernetes. This is where syslog-ng, which we will be running on the pfSense router, comes into play.

First we will install the syslog-ng plugin on the pfSense by going to System → Package Manager and then clicking on available packages. Afterwards, in the services menu there will be an item called “Syslog-ng” where syslog can be configured.

I created two sources for syslog-ng, one of which will be used to collect the logs from the pfSense, and another that will collect logs from the rest of the network. For the former source, I used the following configuration parameters:

• Interface Selection: loopback
• Default Protocol: TLS
• CA: pfSense CA (this is the one we created in the last post)
• Certificate: pfsense web ui
• Default Port: 5140
• Default Log Directory: /var/syslog-ng
• Default Log File: default.log
• Archive Frequency, compression, and max archives were left at their defaults.

Then, under the advanced tab, I added 3 entries to accept logs over lan:

• a source, called s_lan, with the following parameters:

{
    network(
        ip("192.168.1.1")
        transport("tls")
        port(1514)
        tls(
            peer-verify(required-trusted)
            key-file('/var/etc/syslog-ng/syslog-ng.key')
            cert-file('/var/etc/syslog-ng/syslog-ng.cert') 
            ca-dir('/var/etc/syslog-ng/ca.d')
        )
    );
};

• a destination, called d_lan, with the following parameters:

{ file("/var/syslog-ng/lan.log"); };

• and finally a log object called lan to link the two:

{ source(s_lan); destination(d_lan); };

Finally, follow these steps to add a certificate authority to syslog ng.

We then had to setup ESXi to forward its logs to a remote syslog server.

We’ll assume you didn’t win the lottery, and that cost is a factor. You do need some hardware, and with a limited budget, the choice is either used server equipment, or cost effective new equipment. Global shortages of electronic components has perhaps tilted the balance a bit toward the used server option, but what we will be covering could run on either option.

Hardware components

• A Raspberry Pi
• A dedicated small x86 for running a pfSense based router/firewall
• A used, 10 year old server (Dell R710 with a couple upgrades)

Choices for the main server

This server will run ESXi to host VMs, including VMs that host a Kubernetes cluster.

You could buy new equipment such as an Intel NUC example. or go the route of a used server from ebay, craigslist, or the bulletin board of a local colo. If you go the used route, you want something new enough to run vSphere version 7.  

I am using a Dell R710 which is on the outer limit of being able to run vSphere 7. It requires a L5460 CPU swap and a H710 disk controller swap to support a vSphere install running in the unsupported, but functional, allowLegacyCPU mode. Chosen because used R710’s are very cheap but you can spend more and get an easier vSphere install experience. Something less than 5 years old is likely to work without changes or install overrides.
I won't be covering vSphere installation here. A VMUG Advantage membership is recommended as a means to get a vSphere license. vSphere also offers a 60-day free trial.

There were a few pieces of software that fell into the category of core infrastructure, i.e. software that will act as a foundation for the other software in the system (including Kubernetes itself). This included a log server (a centralized location to collect logs from all over the network), an authentication/LDAP server (which will allow us to have shared usernames and passwords throughout the network), and a certificate authority. As a result, rather than running these services as workloads on Kubernetes, they were run as VMs on vSphere.

Google Workspace, Office 365, and other services offer increasingly popular alternatives to local storage on a computer, tablet, or smartphone, thanks to the cross location and cross device access provided by the Internet. However, it is fair to question the safety, security, and economics of these cloud hosted services.

Some services openly indicate that they analyze your data, share it with third parties, and are free to change privacy terms in the future. There’s historical precedent for data breaches, price increases, and even total service shutdowns.

If you want to enjoy the benefits of universal access while maintaining complete control over your own data, you might consider hosting and managing your own online services. The necessary software can be found in free available projects. This series is going to cover the process of hosting a curated list of free and open source software to run in your own homelab.

This will be a multi-part series that will cover shared data access with personal governance, along with privacy-respecting home automation, entertainment media hosting, and environment monitoring.

In the end I aim to create an enterprise-like deployment of this software, without compromises in regards to security, reliability, and ease-of-use. While this series of blog posts covering the deployment will be fairly technical, our end goal is to create a system that anyone, techies and non-techie family members alike can use, while keeping their data safe and secure.

This journey will also be an opportunity to learn about technology and general techniques that might be useful even in larger commercial applications if you work in the IT industry.