Post 0a: Prerequisite Hardware
We’ll assume you didn’t win the lottery, and that cost is a factor. You do need some hardware, and with a limited budget, the choice is either used server equipment, or cost effective new equipment. Global shortages of electronic components has perhaps tilted the balance a bit toward the used server option, but what we will be covering could run on either option.
Hardware components
- A Raspberry Pi
- A dedicated small x86 for running a pfSense based router/firewall
- A used, 10 year old server (Dell R710 with a couple upgrades)
Why the Pi?
We will be implementing some home automation requiring connections to physical IO devices, and we decided that using an external Pi is safer and easier than opening non default mappings and privileges to enable physical device access from VMs and Kubernetes pods. It can also offer a faster return to service after a power outage. Also it is cheap, well documented, readily available, small in physical size, and low in power consumption
Why pfSense on dedicated hardware?
We want something better than most consumer grade routers because we will be using VLANs, and can use the integral plug-ins for DHCP, DNS, certificate management, and logging. This is a critical resource and we want this to start service early after a power outage - thus dedicated hardware is preferred to running this in a VM. The hardware required is not expensive. Minimum recommended: any 64-bit x86 compatible CPU, 4GB RAM, 64GB disk, 2 gigabit ethernet ports, USB port for install. You also want a managed switch that supports VLANs.
I’m using a used Dell Edge Gateway from ebay, featuring an Intel Atom with 2 gigabit ethernet ports, and 8GB of memory with a small SSD. Alongside it I have a Mikrotik RB260GS 5 port switch.
Alternatives to pfSense: OPNsense, an existing router that supports VLANs
Choices for the main server
This server will run ESXi to host VMs, including VMs that host a Kubernetes cluster.
You could buy new equipment such as an Intel NUC example. or go the route of a used server from ebay, craigslist, or the bulletin board of a local colo. If you go the used route, you want something new enough to run vSphere version 7.
I am using a Dell R710 which is on the outer limit of being able to run vSphere 7. It requires a L5460 CPU swap and a H710 disk controller swap to support a vSphere install running in the unsupported, but functional, allowLegacyCPU mode. Chosen because used R710’s are very cheap but you can spend more and get an easier vSphere install experience. Something less than 5 years old is likely to work without changes or install overrides.
I won't be covering vSphere installation here. A VMUG Advantage membership is recommended as a means to get a vSphere license. vSphere also offers a 60-day free trial.
There were a few pieces of software that fell into the category of core infrastructure, i.e. software that will act as a foundation for the other software in the system (including Kubernetes itself). This included a log server (a centralized location to collect logs from all over the network), an authentication/LDAP server (which will allow us to have shared usernames and passwords throughout the network), and a certificate authority. As a result, rather than running these services as workloads on Kubernetes, they were run as VMs on vSphere.