My work as a cloud Solutions Architect is focused on OpenShift. Since I work in the financial sector, my customers are very security focused. These two factorshave converged on me working on OpenShift installs on disconnected networks.

The current emphasis on OpenShift is for virtualization. While virtualization can be nested, it typically has a performance penalty. More important, though, is that virtualization is a technology for taking advantage of bare metal installs.

I need to run OpenShift 4 on baremetal in my homelab via a disconnected install . Here we go.

To read the OpenShift install directions, you need a Red Hat subscription.

Hardware

I’ve written up my Hardware setup before. Not a lot has changed, except that I have a fourth Dell r610 now. This one is problematic, in that I can’t seem to talk to the iDrac on it. I can get it to PXE boot, but I need to go manually push the button myself. I’ve kep it close; the rack is now in my attic, behind a small door.

It is a quieter in my office now, but not silent, when the cluster is running.

Thus, I can install the machines via PXE or via USB. Right now, I am going to work through a USB based install. This removes one layer of technology for most people. I can layer on the PXE approach after.

I do my work from a machine called Nuzleaf. This is a the Bastion host; it has direct network access to the outside world. Nuzleaf runs the HTTPD server that hands out the control planes ignition data. It also serves Yum repositories and container image repositories. It has two interfaces: a wireless one for the outside traffic and an ethernet port connected to the servers.

While I have a programmable switch, I don’t need it for this setup. Instead, Everything is in a very flat setup behind the NUC.

Downloads

I downloaded the ISO rhcos-4.6.8-x86_64-live.x86_64.iso from the Red Hat site. This has been copied to a USB drive with:

sudo dd bs=4M if=rhcos-4.6.8-x86_64-live.x86_64.iso of=/dev/sda

If you look closely you can see the USB stick in the top Server (the one not racked properly, of course) which is going to act as the Bootstrap server.

The bootstrap server has been named Boldore. Mneomnic for Bootstrap. I really should rename the three servers. I think I want to go backwards from Z. I have Zygarde and Zubat…which ever of those is on the bottom should continue. Yungoos. Xatu. I like this. Need to redo things with those names.

If I were to boot the Dell r610s from the USB stick right now, they would not be able to download their ignition data. Thus, when I boot them, I need to intercept the Grub stage and inject the right values for ignition and other things.

You can see why I want to do this all via PXE.

In addition to the coreos image, I grabbed the installer and the command line tools. All of these files are on Nuzleaf in /home/ayoung/apps/ocp4.6.

$ ls ..
homelab  openshift-client-linux.tar.gz  openshift-install  openshift-install-linux.tar.gz  pull-secret  README.md  rhcos-4.6.8-x86_64-live.x86_64.iso

I’ve already extracted the installer:

tar -zxf openshift-install-linux.tar.gz

Install Config

I made a subdir for the generated files, including the one I need to manage by hand: install-config.yaml

BIG RED BLINKTAB WARNING: If you run the installer, you will delete this file. It is essential for sanity and reproducibility that you always have a backup copy of this. I create a file called install-config.yaml.orig. I tend to edit this one and copy it over to the file install-config.yaml. I’m going to try something different this time, and see if I can do it with a symlink instead.

Here is my starter install-config.yaml.orig

apiVersion: v1
baseDomain: home.younglogic.net
compute:
- hyperthreading: Enabled   
  name: worker
  replicas: 0 
controlPlane:
  hyperthreading: Enabled   
  name: master 
  replicas: 3 
compute:
- name: worker
  platform: {}
  replicas: 0

metadata:
  name: homelab 
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14 
    hostPrefix: 23 
  networkType: OpenShiftSDN

  serviceNetwork: 
  - 10.22.21.240/16
platform:
  none: {} 
fips: false 
pullSecret: 'removed'
sshKey: |
  ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSmDjgSrcA9NSTbv1XN7ZFnH7kDE2fq69KaKDqYM/JVM3gYP7+NwRwlsulNrtLjsdbOgocAMGKdCy3UWpKNLCDwIPC2ONW8xMrc3mjVfK2Te7Tbgu1E+QeD7p76nJAn22+hTFa1nMFvkGEpvBE1b2YYEHV+UrsVN/f/+YUNeD2Wvfcaq/H7kgWkwanwTq4NpljEa/5UDXjYopL2W8IpQyKo5xR7srLm8AwAaWVLUBaSRsIHI2BUgIjk07TT+4/9K/BlMtaomL4/a8mW56FbE6ZmsyD7tgKT6TKkezlXAm2VyaWUG8+Ee2jQ2gW5eRvZuafirfSjpXRrO7FdLukygkDtGWDYwZnEi8zn5zPVRw7Eor8jg6xAkHdicGsMxK5JdMfkE7BHB7f5RKIv6je3aH1SG9LAYOKuMvVu7Z0UbgTTAmwITo/A3VxxRD4MWePce0fGhKHEKcT+hb5Aevnjeej08YvT8ErGvIOzq6tP9MKgN01ZrMnJ28idYzQXAfUZLM= ayoung@nuzleaf.home.younglogic.net

Aside: One thing I would love to see (since I am designing…er dreaming here) is the ability to pull in the ssh and pull secrets from remote files.

To generate the manifests:

[ayoung@nuzleaf ocp4.6]$ ./openshift-install create manifests --dir=./homelab/
INFO Consuming Install Config from target directory 
WARNING Making control-plane schedulable by setting MastersSchedulable to true for Scheduler cluster settings 
INFO Manifests created in: homelab/manifests and homelab/openshift 
[ayoung@nuzleaf ocp4.6]$ ls homelab/
install-config.yaml.bak  install-config.yaml.orig  manifests  openshift

It worked! I mean, running this way removed the symlink but left the .orig file intact.

Inside the homelab directory we have two generated sub-directories: manifests and OpenShift

For iterative development, you want to be able to re-establish your starting point quickly. One technique you can use to do this is to make sure that directory that gets filled with auto-generated files can always be wiped out and regenerated.

So I am moving my install config up one level.

[ayoung@nuzleaf ocp4.6]$ rm -rf homelab
[ayoung@nuzleaf ocp4.6]$ mkdir homelab
[ayoung@nuzleaf ocp4.6]$ ln -s $PWD/homelab-install-config.yaml homelab/install-config.yaml
[ayoung@nuzleaf ocp4.6]$ ls -la homelab
total 0
drwxrwxr-x. 2 ayoung ayoung  33 Jan  8 10:21 .
drwxrwxr-x. 3 ayoung ayoung 264 Jan  8 10:21 ..
lrwxrwxrwx. 1 ayoung ayoung  52 Jan  8 10:21 install-config.yaml -> /home/ayoung/apps/ocp4.6/homelab-install-config.yaml

I will put this in an Ansible playbook at some point. For now, I can recreate the install manifests in a clean and easy manner.