This one is going to be a little light on details, as we are still working through it, but I’d just like to share what I’ve been working on the past couple weeks. Note that this is for a proof-of-concept cluster, and is not for production.
My new position has me working with Red Hat customers in the financial services industry. These customers have strict regulations for controlling access to machines. When it comes to installing OpenShift, we often are deploying into an environment that we call “Air Gapped.” What this means in practice is that all install media need to be present inside the data center, and cannot be fetched from online on demand. This approach is at odds with the conveniences of doing an on-demand repository pull of a container image. Most of the effort involves setting up intern registries and repositories, and getting X509 certificates properly created and deployed to make access to those repositories secure.
The biggest things we learned is that automation counts. When you need to modify a file, take the time to automate how you modify it. That way, when you need to do it again (which you will) you don’t make a mistake in the modification. In our case, we were following a step-by-step document that got us about halfway through before we realized we made a mistake. Once we switched from manual edits to automated, we were far more likely to rollback to a VM snapshot and roll forward to make progress. At this point, things really started getting smoother.
The CoreOS boot process is a lot like a machine bring up process, especially in a Virtual Machine environment. The RHEL CoreOS image can be through of like a boot loader, that then needs to download its configuration information through a PXE process; it uses DHCP to get an IP address and the URL of where the configuration information lives. For OpenShift, the configuration information is in a JSON file called bootstrap.ign. We learned to generate a vanilla-simple version of this, make sure it boots, and then roll in changes one at a time as we made them. Using jq to test the format was necessary but not sufficient to checking the format.
The team that did this before us made a good automation tool here:. We made significant progress once we switch to using it. In turn, it uses a nice utility for building the JSON structure.
The general process for deploying OpenShift 4 is to deploy a bootstrap server, use that to configure the controllers, and then remove the bootstrap server. By the end of the day today, we had done that, and also enrolled a couple of compute nodes into the cluster. Got the WebUI working, too. I wish I could take credit for any of this, but the fact is that I work with great people that made it happen. Thanks to the Red Hat Telcom team for paving the way and the Red Hat consultants who executed it.