One of my responsibilities on the Linux Kernel team is to build our version of the Linux Kernel from distinct branched in a git repo. Our approach is a little different than most organizations, and so we have custom code to implement our logic. Over time, I have come to realize a few patterns that make our lives easier in how I approach organizing this code.

Use a real programming language

We chose Python. Maybe Rust would have worked as well. Part of the decision is based on what the team can easily consume. We don’t want to stick a non-trivial language context-swap into the process.

Wrap Shell Commands

Since we use python, we use plumbum to execute commands. This is the norm, not the exception: we even use commands when there are external python libraries that might do the same thing. For example, we use plumbum to execute git commands instead of the git library. Why: first, the git library does not implement every single git command, and we use some esoteric ones. So, instead of splitting our git approach across both plumbum and python-git, we chose a single approach. This reinforces the second choice: if you can do it from the command line, make it easy to understand how to do it from Python.

Unit Tests

Unit tests are not functional tests. Unit tests confirm that the code does what you think it does. The responsibility of unit tests end at the edge of your code. They are not there to test that the shell command you are executing does what you think it does. That is a functional test responsibility. Unit tests should have no side effects.

To make this happen, we put a wrapper around plumbum called exec_log. Exec log essentially logs the command executed and then calls plumbum. It also has a global switch, dry_run, that allows converting all the plumbum calls to mock calls. Each unit test sets up a list of responses, and the exec_log calls return each response in turn without actually executing the plumbum call. There has been a lot written on mock implementations; twe may find that we want something more complex in the future, but this serves well enough for now.

Programming is the serial application of the scientific method. Make a change, and the nsee the result of that change, comparing the result with a mental hypothesis. THe faster you make this cycle, the more progress you can make. Nothing kills productivity like having to wait a couple hours to see the result of your work. But automation often is built such that you have to run it end-to-end to see impact. If a process breaks on minute 99 of a 100 hour process, you are going to be sorely annoyed.

Break up Long Running Tasks

Take the long running process and make it so you can start it after each phase, with an approprate checkpoint. As an example, our Kernel build process has the following steps:

• build a local git repo
• checkout out the appropriate basis branch
• cherry-pick the topic branches onto the basis
• generate a build script and bill-of-goods file
• tag the tree
• push the tagged tree to the central repo

This is called the generate stage. It takes a few minutes to complete. Once it is done, we perform the following steps

• Generate a tarball with the patches from the repo
• reinitialize the git repo to the basis branch
• untar the tarball and apply the patches to the git repo
• build the kernel
• use the kernel’s RPM build process to create an RPM
• upload the tarball and the rpm to a local web server

Obviously, this is a very opinionated work flow. Since my team only provides patches to other organizations, we want to replicate that process as close as possbile, which leads to steps that you would not perform if you were merely providing the RPMS. BUilding the Kernel takes a long time. Uploading the RPM to the local server is quick. However, remote servers change, either location or in access approach. If you want to fix a broken process, you want to start at the point it broke. Being able to run just the upload stage skips an hour-or-more of processing time.

Don’t functional test against production

If you have no side effects, and there are no secrets involved, you can run your code against production. This is rarely the case. If you make changes to any form of persistent storage, you need to have an insulated environment to test against. Using the example above, we want the developer to be able to be able to push and pull from the remote git repo without corrupting the production git repo. We also want them to be able to post to a web server at the end without posting broken, interim builds.

Out automation requires you to specify up front, via an environment variable, whether you want it to go to production or not. The developer has to actively chose which mode they are going to work in. This is in addition to any remote access control rules set up. Sometimes the same person can run development and production runs. We learned the hard way to make this check the very first step, so that no time is wasted if you do actually want to go to production.

Run it from the command prompt first

There are lots of great automation tools out there, like Ansible, Gitlab Runners etc. Those things are not where you should do your development. Running automation from a central controller is only done after you can run it locally. You never want to be at the mercy of your automation pipeline tool to check to see if something broke. This is really a part of breaking up long running tasks, but there is more to it than this. You want to avoid vendor lock in. So don’t build your automation around a certain vendors approach to automation; make that the last mile hookup. If your Ansible tower or gitlab runner goes down, you should be able to run the same automation on a shared server. If you move from gitlab to github, you should not have to change your automation, only how you trigger it from the server.

Documentation lives in the repository with the project

README is your friend. Put into to it everything. When it gets too big, split it into smaller documents, and use the README to point to them. If you have a central knowledge base, make sure it points to your README. When you make changes, update the README, or the appropriate sub-document.

Avoid config files

Any change you make needs to go into git. If you need a config file to tell your automation something, and that file is going to be different for each person that runs it, try to make that a configuration option based on the user name instead.

If you have to track a version of a project (like we do with building the Linux Kernel) calculate what version to build from git (use git like a database). Don’t have yet-another-place that you have to keep data consistent.