Make long running tasks short for development

Posted on by
Reply

COmpiling the Linux Kernel is a long running task. Oh, sure, you can put -j 32 and it speeds it up tremendously. But it still is a long running task. And by that, I mean it fits the following definition:

A Long Running task is one that takes so long that you start doing something else before it is done.

“aDAM YOUNG”

If you are familiar with the concept of Flow, then a long running task is one that breaks the flow.

It seems that these kind of “flow Blockers” are common in my line of work. I tend to work on distributed systems, and network based workflows are usually long running tasks. Things like:

  • Ansible Playbooks that allocate a bunch a virtual machines before install software on them.
  • Software install process like ipa-server install
  • Automatically build, install, and test the Linux Kernel

This last one is the thing I am working on now.

The funny part is that I don’t have to do the hard work of making the Kernel compile. This is the responsibility of the team I am building this for, and I can assume that it works OK. This assumption will allow me to skip actually building the Kernel for the majority of my effort.

What do I need to do? I need to write a small yaml file that gitlab will use to kick off a CI/CD Pipeline. This file is a fairly thin wrapper around a bash script (similar pattern to RPM spec files, Ansible, and so forth) that will be executed on the remote system when triggered by a new or modified merge-request. This code needs to install a bunch of RPMs via dnf, kick off the kernel build, and then upload the artifacts to an object store. Right now, I am working on that last bit.

If I wait for the Linux Kernel to build, each iteration of code changes in my development cycle will take more than an hour. Thus, I want to put off actually building the Kernel until I know the rest of the tasks work. All I need to do is test that something gets uploaded, I don’t really care what that thing is. So I comment out the make and replace it with the simplest command I can think of to create a file:

 #- make -j 16 rpm-pkg mkdir ./artifacts
 - touch  ./artifacts/test.test

There is actually a bunch of stuff I comment out including the part where I install packages from dnf. If I know that the code is working, there is little benefit to running it during the debug cycle for the upload stage.
However, I only want to comment out lines, not remove, Why? Because the code needs to be put back in to place in the final assembly, and I don’t want to have to debug the long running process.

With the actual build elided from my process, I can test the system by submiting a no-op merge request and see that gitlab triggers the runner to run my code. My round trip is now about a minute, short enough that I can keep myself on task.

Usually.

Date format suitable for file names

Posted on by
2

It is rare that you want to write something without later wanting to be able to read it back. One common way of organizing files that are generated regularly is by time stamp. If you want to add a timestamp to a file name, you can do so using the date command.

In order for the filenames to sort in the right order, you want the name to go from largest unit to smallest.

Here is an example that creates a filename-suitable string formed Year->second. I remove all unnecessary formatting characters.

date --rfc-3339=seconds | sed -E -e 's! |-|:!!g'

The date command reads the current date/time on the local system. –rfc-3339=seconds produces output that looks like this:

$ date --rfc-3339=seconds 
2021-11-03 10:57:14-04:00

In order to keep the regular expression concise inside the sed command, the -E switch tells it to use extended regular expressions, including the alternation character ‘|’ . Thus, the regex ‘ |-|:’ matches a space, a dash, and a colon.

bkr job status

Posted on by
Reply

Here’s a one liner for showing the status of all your beaker jobs.

for JOB in $( bkr job-list -o $( bkr whoami | jq -r '.username' )  | jq -r ".[]"   ) ; do bkr job-results $JOB | xpath -q -e "string(/job/recipeSet/recipe/@status)" ; done

Select only the Jades

Posted on by
Reply

Some custom jq for RegEx selection of OpenStack Ironic baremetal nodes. Our Server types show up in their names. I want to be able to build lists of only the Mt. Jade Servers, which have names that look like this:

jade09-r097

openstack baremetal node list  --sort provision_state:asc   -c UUID -c Name -f json | jq '.[] | select(.Name | test("jade."))'