Converting a Shell Script to Python

We have a build system that has grown organically. It started as a shell script. We needed to run it from gitlab, so we wrote helper scripts to insulate our code from gitlab. Then we added some helper functions to mimic the gitlab interactions when working with them from the comand line. The helper functions grew until you could not practically run the original shell script without them.

It is a mess.

I want to refactor it.

Refactoring Shell is painful.

I want objects. I want python.

So I am rewriting the gitlab and functions layer in python with an eye to rewriting the whole thing. Here’s what I have learned;

Continue reading

Reading envvars using plumbum

One of my new years resolutions this year was to try and replace bash scripting with python in some of my projects. The reasons for this include the ability to use objects to group parameters together.

At today’s (Jan 19, 2025) Python over Coffee meetup, Ricardo suggested that I look at the plumbum library as a tool to help make that transition. In my first hack at using it, I came across an issue: I need to read in environment variables.

Continue reading

GitPython do two files differ in different branches

Not quite as succinct as I would have liked, but this does work. This is a one liner in Bash, and I had hoped to do it with a single python call once I had the repo object established. In general, diffs are not simple to do on blob objects. I had trouble using the difflib and set approaches to diffing the two files, possibly due to the fact that I was originally not decoding the streams. Thus all the splitlines work probably can be simplified.

Continue reading

Building the Linux Kernel on one machine, deploying on another

Software Development flow is an elusive beast, easily frightened away. The easiest way to lose the flow is to have a long development cycle. Building the entire Linux Kernel takes time. If you are developing a single driver that can be built as a stand alone module, you do not want to have to rebuild the entire Kernel each time you make a change to that module. If that module crashes the machine when it runs into an error, you do not want to have to wait for that machine to reboot before digging in to the debugging process.

These two guidelines lead to a pattern of building on one machine and deploying on another. Here is one way to go about this process.

Continue reading

Long Refactoring: Resetting the class model

With the inclusion of the Generators, the main functions seem to be in the right shape. What seems off is the naming of things. For example, we have a couple functions that take a parameter named “board” but it turns out they mean different things. The names of classes do not seem to align with what they are doing.

Lets continue with the renaming of things and see where it leads us.

But first: we can remove the exceptions to the flake8 testing, and, with the removal of some white space, we can run with all the pep8 rules intact. Now, on to renaming.

I am not going to go through each commit, but will try to provide enough context so you could follow the logic in the commits in github.

This is the last article in my Long Refactoring series. You can read the first article here.

Continue reading

Long Refactoring: Generators

While the main solving function is now cleaned up, I am not very happy with the code that checks the validation on each cell.

I am not going to go through each step here, as the pattern is well established now. Pull out of the function anything that is a distractor. Once you have the heart of the algorithm, clean it up. Then put things back together.

Continue reading

Getting a File from gitlab using a TOKEN

I work on a system in our lab that I connect to via SSH. Fetching a file via the web and pushing it to this machine is quite slow, and I want to pull on this machine directly. The trick to make this work is to use a TOKEN from gitlab and oauth2 as the username portion of the URL. For example

curl -O   https://oauth2:$CI_JOB_TOKEN@gitlab.com/api/v4/projects/$PROJECT_ID/packages/generic/$PROJECT_NAME/$PROJECT_VERSION/$FILE

The CI_JOB_TOKEN must be set to a token that has appropriate permissions on your project. Most of the values I wrote as envvars could be copy-pasted from the gitlab packages link on the web.