Dogfooding

When I was contracting at Sun, someone sent out a memo using the phrase “eat our own dog food.”  Scott McNeally sent out a response to the effect that Sun did not make dog food, “We Fly our own airplanes.”

In the software world, the phrase has become so used that it has been verbed; If you work at a software company that uses its own products in house,  you are Dogfooding.

Since so much of my professional work has been on Red Hat Enterprise, I’ve tended to run Debian based systems for my desktop, to keep abreast of what is happening in both worlds.  At Red Hat, I’ve finally had reason to run Fedora as my desktop machine, and have been enjoying the experience.  The vast majority of the Software I run now is distributed by the company I work for, and is Open Source.  It really is an amazing feeling. I am now running Fedora 11 not only on my work laptop, but my wife gave me permission to blow away the Windows install on her eeepc and install it there as well.  Fedora makes a great Netbook OS.

However, one tenant of Software Development is to develop on the platform on which you are going to ship.  For Red Hat, that is RHEL5, and so I need access to a RHEL5 install, and in fact need both 32 and 64 bit, since the world has not completely moved to 64 bit yet.  I’ve used virtual machines in the past, but always from VMware.  Now I am running QEMU/KVM on my Laptop.  Like most things Linux-y, the command prompt method of controlling the virtual machine subsystem is a first class citizen:  I don’t need a visual console to start up a VM.  I realize this is old hat to some people, but It is new to me, and I am enjoying it.

That is the wonderful thing about the Open Source development model:  you very quickly take ownership of the software that is essential to you.  Whenever a user becomes a contributor, that software is no longer just something out there.  It has become personal.

Anyways. As I Fly the Red Hat virtualization airplane, I’ve learned a few things.  The GUI, Virtual Machine Manger, is great for getting over the learning curve.  The command line is virsh.  These tools are a  part of the libvirt project.  There is a command to start Qemu based vms directly, but his seems to bypass the libvirt infrastructure.  Rnning qemu-kvm allowed me to start a VM saved in /var/lib/libvirt/images, but was not able to talk to the kvm subsystem.  One thing that threw me was that connecting to the virtual shell and running the list command did not show my virtual machine;  by default, that only shows running virtual machines, and you need to add the –all option to see non-running VMs..important if you want to only run them occasionally as I do.  To connect to the system, run:

sudo virsh -c qemu:///system

There is also another URL qemu:///session that I am not yet familiar with.

Working with VMware, I was comfortable with the split of the info into a vmdk and vmx files for binary data and configuration information.  IN KVM/Qemu/libvirt land, the equivalent of the VMDK is  a boot image.  This seems right to me, in keeping with the fearless Unix way of not inventing a new technology if an existing one makes sense.  The Analogue of the vmx file is in /etc/libvirt/qemu/.

One thing I would like to get set up is bridged networking for my VMs to the corporate lan.  The base install takes the conservative view that the network should be confined to the local system.  I’ve seen some write ups to getting TAP interfaces set up to allow your virtual NICs to get packets to and from the physical NICs, but haven’t done that yet.  The configuration for the host local network can be viewed from the virtual machine manager, and it shows the range of DHCP address given out to the hosts.  It contains a wizard for adding new networks, but I am not sure if the VMware paradigm of a bridged network maps cleanly to the Linux view…I suspect not.  I see under advanced options when creating the VM that I can set the Network to bridged, but it doesn’t seem to find my DHCP server to PXE boot.  As an aside, I’d like to understand how this operates in an Ipv6 environment, as Uncle Sam is now dictating IPv6 for all new Software purchases.    So many things to learn!

True Two Tiered OS Deployment

JBoss clustering and Penguin’s Clusterware (bproc) have one thing in common: the view of the the system spans more than a single underlying system. Other systems have this concept as well, but these two are the ones I know best. Virtualization is currently changing how people do work in the datacenter. Many people have “Go virtual first” strategies: all software deployed can only be deployed inside virtual machines. While this simplifies some aspects of system administration, it complicates others: Now the system administrators need tools to manage large arrays of systems.

If you combine current virtualization practices with current cluster practices, you have an interesting sytem. Make a clustered OS instance of Virtual machines and deploy it across an array of embedded Hypervisors. Any one of the VMs that make up the clustered OS image can migrate to a different machine: after running for a length of time, no VM may be on any of the machines that were originally used to run the clustered OS Image.
Such a system would have many benefits. The virtualization technology helps minimize points of failure such that, in theory the whole system could be checkpointed and restarted from a n earlier state, assume that the networking fabric plays nice. System administration would be simplified as a unified process tree allows for killing remote processes without having to log in to each and every node to kill them. Naming service management is centralized, as is all policy for the cluster. Additionally, multiple OS images could be installed on the same physical cluster, allowing clear delineation of authority, while promoting resource sharing. Meta system administrators would see to the allocation of nodes to a clustered image, while department system admins would manage their particular cluster, without handling hardware.

Reading the links from a webpage

I needed to see the set of RPMs in a YUM repository. I wanted to do this as part of a larger script. To do this, I fetched the page via wget, and then applied an xsl transform on it using the command line tool xsltproc.

Here is how I called it:

wget -q -O – http://spacewalk.redhat.com/yum/0.5/Fedora/10/x86_64/os/Packages/ | xsltproc –html showhrefs.xslt –

And here is the xslt file showrefs.xslt

<?xml version=”1.0″ encoding=”UTF-8″?>
<xsl:stylesheet xmlns:xsl=”http://www.w3.org/1999/XSL/Transform” version=”1.0″>
<xsl:output method=”xml” indent=”yes”/>

<!–  shut off the default matchin rule –>
<xsl:template match=”text()” />

<!– print the href value of the hyperlinks –>
<xsl:template match=”a”>
<xsl:value-of select=”@href” />
<xsl:text >
</xsl:text>

</xsl:template>

</xsl:stylesheet>

Booting into single user mode with grub

I recently had a problem where one of the daemons run at startup would hang.  In order to disable this, I needed to boot into single user mode, and chkconfig the service off.

The key to booting into single user mode is the word ‘single’ on the kernel command line.

Heres the steps:

  1. Reboot the system
  2. At the grub screen, hit esc to stop the default boot sequence
  3. use the arrow keys select the kernel image you wish to boot
  4. type ‘e’ to edit the kernel image startup parameters
  5. on the screen use the arrow keys again to select the line with the startup parameters.  This should be the  the middle line, the one that starts with the word ‘kernel’.
  6. Type ‘e’ to edit this line
  7. append the word ‘single to the end of the line and type enter
  8. type the ‘b’ key to boot the kernel

Dependency Licenses

The following script, when run on a RPM based system, tells you what licenses your application might trip by linking with other code.  You have to  set $MYEXE to som path to an executable.

for LIB in `ldd $MYEXE | sed -e ‘s!.*=>!!’ -e ‘s! (.*!!’` ; do rpmquery –whatprovides $LIB ; done | grep -v “any package” | sort -u | xargs rpmquery –qf %-30{NAME}:%{LICENSE}”\n

Here is a sample run for perl:

glibc                         :LGPL
perl                          :Artistic or GPL

Note that this is only for the linked files that are registered with RPM.

Small Scale High Performance Computing

At the top end of computing there is are the Supercomputers.  At the bottom end there are embedded devices.   In between, there are a wide array of types of computer systems.  Personal computers, workstations and servers are all really just a sliding scale of the same general set of technologies.  These systems are , more and more, the building blocks of the technologies higher up on the scale. Enterprise computing typically involves high-availability and high Input/Output (I/O) based systems.  Scientific and technical computing is similar, but high availability is not as important as performance.  Three of the variables that factor into system design are parallelization, running time and (disk) storage requirements. If a job is small enough that it can run on a single machine in a reasonable amount of time, it is usually best to leave it to do so.  Any speedup you would get in parallelizing the job and distributing the workload is offset by (Amdhals law) the serial portion of the job, the added overhead of parallelization, and the fact that you could run a different job on the other machine.  If your task is parallelizable, but is very storage intensive, you need a high speed disk interconnect.  Nowadays that means fiber channel.

Only if a job takes so long that it makes sense to parallelize, and that job does not require significant access to storage does it make sense to go to a traditional Beowulf cluster.  Although Infiniband does handle the interconnect for both network and storage access, the file systems themselves do not yet handle access by large clusters.

This is the point for which we need a new term:  storage bound, single system jobs that should be run on their own machine. Examples of this abound throughout science, engineering, enterprise, and government.  Potential terms for this are:Small Scale HPC, Single System HPC,  Storage Bound HPC, but none of them really roll of the tongue.