dpkg-others

dpkg-query -S `which $1` | cut -d: -f1 | xargs dpkg-query -L

This command finds other files in the same debian package as a given executable.  I put it into a file called dpkg-others.  Example run:

/home/adyoung/bin/dpkg-others ODBCConfig
/.
/usr
/usr/bin
/usr/bin/DataManager
/usr/bin/DataManagerII
/usr/bin/ODBCConfig
/usr/bin/odbctest
/usr/share
/usr/share/doc
/usr/share/doc/unixodbc-bin
/usr/share/doc/unixodbc-bin/copyright
/usr/share/doc/unixodbc-bin/changelog.gz
/usr/share/doc/unixodbc-bin/changelog.Debian.gz
/usr/share/menu
/usr/share/menu/unixodbc-bin
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/ODBCConfig.1.gz

War, Politics, and the Army

I’ve chosen not to make this a political or military blog. There are enough people out there with more experience and more firsthand knowledge on both sides, that I feel like I would be talking out of the wrong orifice.

This is more a rumination, an attempt to place my current wash of thoughts into a coherent structure than a policy statement. I really feel the weight of Socrates advice to admit that I am not wise.  I won’t make many of these.

I think that going into Afghanistan was the right call. Yes, I know that this is not really a surprise, not is it even that controversial a statement. The Taliban government was still in process of taking control of the whole country, and thus had questionable legitimacy at best. Since the government decided to protect the organization claimed responsible for attacking the US on September 11, 2001, I feel we were justified in returning fire.
The war in Iraq is hurting this country. I won’t go through my initial views on the war or my background as an Army officer. Instead, let me state I think the economic burden to this country is more than we can continue to bear. It breaks my heart to say this, because I think it means we are going to do to the Iraqi people what we did to the Montagnards in Viet Nam: pull out and leave them to be massacred. We left Viet Nam because the long term cost was wearing down our Nation. Iraq is less bloody than Viet Nam, but war is far more expensive than it was even forty years ago. If we continue this way, and other nations continue to buy up our debt, we risk losing control of our own country. It seems like General Petraeus and his staff have done an outstanding job this past year in reducing the violence in Iraq. I know that the people on the ground are good soldiers.  Probably better soldiers than I ever was. They may hate being in Iraq, but they want to win the fight.

I remember meeting Viet Nam vets from the VA when I was in high school. These guys were obviously emotionally troubled by what they had seen. One guy said that he still wanted to go back and finish the job. This was 15 years after the last troops had left country. Very few people think that it was in either America’s or Viet Nam’s interest for us to prolong the conflict. But a soldier who has fought, killed, and watched his buddies die for a cause can’t help but either shut down or give his heart to the effort. I think the same is true of our guys in country right now. They want to win in Iraq. They want the Iraqi people to live free of fear, free to raise their kids, and free to rejoin the international community. I want that, too. But I think the overall conflict is much more difficult than that. We have stirred up vast swaths of resentment by being a foreign invader. We have resurrected the ghosts of the Crusades and colonization that bring forth the resistance fighters. Iraq was barely pacified under Hussein. It is going to take a lot more than a Band Aid to fix this sucking chest wound. It would be helpful if we could have truly pulled in an international effort to rebuild Iraq, but we lost that opportunity.

The recent news from Pakistan is troubling.   I think we blew it in Afghanistan by being so quick to rush to war in Iraq.  If the funding that went to the war effort instead went to building infrastructure in Afghanistan (“Thank you for helping us defeat the Soviet Union, here is your pay back.”) we would not be fighting the battles there that we are now. I think the real cost was much higher than anticipated, and I wonder now if we can truly afford it.

One thing that bothers me is that we no longer declare war. The way the constitution was written, it was up to congress to declare war. Not to grant to the President the authority to do so. Deploying troops into a firefight means we are at war. Until World War II, it was expected practice. Yes, we fought the Native Americans, and rebels in many third world countries without such a declaration during this countries history. But if we are going to remove a government from power, we need to state it in no uncertain terms. International Law expects this. I think we have removed one of the checks from the separation of powers in the National Government. Perhaps the language of the constitution should be more a long the lines of: Any deployed forces are allowed to defend themselves. To do anything more than this requires explicit approval from congress. This approval can not be granted a-priori.  But I think the constitution states this, just in the language of 1780.

To the soldiers in Iraq and Afghanistan: Thank you. Especially the burnt out privates that are just looking to get home, and the fathers and mothers that just want to see their kids again. May the remainders of your tours be boring and may you come home whole.

To those that have lost limbs, eyesight, faculties, or to the survivors that have lost loved ones…I don’t have the words to say that will not trivialize your loss. These losses will be remembered and honored.

High Availability and dealing with failures

Time to use my pubic forum to muddle through some design issues I’m struggling to lay straight.

A Data center is made up of several objects: Servers ( computers, usually horizontal), racks(hold the computers), switches(a network device that connects two or more computers together), power sources, and cables (both electric and network cables, to include fiber optic for storage devices). A server in the data center can serve on or more roles: storage host, computation host, administration, or user interface. If an application is hosted in a data cetner, it is usually important enough that it requires some guarantee of availability. This application will resided on a computation host, and require access to other types of hosts. An email server stores the received messages in a storage host, probably connected to the computation host via fiber optics. It receives and sends messages via a network connection that goes to the outside world. It may also talk to a User Interface machine that runs a web server and an application that allows web access to email. If the computation host loses connectivity with either the public network or the storage host, it cannot process mail. If the web server loses connectivity to the mail server, certain users cannot access their mail.

There are many reasons that connectivity can fail. The major links in the chain are: OS failure, Network interface card (NIC) failure, bad cable, disconnected cable, bad switch, unplugged switch, switch turned off. Once you pass the switch, the same set of potential problems exist on to the other host. To increase reliability, a network will often have two switches, and each server will have two NICs, one plugged into each switch. The same set up goes for storage, although different technologies are used. As a general rule, you don’t want to have things running in hot standby mode. It is a waste of resources, and it doesn’t get tested until an emergency hits. Thus, the double network connectivity usually gets set up also as a way to double bandwidth. Now if one of the cables breaks, that server merely operates in a degraded mode. The second cable has been passing network traffic already, now it just gets all of it.

A typical data center has many machines. Typical server loads are very low, sometimes in the 1-3% range of overall capacity. Thus, if a machine fails, a data center often has plenty of servers that could absorb the load from the failed server. Figuring out how to cross load services in a data center has been a major topic in the IT world over the past decade. This goes by many names, one of which is grid computing. I’ll use that term myself here. There are several problems any grid system has to solve, but most can be clumped under the term application provisioning. This means getting all of the resources together that a given application requires so that they available on the computation host. These resources include the network and storage connections described above, as well as the executables, data files, licenses, and security permissions required to run the application.

When a host fails, some remote monitoring system needs to act. First, it needs to know that the host has failed. This is typically performed through a heartbeat sensor. This is a simple network communication sent by the computation host saying “I’m still alive.” Cue Mike McCready. When a heartbeat fails, the monitor needs to make sure that the application is up online somewhere as soon as possible. Now, the reason the heartbeat failed might have been because of a problem on the heartbeat network, and the application is actually up and running just fine. An advanced solution is to test the system through some alternative method. In the case of the email server, it may be to connect to the email port and send a sample message. This delays the restart of the application, but may minimize downtime.

Sometimes, two copies of the applications can’t run at the same time. In this case, you have to be sure that the original one is gone. To achieve this, you shut off the original server. This is called “Shoot the other node in the head.” or STONITH. Sometimes the word node is replaced with guy and you get STOGITH. If you do this incorrectly, you may take yourself down, a situation referred to as SMITH. Or you take all servers down, and this is called SEITH. But I digest…

Here’s the part that I am currently trying to decide. If an application depends on a resource, and that resource fails, you can bring the application up on a different server. It will take a non-trivial amount of time (estimate it a minute) to shut down the old instance and bring up the new instance. If, on the other hand, the disconnect is temporary, we can have minimal down time by just waiting for the network to come back up. If someone disconnects a cable by accident, that person can just plug the cable back in. If the network is redundant, removing one cable may result in degraded performance, but it may not.
If the failure is due to the switch being down, just selecting another host connected to the same switch will result in downtime and a situation that is no better than the original. If the problem is the storage host being down, there probably is nothing you can do to recover outside of human or divine intervention.

If a switch goes down but there is another set of hosts on a different switch, you can migrate applications to the new host. But you may end up overloading the new switch. This is referred to as the stampeding herd effect. If the lost switch is degrading performance for all applications dependent on that switch, you best strategy is to migrate a subset of applications to balance the load. After each application is moved, recheck the network traffic to determine if you’ve done enough. Or done too much.

A malfunctioning NIC or switch might manifest in intermittent connectivity.  In this case, you want to get the application off of the original server and on to a new server.  The problem is in distinguishing this from the case where the cable just got unplugged once, and then plugged back in.  From the server’s perspective, the network is gone, and then it is back.  This leads to a a lot of questions. What interval, and how many iterations do you let go by before you decide to bail from that server?  If you have redundancy, does the failing connection impact the user’s experience, or does proper routing ensure that they have seamless connectivity?

Naming in code

int i;

How many times have we seen this? The reason we process this quickly is because we have seen it in code examples in K & R , THinking in C++/Java, or whatever your intro to structured programming was.

Even before that, we saw i used in sequences and series in Algebra. Assuming you took algebra before learning to code, no longer a safe assumption. Actually, I learned Basic before I took Algebra. Anyways, i as an index is very common. If we need multiple we go to j, and then to k.

char c;

Is pretty common too.

float f;

double d;

Probably the defaults, but people here tend to use more explicit names:

float r ; //rate

double a; //accelleration

All these come from the world of math done on a chalk board. If you’ve done any electricity you know:

v= i* (r*r); //twinkle twinkle little start, power equals i squared r.

e=m*(c*c);

a=(b*h)/2;

But it still takes a second to translate:

a=pi*(r*r);

into the more familiar.

a=Пr²

The fact that we can read these so quickly comes from their familiarity. But since we can’t use the Greek letter pi as a variable (at least not without knowing significantly more key bindings in vi than I have at the tips of my finger. I am doing this up in HTML, so I have to go and lookup not only the pi character, but the superscript 2 character as well.

This basic introduction to variable naming should illustrate the point that short names are a benefit if you can quickly translate them to their original intentions. It should not be too hard to show that the contrapositive is also true: if you cannot quickly translate them back to their original meaning, they are not a benefit. I would go so far as to state that they are a hindrance.

How many people can quickly translate the following acronyms:

CINCGARS

CINCPACFLT

RPM

TANSTAAFL

TARFU

RPG

If you have worked in the Army,you know the CINCGARS as the encrypted radio system, although you probably couldn’t state what it meants. If you were in the the Navy, or drove around Pearl Harbor, you would recognize CINCPACFLT as the Commander in CHeif, Pacific Fleet. RPM is rotations per minute, to most people, or the Redhat Package Manager to Linux people. To people who studiedeconomics, TANSTAAFL is short for “There ain;t no such thing as a free lunch.” TARFU is an old WWII slang expression saying that things are two levelsabove SNAFU (Situation normal, all …), having surpassed FUMTU (… more than usual) to “Things are really …” Which is one step below our well loved FUBAR. In all these cases, the acronym is helpful to people already familiar with the term, and a barrier to understanding to people on the outside.

I put RPG last because it illustrates the real problem with abbreviation. We can accept that RedHat chose RPM because it was already popular as an acronym due to the geek cache of rotational dynamics. But RPG is common enough that at least three different realms have used it as an acronym. In the Army, it means rocket propelled grenade. In the mainframe world, it means Report Generator (ver COBOLesque). In the Geek civilian world it means Role Playing Game. These three circles can intersect, cauuing a need to disabiguate the acronym. If you were designing a military role playing game system, you might accept that RPG means the game, and you use the full term for the weapon. If you were building a mainframe program to track the concepts in various Role Playing Game, you might need to use RPG to list the RPGs that refer to RPGs. While you can probably parse the previous sentence, you have to expand each of the acronyms to clarify. Each of these terms resides inside a namespace. Once you know which namespace you are using, you can revert back to the TLA (Three Letter Acronym).

A common pattern of software application development is to have a three tiered design. Say you are writing a program to design and control data centers. Your domain model will contain such objects as Hosts (physical computers), Hostnames, Racks, Ports, Operating Systems, install images, and so forth. Adding a new host to the system may be a multi stage process: specify the physical make up, figure out where to put it, give it a name, put an OS on it, etc. Now you might have differentviews of a host depending on whether you are in the User Interface, business logic, or Data tier of your application. I would tend to create a namespace around each. For instance, in C++:

Web::Host, Business::Host, Data:Host.

All three of these would live in the namespace of the application:

Datacenter::Web::Host, Datacenter::Business::Host, Datacenter::Data:Host.

Note that I chose the specific UI type to name it. Say we alter want a Qt based Application, or a command line interface, the namespaces could reflect that:

Datacenter::Qt::Host Datacenter::CLI::Host

But suppose there is some commonoality between these two. Some might go for a generic term like “Common” But I would caution against that: you want to go as specific as possible. Say the code for validating an IPv6 address is common to all of the UI layers. Put it into a validation package:

Datacenter::Validator::IPv6Address

Note that I move between the terms package and namespace. From a programmatic perspective, they provide the same functionality, they are just different mechanisms depending on the language used (C++ vs Java).

Note that now the Validator package can be used by any stage of the application. You can validate a file load in the Data layer using the same mechanism as the UI layer.

Here’s a rule of thumb:  Make a name as simple as you can.  Use a namespace to avoid name clashes.

Note that the rule states simple, not short. Acronyms require an added levle of translation.  While that may make it easier to type, it does not make it easier to understand.

Here’s another rule of thumb.  The primary focus of the developer should be maintainability.

Not performance.,   Not getting features out of the door as fast as possible, not robustness, not correctness, but maintainability.  Why?  Because it is a crosscutting concern that will support all of those other goals.  The easier it is to maintain your code, the easier it will be to change it for any purpose.   The lead programmer on your team will get hit by a bus, or promoted, or hired away to work at a cool new startup, or decide that he wants a career change, or go on maternity leave, or…and leave someone newly hired to your organization to take over.  Or newly transfered to this group that has never looked at the code before.  You might even be lucky enough to have a change over before the newbie gets the full brunt of the change requests, but it doesn’t matter, because it won’t sink in until the person who wrote is gone…see ya and I wouldn’t want to be ya.

How do you understand a code base?  What tools do you have? One powerful one is refactoring.  Make a copy of the working tree and go crazy renaming methods, extrating blocks from long functions, move things around, etc.  Make the code as pretty as you like.  After a long painful night of it, you will understand the code, in both its pre-and post refactored form.  Then be prepared to throw out the refactored version and work with the code.  But now you can refactor a little at a time.

One of my favorite lines of code from early bproc had a fragment something like this:

req->req.req

THe First req awas a pointer to a messsage that had come in.  The second was the message header, and the third was the message type.  THe current code reads more like this:

message->header.type

Aside from being almost  twice as many characters, the second version is also instantaneously understandable.