About Adam Young

Once upon a time I was an Army Officer, but that was long ago. Now I work as a Software Engineer. I climb rocks, play saxophone, and spend way too much time in front of a computer.

The structure of a postgres database

Since my work at Penguin was primarily system work, I’ve not had to work with databases much for quite a while. Which is really a pity, since I love the elegance and simplicity of the interface, and the strength of the pre-and-post condition constraints. Anyone who has managed to push my Database hot button knows that my Database of choice is PostgreSQL. It is opensource to the core. The developers treated transactions processing as baseline, no a feature to be added later. It supports nested SQL statements (the S ins SQL is for “Structured”). I am happy to be plugging away at a Postgres Database again.

Part of my current project entails porting the backing store for an application from Oracle to Postgres. There is a nuke script that attempts to remove all objects from a database that have a certain prefix. I will change here to protect the guilty. Let’s pretend that my database is to hold overly emotional music references, and thus all of the tables, views, triggers, and so on are prefixed with ‘EMO’. The oracle script does this:

declare
sqlstmt varchar(200);
varowner varchar(50);
TYPE cur_type IS REF CURSOR;

objectcursor cur_type;

Begin
select USER into varowner from global_name;

open objectcursor for
select ‘drop ‘ || object_type || ‘ ‘ || owner || ‘.’ || object_name
|| decode(object_type,’TABLE’, ‘ cascade constraints’)
from all_objects
where owner = varowner
and (object_name like ‘EMO%’ or object_name like ‘%STAT%PROC’)
and object_type in (‘TABLE’, ‘VIEW’, ‘SEQUENCE’, ‘SYNONYM’,
‘FUNCTION’,’PROCEDURE’, ‘PACKAGE’);

Loop
fetch objectcursor into sqlstmt;
exit when objectcursor%NOTFOUND;
execute immediate sqlstmt;
End Loop;
Close objectcursor;
End;

But tring to port this directly to postgres gets an error right at the beginning:

ERROR: syntax error at or near “varchar” LINE 13: sqlstmt varchar(200);

So I’ll bring this whole thing into a function instead. At then end of my script, I will just drop the function. This has the advantage that I can check the format of my code without having to execute it each time.

When working in an an unfamiliar language (an I’ve never really done PG\PLSQL before) I apply a simple approach. First, I get a “Hello, World” type program working. In this case, I used an article from ONLamp.com and got that to run. Next I strip away everything that is context specific until I get a piece of code that is minimal but syntactically correct. It looks like this:

CREATE OR REPLACE FUNCTION dbemo_nuke_postgres ()
RETURNS integer AS $$

declare

BEGIN
return 1;

End;

$$ LANGUAGE plpgsql;

Note that I modified the prefix so that it won’t Hit the match for EMO% in the like statement I am going to need later. Not sure what would happen if I dropped my function as I ran it. Most likely it would work fine, but why chance it?

Now I start adding in parts of the original function piece by piece. First step is to put in the declares clause. The postgres porting guide says I don’t need a cursor. I quickly learn that this goes between the function declaration and the BEGIN statement:

CREATE OR REPLACE FUNCTION dbemo_nuke_postgres ()
RETURNS integer AS $$

declare
sqlstmt varchar(200);
varowner varchar(50);
objectcursor RECORD;

BEGIN
return 1;

End;

$$ LANGUAGE plpgsql;

This is saved in the file EMO_nuke_postgres_part.sql. I run this from the command prompt:

adyoung@adyoung-laptop$ psql -f EMO_nuke_postgres_part.sql
CREATE FUNCTION

And we’re off. Now to the Oracle specifics.

select USER into varowner from global_name;

This command seems to be asking for info regarding the current user.

select ‘drop ‘ || object_type || ‘ ‘ || owner || ‘.’ || object_name
|| decode(object_type,’TABLE’, ‘ cascade constraints’)
from all_objects
where owner = varowner
and (object_name like ‘VPX%’ or object_name like ‘%STAT%PROC’)
and object_type in (‘TABLE’, ‘VIEW’, ‘SEQUENCE’, ‘SYNONYM’,
‘FUNCTION’,’PROCEDURE’, ‘PACKAGE’)

This code is dynamically creating a drop statement for various objects in the schema. Where is this information stored in postgresql? Let’s take a look:

adyoung@adyoung-laptop$ psql
Welcome to psql 8.2.6, the PostgreSQL interactive terminal.

Type: \copyright for distribution terms
\h for help with SQL commands
\? for help with psql commands
\g or terminate with semicolon to execute query
\q to quit
adyoung=# SELECT * from pg_
pg_aggregate pg_group pg_settings pg_stats
pg_am pg_index pg_shadow pg_stat_sys_indexes
pg_amop pg_indexes pg_shdepend pg_stat_sys_tables
pg_amproc pg_inherits pg_shdescription pg_stat_user_indexes
pg_attrdef pg_language pg_stat_activity pg_stat_user_tables
pg_attribute pg_largeobject pg_stat_all_indexes pg_tables
pg_authid pg_listener pg_stat_all_tables pg_tablespace
pg_auth_members pg_locks pg_stat_database pg_temp_1.
pg_autovacuum pg_namespace pg_statio_all_indexes pg_timezone_abbrevs
pg_cast pg_opclass pg_statio_all_sequences pg_timezone_names
pg_catalog. pg_operator pg_statio_all_tables pg_toast.
pg_class pg_pltemplate pg_statio_sys_indexes pg_trigger
pg_constraint pg_prepared_statements pg_statio_sys_sequences pg_type
pg_conversion pg_prepared_xacts pg_statio_sys_tables pg_user
pg_cursors pg_proc pg_statio_user_indexes pg_views
pg_database pg_rewrite pg_statio_user_sequences
pg_depend pg_roles pg_statio_user_tables
pg_description pg_rules pg_statistic
adyoung=# SELECT * from pg_

I know that metadata is stored in the pg_.* family of objects, so I use statement completion to show me what is there. Nothing is obvious to me yet, so I poke around at pg_tables.

adyoung=# \d pg_tables;
View “pg_catalog.pg_tables”
Column | Type | Modifiers
————-+———+———–
schemaname | name |
tablename | name |
tableowner | name |
tablespace | name |
hasindexes | boolean |
hasrules | boolean |
hastriggers | boolean |
View definition:
SELECT n.nspname AS schemaname, c.relname AS tablename, pg_get_userbyid(c.relowner) AS tableowner, t.spcname AS “tablespace”, c.relhasindex AS hasindexes, c.relhasrules AS hasrules, c.reltriggers > 0 AS hastriggers
FROM pg_class c
LEFT JOIN pg_namespace n ON n.oid = c.relnamespace
LEFT JOIN pg_tablespace t ON t.oid = c.reltablespace
WHERE c.relkind = ‘r’::”char”;

Ah Cha! It is a view into a relation called pg_class that seems to hold all of the metadata. Let’s take a look at that:

adyoung=# \d pg_class
Table “pg_catalog.pg_class”
Column | Type | Modifiers
—————-+———–+———–
relname | name | not null
relnamespace | oid | not null
reltype | oid | not null
relowner | oid | not null
relam | oid | not null
relfilenode | oid | not null
reltablespace | oid | not null
relpages | integer | not null
reltuples | real | not null
reltoastrelid | oid | not null
reltoastidxid | oid | not null
relhasindex | boolean | not null
relisshared | boolean | not null
relkind | “char” | not null
relnatts | smallint | not null
relchecks | smallint | not null
reltriggers | smallint | not null
relukeys | smallint | not null
relfkeys | smallint | not null
relrefs | smallint | not null
relhasoids | boolean | not null
relhaspkey | boolean | not null
relhasrules | boolean | not null
relhassubclass | boolean | not null
relfrozenxid | xid | not null
relacl | aclitem[] |
reloptions | text[] |
Indexes:
“pg_class_oid_index” UNIQUE, btree (oid)
“pg_class_relname_nsp_index” UNIQUE, btree (relname, relnamespace)

Lots of good stuff here. OK, we now have enough raw information to generate some real knowledge. In Postgres, everything is a class. To find out if something is a table you select only those fields with the right type:

adyoung=# SELECT distinct (relkind) from pg_class ;
relkind
———
S
i
r
t
v

Funny that it isn’t type ‘t’ for table, but that seems to be of type toast:

adyoung=# SELECT relname from pg_class where relkind =’t’ limit (3);
relname
—————-
pg_toast_10757
pg_toast_10762
pg_toast_10767
(3 rows)
OK, our original SQL seems to be creating a statment to then execute to drop for each class. WHat collumns it is selecting. Here’s our where clause.:

where owner = varowner
and (object_name like ‘EMO%’ or object_name like ‘%STAT%PROC’)
and object_type in (‘TABLE’, ‘VIEW’, ‘SEQUENCE’, ‘SYNONYM’,
‘FUNCTION’,’PROCEDURE’, ‘PACKAGE’).

Let’s start with the like clause.

adyoung=# SELECT count(relname) from pg_class where relname like ‘vpx%’ or relname like ‘%STAT%PROC’;
count
——-
228

Looks good. But S, i, r, t v doesn’t seem to cover the array we want. I don’t see any functions. Let me chgeck the pg_proc:

adyoung=# \d pg_proc
Table “pg_catalog.pg_proc”
Column | Type | Modifiers
—————-+———–+———–
proname | name | not null
pronamespace | oid | not null
proowner | oid | not null
prolang | oid | not null
proisagg | boolean | not null
prosecdef | boolean | not null
proisstrict | boolean | not null
proretset | boolean | not null
provolatile | “char” | not null
pronargs | smallint | not null
prorettype | oid | not null
proargtypes | oidvector | not null
proallargtypes | oid[] |
proargmodes | “char”[] |
proargnames | text[] |
prosrc | text |
probin | bytea |
proacl | aclitem[] |
Indexes:
“pg_proc_oid_index” UNIQUE, btree (oid)
“pg_proc_proname_args_nsp_index” UNIQUE, btree (proname, proargtypes, pronamespace)

Not a view. OK:

adyoung=# SELECT proname from pg_proc where proname like ‘vcdb_nuke_postgres’ ;
proname
——————–
vcdb_nuke_postgres
vcdb_nuke_postgres
(2 rows)
We know that my stuff is in there.

More on this to come…

dpkg-others

dpkg-query -S `which $1` | cut -d: -f1 | xargs dpkg-query -L

This command finds other files in the same debian package as a given executable.  I put it into a file called dpkg-others.  Example run:

/home/adyoung/bin/dpkg-others ODBCConfig
/.
/usr
/usr/bin
/usr/bin/DataManager
/usr/bin/DataManagerII
/usr/bin/ODBCConfig
/usr/bin/odbctest
/usr/share
/usr/share/doc
/usr/share/doc/unixodbc-bin
/usr/share/doc/unixodbc-bin/copyright
/usr/share/doc/unixodbc-bin/changelog.gz
/usr/share/doc/unixodbc-bin/changelog.Debian.gz
/usr/share/menu
/usr/share/menu/unixodbc-bin
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/ODBCConfig.1.gz

War, Politics, and the Army

I’ve chosen not to make this a political or military blog. There are enough people out there with more experience and more firsthand knowledge on both sides, that I feel like I would be talking out of the wrong orifice.

This is more a rumination, an attempt to place my current wash of thoughts into a coherent structure than a policy statement. I really feel the weight of Socrates advice to admit that I am not wise.  I won’t make many of these.

I think that going into Afghanistan was the right call. Yes, I know that this is not really a surprise, not is it even that controversial a statement. The Taliban government was still in process of taking control of the whole country, and thus had questionable legitimacy at best. Since the government decided to protect the organization claimed responsible for attacking the US on September 11, 2001, I feel we were justified in returning fire.
The war in Iraq is hurting this country. I won’t go through my initial views on the war or my background as an Army officer. Instead, let me state I think the economic burden to this country is more than we can continue to bear. It breaks my heart to say this, because I think it means we are going to do to the Iraqi people what we did to the Montagnards in Viet Nam: pull out and leave them to be massacred. We left Viet Nam because the long term cost was wearing down our Nation. Iraq is less bloody than Viet Nam, but war is far more expensive than it was even forty years ago. If we continue this way, and other nations continue to buy up our debt, we risk losing control of our own country. It seems like General Petraeus and his staff have done an outstanding job this past year in reducing the violence in Iraq. I know that the people on the ground are good soldiers.  Probably better soldiers than I ever was. They may hate being in Iraq, but they want to win the fight.

I remember meeting Viet Nam vets from the VA when I was in high school. These guys were obviously emotionally troubled by what they had seen. One guy said that he still wanted to go back and finish the job. This was 15 years after the last troops had left country. Very few people think that it was in either America’s or Viet Nam’s interest for us to prolong the conflict. But a soldier who has fought, killed, and watched his buddies die for a cause can’t help but either shut down or give his heart to the effort. I think the same is true of our guys in country right now. They want to win in Iraq. They want the Iraqi people to live free of fear, free to raise their kids, and free to rejoin the international community. I want that, too. But I think the overall conflict is much more difficult than that. We have stirred up vast swaths of resentment by being a foreign invader. We have resurrected the ghosts of the Crusades and colonization that bring forth the resistance fighters. Iraq was barely pacified under Hussein. It is going to take a lot more than a Band Aid to fix this sucking chest wound. It would be helpful if we could have truly pulled in an international effort to rebuild Iraq, but we lost that opportunity.

The recent news from Pakistan is troubling.   I think we blew it in Afghanistan by being so quick to rush to war in Iraq.  If the funding that went to the war effort instead went to building infrastructure in Afghanistan (“Thank you for helping us defeat the Soviet Union, here is your pay back.”) we would not be fighting the battles there that we are now. I think the real cost was much higher than anticipated, and I wonder now if we can truly afford it.

One thing that bothers me is that we no longer declare war. The way the constitution was written, it was up to congress to declare war. Not to grant to the President the authority to do so. Deploying troops into a firefight means we are at war. Until World War II, it was expected practice. Yes, we fought the Native Americans, and rebels in many third world countries without such a declaration during this countries history. But if we are going to remove a government from power, we need to state it in no uncertain terms. International Law expects this. I think we have removed one of the checks from the separation of powers in the National Government. Perhaps the language of the constitution should be more a long the lines of: Any deployed forces are allowed to defend themselves. To do anything more than this requires explicit approval from congress. This approval can not be granted a-priori.  But I think the constitution states this, just in the language of 1780.

To the soldiers in Iraq and Afghanistan: Thank you. Especially the burnt out privates that are just looking to get home, and the fathers and mothers that just want to see their kids again. May the remainders of your tours be boring and may you come home whole.

To those that have lost limbs, eyesight, faculties, or to the survivors that have lost loved ones…I don’t have the words to say that will not trivialize your loss. These losses will be remembered and honored.

High Availability and dealing with failures

Time to use my pubic forum to muddle through some design issues I’m struggling to lay straight.

A Data center is made up of several objects: Servers ( computers, usually horizontal), racks(hold the computers), switches(a network device that connects two or more computers together), power sources, and cables (both electric and network cables, to include fiber optic for storage devices). A server in the data center can serve on or more roles: storage host, computation host, administration, or user interface. If an application is hosted in a data cetner, it is usually important enough that it requires some guarantee of availability. This application will resided on a computation host, and require access to other types of hosts. An email server stores the received messages in a storage host, probably connected to the computation host via fiber optics. It receives and sends messages via a network connection that goes to the outside world. It may also talk to a User Interface machine that runs a web server and an application that allows web access to email. If the computation host loses connectivity with either the public network or the storage host, it cannot process mail. If the web server loses connectivity to the mail server, certain users cannot access their mail.

There are many reasons that connectivity can fail. The major links in the chain are: OS failure, Network interface card (NIC) failure, bad cable, disconnected cable, bad switch, unplugged switch, switch turned off. Once you pass the switch, the same set of potential problems exist on to the other host. To increase reliability, a network will often have two switches, and each server will have two NICs, one plugged into each switch. The same set up goes for storage, although different technologies are used. As a general rule, you don’t want to have things running in hot standby mode. It is a waste of resources, and it doesn’t get tested until an emergency hits. Thus, the double network connectivity usually gets set up also as a way to double bandwidth. Now if one of the cables breaks, that server merely operates in a degraded mode. The second cable has been passing network traffic already, now it just gets all of it.

A typical data center has many machines. Typical server loads are very low, sometimes in the 1-3% range of overall capacity. Thus, if a machine fails, a data center often has plenty of servers that could absorb the load from the failed server. Figuring out how to cross load services in a data center has been a major topic in the IT world over the past decade. This goes by many names, one of which is grid computing. I’ll use that term myself here. There are several problems any grid system has to solve, but most can be clumped under the term application provisioning. This means getting all of the resources together that a given application requires so that they available on the computation host. These resources include the network and storage connections described above, as well as the executables, data files, licenses, and security permissions required to run the application.

When a host fails, some remote monitoring system needs to act. First, it needs to know that the host has failed. This is typically performed through a heartbeat sensor. This is a simple network communication sent by the computation host saying “I’m still alive.” Cue Mike McCready. When a heartbeat fails, the monitor needs to make sure that the application is up online somewhere as soon as possible. Now, the reason the heartbeat failed might have been because of a problem on the heartbeat network, and the application is actually up and running just fine. An advanced solution is to test the system through some alternative method. In the case of the email server, it may be to connect to the email port and send a sample message. This delays the restart of the application, but may minimize downtime.

Sometimes, two copies of the applications can’t run at the same time. In this case, you have to be sure that the original one is gone. To achieve this, you shut off the original server. This is called “Shoot the other node in the head.” or STONITH. Sometimes the word node is replaced with guy and you get STOGITH. If you do this incorrectly, you may take yourself down, a situation referred to as SMITH. Or you take all servers down, and this is called SEITH. But I digest…

Here’s the part that I am currently trying to decide. If an application depends on a resource, and that resource fails, you can bring the application up on a different server. It will take a non-trivial amount of time (estimate it a minute) to shut down the old instance and bring up the new instance. If, on the other hand, the disconnect is temporary, we can have minimal down time by just waiting for the network to come back up. If someone disconnects a cable by accident, that person can just plug the cable back in. If the network is redundant, removing one cable may result in degraded performance, but it may not.
If the failure is due to the switch being down, just selecting another host connected to the same switch will result in downtime and a situation that is no better than the original. If the problem is the storage host being down, there probably is nothing you can do to recover outside of human or divine intervention.

If a switch goes down but there is another set of hosts on a different switch, you can migrate applications to the new host. But you may end up overloading the new switch. This is referred to as the stampeding herd effect. If the lost switch is degrading performance for all applications dependent on that switch, you best strategy is to migrate a subset of applications to balance the load. After each application is moved, recheck the network traffic to determine if you’ve done enough. Or done too much.

A malfunctioning NIC or switch might manifest in intermittent connectivity.  In this case, you want to get the application off of the original server and on to a new server.  The problem is in distinguishing this from the case where the cable just got unplugged once, and then plugged back in.  From the server’s perspective, the network is gone, and then it is back.  This leads to a a lot of questions. What interval, and how many iterations do you let go by before you decide to bail from that server?  If you have redundancy, does the failing connection impact the user’s experience, or does proper routing ensure that they have seamless connectivity?

Naming in code

int i;

How many times have we seen this? The reason we process this quickly is because we have seen it in code examples in K & R , THinking in C++/Java, or whatever your intro to structured programming was.

Even before that, we saw i used in sequences and series in Algebra. Assuming you took algebra before learning to code, no longer a safe assumption. Actually, I learned Basic before I took Algebra. Anyways, i as an index is very common. If we need multiple we go to j, and then to k.

char c;

Is pretty common too.

float f;

double d;

Probably the defaults, but people here tend to use more explicit names:

float r ; //rate

double a; //accelleration

All these come from the world of math done on a chalk board. If you’ve done any electricity you know:

v= i* (r*r); //twinkle twinkle little start, power equals i squared r.

e=m*(c*c);

a=(b*h)/2;

But it still takes a second to translate:

a=pi*(r*r);

into the more familiar.

a=Пr²

The fact that we can read these so quickly comes from their familiarity. But since we can’t use the Greek letter pi as a variable (at least not without knowing significantly more key bindings in vi than I have at the tips of my finger. I am doing this up in HTML, so I have to go and lookup not only the pi character, but the superscript 2 character as well.

This basic introduction to variable naming should illustrate the point that short names are a benefit if you can quickly translate them to their original intentions. It should not be too hard to show that the contrapositive is also true: if you cannot quickly translate them back to their original meaning, they are not a benefit. I would go so far as to state that they are a hindrance.

How many people can quickly translate the following acronyms:

CINCGARS

CINCPACFLT

RPM

TANSTAAFL

TARFU

RPG

If you have worked in the Army,you know the CINCGARS as the encrypted radio system, although you probably couldn’t state what it meants. If you were in the the Navy, or drove around Pearl Harbor, you would recognize CINCPACFLT as the Commander in CHeif, Pacific Fleet. RPM is rotations per minute, to most people, or the Redhat Package Manager to Linux people. To people who studiedeconomics, TANSTAAFL is short for “There ain;t no such thing as a free lunch.” TARFU is an old WWII slang expression saying that things are two levelsabove SNAFU (Situation normal, all …), having surpassed FUMTU (… more than usual) to “Things are really …” Which is one step below our well loved FUBAR. In all these cases, the acronym is helpful to people already familiar with the term, and a barrier to understanding to people on the outside.

I put RPG last because it illustrates the real problem with abbreviation. We can accept that RedHat chose RPM because it was already popular as an acronym due to the geek cache of rotational dynamics. But RPG is common enough that at least three different realms have used it as an acronym. In the Army, it means rocket propelled grenade. In the mainframe world, it means Report Generator (ver COBOLesque). In the Geek civilian world it means Role Playing Game. These three circles can intersect, cauuing a need to disabiguate the acronym. If you were designing a military role playing game system, you might accept that RPG means the game, and you use the full term for the weapon. If you were building a mainframe program to track the concepts in various Role Playing Game, you might need to use RPG to list the RPGs that refer to RPGs. While you can probably parse the previous sentence, you have to expand each of the acronyms to clarify. Each of these terms resides inside a namespace. Once you know which namespace you are using, you can revert back to the TLA (Three Letter Acronym).

A common pattern of software application development is to have a three tiered design. Say you are writing a program to design and control data centers. Your domain model will contain such objects as Hosts (physical computers), Hostnames, Racks, Ports, Operating Systems, install images, and so forth. Adding a new host to the system may be a multi stage process: specify the physical make up, figure out where to put it, give it a name, put an OS on it, etc. Now you might have differentviews of a host depending on whether you are in the User Interface, business logic, or Data tier of your application. I would tend to create a namespace around each. For instance, in C++:

Web::Host, Business::Host, Data:Host.

All three of these would live in the namespace of the application:

Datacenter::Web::Host, Datacenter::Business::Host, Datacenter::Data:Host.

Note that I chose the specific UI type to name it. Say we alter want a Qt based Application, or a command line interface, the namespaces could reflect that:

Datacenter::Qt::Host Datacenter::CLI::Host

But suppose there is some commonoality between these two. Some might go for a generic term like “Common” But I would caution against that: you want to go as specific as possible. Say the code for validating an IPv6 address is common to all of the UI layers. Put it into a validation package:

Datacenter::Validator::IPv6Address

Note that I move between the terms package and namespace. From a programmatic perspective, they provide the same functionality, they are just different mechanisms depending on the language used (C++ vs Java).

Note that now the Validator package can be used by any stage of the application. You can validate a file load in the Data layer using the same mechanism as the UI layer.

Here’s a rule of thumb:  Make a name as simple as you can.  Use a namespace to avoid name clashes.

Note that the rule states simple, not short. Acronyms require an added levle of translation.  While that may make it easier to type, it does not make it easier to understand.

Here’s another rule of thumb.  The primary focus of the developer should be maintainability.

Not performance.,   Not getting features out of the door as fast as possible, not robustness, not correctness, but maintainability.  Why?  Because it is a crosscutting concern that will support all of those other goals.  The easier it is to maintain your code, the easier it will be to change it for any purpose.   The lead programmer on your team will get hit by a bus, or promoted, or hired away to work at a cool new startup, or decide that he wants a career change, or go on maternity leave, or…and leave someone newly hired to your organization to take over.  Or newly transfered to this group that has never looked at the code before.  You might even be lucky enough to have a change over before the newbie gets the full brunt of the change requests, but it doesn’t matter, because it won’t sink in until the person who wrote is gone…see ya and I wouldn’t want to be ya.

How do you understand a code base?  What tools do you have? One powerful one is refactoring.  Make a copy of the working tree and go crazy renaming methods, extrating blocks from long functions, move things around, etc.  Make the code as pretty as you like.  After a long painful night of it, you will understand the code, in both its pre-and post refactored form.  Then be prepared to throw out the refactored version and work with the code.  But now you can refactor a little at a time.

One of my favorite lines of code from early bproc had a fragment something like this:

req->req.req

THe First req awas a pointer to a messsage that had come in.  The second was the message header, and the third was the message type.  THe current code reads more like this:

message->header.type

Aside from being almost  twice as many characters, the second version is also instantaneously understandable.

Rethink Web Apps in the light of Generics

The following is some musings on Templates and generics. I am not advocating programming this way, just trying to think through these issues in a framework I’ve used before.

The Struts framework dominated web application development in the Java world until the introduction of Java Server Faces. Since Java 1.4 and earlier provide great introspection capabilities and no generics, Struts did heavy runtime introspection to initialize the application and interpret URLS into the paths of the Java classes that should process them. The heart of form processing in Struts is the Action and ActionForm classes. Bascially, struts uses property names to map HTML parameters to bean ‘set’ methods. These can be nested. For instance

orderForm.billingAddress.phoneNumber.areaCode=415

would map to

OrderForm form = session.getBean(“order”);

form.getAddress().getBillingAddress().getPhoneNumber().setAreaCode(“415”)

albeit executed via reflection. Once the form object is populated it is passed to an Action Object. The Action object is an instance of the Flyweight Design Pattern. A Single Action instance can process all of the forms submitted at a time. Note that an Action object is not a singleton: Two instances of the same subclass of Action can be instantiated and have state assigned from the configuration file. They are only “Stateless” with regard to the HttpRequest, HttpResponse, and HttpSessions. Here is the API:

public class ExampleAction extends Action
{

public ActionForward execute(
ActionMapping mapping,
ActionForm form,
HttpServletRequest request,
HttpServletResponse response) throws Exception{}

}

The ActionMapping allows you to get specifics about how the action was configured. Mostly it is used to figure out where to go next. The Action form is either down-cast to the correct type (throwing an exception if it is the wrong type) or ignored if there is no submitted data. The mapping class contains a list of forwards. These forwards are kept in a map. The action is going to return a forward configured like this:

return mapping.findForward("success");

We have here several instances of Runtime processing about decisions that could be made at compile time. The code that makes these decisions could be optimized by the compiler. Let’s start with the mapping. A map lookup based on a string is fairly expensive: Either has the string and do a hash-table lookup, or do a series of string compares. Really, what we want is code like:

template <typename Mapping, typename Form>

ActionForward execute(Form form, Mapping map, HttpRequest req, HttResponse resp) {

return map.forwardSuccess;

}

The java Bean API is based on a Java d design decision that everything class type variable a reference. To ensure that you don’t get a Null pointer, you have to fetch something through a ‘get’ method that often acts as a Lazy accessor. This is not necessary for most uses. C++ has the alternative of objects owning the objects they create. Thus the map object, even if created on the stack, can own the forward object. To transform the above in C++, it would be more like:

ActionForward& execute(Form& form, Mapping& map, HttpRequest& req, HttResponse& resp) {

return map.forwardSuccess;

}

The ‘&’ ensures that there is no copying of the objects through the function call. Because the map and returned action forward object both live outside the scope of this function, you can return a reference to the forward. This is true even if the map was originally created on the stack. We’ll come back to that in a little bit.

One task that actions often have to perform is validate a form an return to the previous screen if the form is invalid. In our above code, the logic would be:

ActionForward& execute(Form& form, Mapping& map, HttpRequest& req, HttResponse& resp) {

if (form.isValid() ){

return map.forwardSuccess;

} else {

return map.forwardPrevious;

}

Note that with templates, you could go crazy and have the forward and previous actions, as well as the isValid function name all passed in as template parameters. This would be the ulitmate in flexibility. The question here is the middle ground of settling on an acceptable implied interface.

Let’s take the case of the bean setters being called.

orderForm.billingAddress.phoneNumber.areaCode=415

Code like this is usually generated from a struts tag embedded inside a JSP. Here, code generation (JSP compilation) is generating late bound code. Even without the Generics capabilities of Java 1.5 and later, this can be bypassed, and in fact “pure” jsp processing does that by doinbg the bean API translation in line. However, struts does not have access to the JSP grammer and could not define rules for translation at that point. It only had the weaker rules of the JSP tag API, and thus had to be content with run time binding.

When the servlet API calls the struts servlet onPost method, the struts servel iterates through all of the parameters performs a binding on each one. The cope of the Form is defined by the action mapping that thje JSP refers to in the <FORM> tag. The code in the JSP is something like this:

<form action=”myurl.com/struts/thisform/sumbit.do”>

<input type=”text” name=”billingAddress.phoneNumber.areaCode”>

The URL will be translated to the mapping in the struts config file (minus the .do). The form object is a Bean, usually Session scoped, defined in the same file. Once that is resolved, the rest of the lookup is done using a util function for properties. To convert this to Template style, the first lookup would have to remain the same. URLS can be either absolute or relative. However, we can use functors for the second part: Validating and setting property values. We know what values will be used on a given page. A functor can handle the validation and the setting.
Map <string, string> properties;

Map <string, string>:: data_type data;

Map <string, string>:: iterator_type iterator;

for(iterator = properties.begin(); iterator != properties.end() iterator++){

data = *iterator;

propName,propValue in request

try{

mapping.propertyMap.get(data.first).set(form,data.second));

}catch(ActionException& ae){

actionErrors.add(ae);

}

}

Some people question the efficiency of using Exceptions for handling code of this type. I’ve addressed that in an earlier blog post. Basically, entering a try block generated by g++ is free. The only cost is when the exception is thrown: A new object has to be created and the stack unwound to the point where that exception is handled. The cheapest way to write this code would be to return an error code from action errors, but that propertyMap.get(“”).set(form,””). But then the setter has no way to customize the error message. We could pass back a string or null to signify error or success, but now we are already allocating objects, and adding in an additional test done for every success condition.

If we were to peek beneath the covers of struts into how the code was written for the ActionMapping we would see something like:

getAction().execute(request, response, form,this);

This is where the template parameter comes into play:

template<typename Form, typename Action> ActionMapping : public Mapping{

Mapping(Action& action){

}

process(HttpRequest req HttpResponse resp){

action.execute(request, response, getForm(req),this);

}

}

We still need to descend from a common Mapping baseclass so that the Action and Mapping classes don’t have a circular dependency. But now we have type safety in the call into the Action. Also, the construction of the mapping depends on a specific class. Initializing this code now has to be done in C++. Both the Mapping and the Action objects should be scoped global to the application. The Form can either be request or session scoped. Thus the construction of this complex set of relationships will require a lot of thought.

This article could go on and on. I’m going to add one more related concept and then declare victory and post it.

The difference between a Form object and the domain model is that the form object shuld be able to work with half formed, invalid data. Quite often, the form will contain builders for objects in the domain model. A Trivial example is a social security number. This is of the format 000-11-2222. Often, the UI will put each of these fields into a separate text box. The builder will then accept each field, validate their length and that it only has didgits, and produce a SocialSecurityNumber.

One way that this could be made generic is to provide a Template class that represents a validator for a regular expression. The SocialSecurity Number would be one instance, but each of the sub fields would be instances as well. Someething like this:

typedef ValidatedString<“\d\d\d-\d\d-\d\d\d\d> SocialSecurityNumber;

This is not valid C++, as the string cannot be passed to a type. It might be OK in Java where Regular Expressions are processed at run time.

Doing Unspeakable things with Templates

The C++ Template language is Turing complete.  That means it should easily be able to build an Acceptor for a Regular expression.  Yes, this is headed where you think it is. Get out while you still can.

Still with me?  Too bad for you:

#include <string>
using std::string;
namespace acceptor{
template <typename Next> class Expression{
    public:
    static bool accept(string& s){
        return Next::accept(s, s.begin());
    }
};
class EndType {
    public:
    static bool accept(string& s, string::iterator itr){
       return(itr == s.end());
    }
};
class NoOp {
    public:
    static bool accept(string& s, string::iterator itr){
        return true;
    }
};
template <char valid,typename Next = NoOp > class Char{
    public:
    static bool accept(string& s, string::iterator  itr){
        return (valid == *itr) && Next::accept(s,++itr);
    }
};
template <char first,char last,typename Next = NoOp >
class RangeAcceptor{
    public:
    static bool accept(string& s, string::iterator itr){
        return ((*itr >= first)
            && (*itr <= last)
            && Next::accept(s,++itr));
    }
};
template <int count, typename Acceptor ,typename Next = NoOp>
class Count{
    public:
    static bool accept(string& s, string::iterator itr){
        int i;
        for ( i = 0; i < count ; ++i){
            if (Acceptor::accept(s,itr)){
                ++itr;
            }else{
                return false;
            }
        }
        return Next::accept(s, itr);
    }
};
template <typename First, typename Second >
class Or {
    public:
    static bool accept(string& s, string::iterator itr){
        if ( First::accept(s,itr)) {
            return true;
        } else {
            return Second::accept(s,itr);
        }
    }
};
template <typename First, typename Second >
class And {
public:
    static bool accept(string& s, string::iterator itr){
        return (First::accept(s,itr) && Second::accept(s,itr));
    }
};
template <typename Next = NoOp >
class Digit: public RangeAcceptor<'0','9',Next>{};
template <typename Next>
class LowerLetter : public RangeAcceptor<'a','z',Next>{};
template <typename Next>
class UpperLetter : public RangeAcceptor<'A','Z',Next>{};
template <typename Next = NoOp >
class Letter : public Or < LowerLetter<NoOp> , UpperLetter<NoOp> > {};
};

How would you use this?

void testDigit(){
Expression<
Digit <
Digit <
Digit <
Char  < '-' ,
Digit <
Digit <
Char  < '-' ,
Digit <
Digit <
Digit <
Digit <
EndType
> > > > > > > > > > > >ssn;
string s("000-22-9999");
assertTrue(s, ssn.accept(s));
s = "abcdefghij";
assertFalse(s, ssn.accept(s));
};

Not shown:  Implementation of the Kleene Star, as that would require a writing a greedy algorithm, something I have no desire to do right now.

Dynamic Frequency Assignment Protocol

A book about the war in Iraq had an brief passage about an Army Unit that needed to stop, dismount, and reset their frequency because they were getting stepped on by a marine Corp unit using the same freq. The Marines were in contact and were not about to change Frequencies. This lead me to thinking about a way to solve this problem via network protocols and public/private key exchanges.

Each unit has a radio. These radios already can encrypt traffic and frequency hop. In fact, frequency hopping would have prevented the unit from getting stepped on. But let’s assume that freq hopping is out of the picture for now. Instead, we want to set up a protocol where a unit can reassemble on the network without having to expose itself and meet face to face.

1. Each machine would get a private key. This allows it to be uniquely identified on the network.

2. Each machine gets a list of public keys and their assigned units (a heads up display could then show who was talking, removing the need for “Black 8 this is Black 6 over.”)

3. A set of frequencies would be set aside for meta-data commo. I’ll call this the MDF. The MDF would be shared by all units within a theater of operations, and would be redundant. Just like name servers, a radio would get a list of MDFs to try. Thus four MDFs could be assigned, and four units would get a different MDF to use as it’s primary.

4. The radio would monitor both its main frequencies (the Radios I used when I were could monitor two, I assume the new ones can do better) and the MDF. MDF traffic would all be binary data. Listening to it would be like listening to a 300baud modem.

5. A “Lost” Unit would broadcast a connection request on the metadata frequency. Any other unit in the area could answer.

6. Higher would send down the new frequency using the lost units public key.

7. The lost unit’s radio would set the new frequency and drive on.

One glaring weakness is that the MDF itself could be jammed. One faulty radio could crap-flood the frequency.

It also adds overhead to the whole system by removing frequencies that could otherwise be used. An optional interactive authentication step could provide for the soldier entering a pin or password if there is some question of the radio being compromised. Two valid response could be provided, one that means , “I’m me” and one that means “I’m me and doing this under duress.”

Note that none of this would prevent manual resets, just provide an additional capability for automated resets.

Of course, this is yet more electronics subject to the beating an infantryman gives it.

Update:  Of course the Military is way ahead of this.

http://en.wikipedia.org/wiki/AN/PRC-148

And

http://en.wikipedia.org/wiki/Joint_Tactical_Radio_System

A new outlook on some old activities

The time from Dec 24th until January 2nd was spent in New Hampshire with my family.  It was a real vacation, something I have not had in a long time.  Aside from the time off, I got outside for a few winter sports:  Cross Country Skiing, Sledding, and Snowshoeing.

Growing up, my family was all about downhill skiing, but the cost, plus the time away from my family meant that I had no drive to go.  Even with all of the great snow this past week.

For cross country, we went out the East Branch of the Pemigewasset  river by Lincol Woods.  The snow was so hard packed that many people were out hiking using just boots and simple crampons.

My folks had picked up a pair of snow shoes.  My wife and I tried them out and I fell instantly in love with the sport.  I have always loved the New England woods, especially the ability to wander aimlessly.  Snowshoeing opens up the woods in winter time.  With the snow covering all the entangling underbrush that hadn’t dies off in the fall, and the metal teeth biting in allowing you to navigate the steepest slope, the woods are open to a degree you don’t find any other time. Yes, snowshoeing is slower than hiking or skiing, but the mobility is amazing.  So now I have to do research to buy myself and my wife a pair.

My Dad has a tractor and an excavator, and I swear he never stops playing on them when we go to NH.  This year, he ignored our entreties not to tear up the meadow and used the excavator to build a sledding run.  The meadow was always a tough sled run, as the long line also was the crest of the hill, and it constantly wanted to throw you off into the flats or the underbrush.  He built a track right down the crest, augmented with a little shovel work.  The packed down snow soon became Ice.  A little Silicone lubricant applied to the bottom of the tubes made for one hell of a fast sled run.