Dependency Licenses

The following script, when run on a RPM based system, tells you what licenses your application might trip by linking with other code.  You have to  set $MYEXE to som path to an executable.

for LIB in `ldd $MYEXE | sed -e ‘s!.*=>!!’ -e ‘s! (.*!!’` ; do rpmquery –whatprovides $LIB ; done | grep -v “any package” | sort -u | xargs rpmquery –qf %-30{NAME}:%{LICENSE}”\n

Here is a sample run for perl:

glibc                         :LGPL
perl                          :Artistic or GPL

Note that this is only for the linked files that are registered with RPM.

Dependency Injection in C++

One goal of object oriented programming is to encapsulate functionality within a class.  If one class requires another in order to collaborate on a larger process, the programmer must deal with the wiring up of the two classes.  Much has been written about the correct way to do this in various object oriented languages.  The term “Inversion of Control” and the related concept of “Dependency Injection” have become part of the common language of software development, at least in the Java world, due to projects like Pico/Nano container, The Spring framework, and earlier efforts inside J2EE and ATG Dynamo.  These frameworks make use of the introspection mechanisms in Java to create instances of classes based on demand criteria.


Updated: links and using the code formatter. Changed Class from Single Letter to full names to be clearer and to not conflict with HTML tags.

Continue reading

Interview Question for Distributed Computing

This is an updated version of my interview question,  made non-bproc like.

Using only the following API:

  • printf(…)
  • int get_node_count() // number of compute nodes attached to the head node.  0 means no other nodes
  • int get_current_node()// 0 for the head node, 1-n for the compute nodes.
  • int remote_fork(int node) // like fork, but returns an fd to the child/parent process
  • void send_long_sync(int fd, long value)//send and wait, blocks until receipt
  • long recv_long_sync(int fd)//block until value is available
  • long gettime()

Calculate the average clock skew on the cluster.  Return 0 on success, and -1 on any failures.

Assume that all nodes are up and running.  This is a c subset of c++.  Each of these functions throw an exception upon failure.  rfork has the same semantics as fork:  when it returns, there are two copies of the program running, just on separate machines.  The  next line of code to execute will be the line immediately after the fork on both machines.  However, the returned value is not a process ID, and the parent process  does not need to wait for the remote process to finish: the child process is automatically reaped by init.

Oracle to Postgresql part 1

Here are the steps I am going through to port some code from Oracle PL/SQL to PostgreSQL PLPGSQL.

Here is the first line in the Oracle version

create or replace procedure stats_rollup1_proc is

This becomes

create or replace function stats_rollup1_proc() returns int as $$

DECLARE

Now Postgres is not my fulltime focus, just a part of the overall job. a PG expert could probably do it better.

The things to note here:

  • procedure is not a keyword in plpgsql. Thus function and returns. I suspect I could return a void type, but haven’t looked that hard.
  • Postgres requires the text of a stored procedure to be qutoed. The $$ is a nice way to deal with the requirement.
  • DECLARE is optional in Oracle, but required in postgreSQL

At the end of the function:

end stats_rollup1_proc;

Becomes

return 0;

end /*stats_rollup1_proc;*/

$$ LANGUAGE plpgsql

I like leaving the comment in there to match the original begin, since the functions get long enough that it is hard to track. There is no harm in returning 0, even if we don’t really use it as a return code. The final $$ closes out the one from the start of the function. We have to specify the language used, as this same mechanism can be used for any of the languages embedded inside PostgreSQL. Yes, even Python.

Ok, for some mechanics.

In the Declare section of the oracle code we have:

cursor time_cur(current_time_in date) IS
select distinct sample_time
from VPX_SAMPLE_TIME1
WHERE ROLLUP_COUNTER IS NULL
AND SAMPLE_TIME < current_time_in-1/24
order by 1 asc;
v_time VPX_SAMPLE_TIME1.sample_time%type;

which is later used like this:

open time_cur(v_rollup_start_time);
loop
fetch time_cur into v_time;
exit when time_cur%notfound;

In PostgreSQL these can be inlined like this:

for v_time in
select distinct sample_time
from VPX_SAMPLE_TIME1
WHERE ROLLUP_COUNTER IS NULL
AND SAMPLE_TIME < current_time_in-1/24
order by 1 asc
loop

Although I have not yet figured out how to handle the notfound.

Small Scale High Performance Computing

At the top end of computing there is are the Supercomputers.  At the bottom end there are embedded devices.   In between, there are a wide array of types of computer systems.  Personal computers, workstations and servers are all really just a sliding scale of the same general set of technologies.  These systems are , more and more, the building blocks of the technologies higher up on the scale. Enterprise computing typically involves high-availability and high Input/Output (I/O) based systems.  Scientific and technical computing is similar, but high availability is not as important as performance.  Three of the variables that factor into system design are parallelization, running time and (disk) storage requirements. If a job is small enough that it can run on a single machine in a reasonable amount of time, it is usually best to leave it to do so.  Any speedup you would get in parallelizing the job and distributing the workload is offset by (Amdhals law) the serial portion of the job, the added overhead of parallelization, and the fact that you could run a different job on the other machine.  If your task is parallelizable, but is very storage intensive, you need a high speed disk interconnect.  Nowadays that means fiber channel.

Only if a job takes so long that it makes sense to parallelize, and that job does not require significant access to storage does it make sense to go to a traditional Beowulf cluster.  Although Infiniband does handle the interconnect for both network and storage access, the file systems themselves do not yet handle access by large clusters.

This is the point for which we need a new term:  storage bound, single system jobs that should be run on their own machine. Examples of this abound throughout science, engineering, enterprise, and government.  Potential terms for this are:Small Scale HPC, Single System HPC,  Storage Bound HPC, but none of them really roll of the tongue.

Faking out PAM Authentication

I am working on a server application that uses Pluggable Authentication Modules (PAM) for authentication support.  This application  must run as root. As part of development, people need to log in to this server.  I don’t want to give out the root password of my development machine to people.  My hack was to create a setup in /etc/pam.d/emo-auth that always allows the login to succeed, provided the account exists.  The emo-auth configuration is what the application looks up to authenticate network connections.

$ cat /chroot/etc/pam.d/emo-auth

account   required   pam_permit.so
auth      required pam_permit.so
session   required pam_permit.so

Now people login with root, and any password will allow them to get in.
Since this is only for development, this solution works fine, and does not require any code changes.

Scripting GDB For a stack trace

I got a stack trace generated from an application like this:

Backtrace[0] 0xee8df698 eip 0x87c3b39
Backtrace[1] 0xee8df6e8 eip 0x87c9dbc
Backtrace[2] 0xee8df838 eip 0x85e3f19

And so on. Here’s how I converted it to something useful:

Copy and past the trace into emacs.

Mark the top left corner (ctrl-space)

Move to the last line, right at the end of the eip.

Alt-X kill-rectangle.

This is a great way to do editing by columns in emacs.

Added the words “info symbol to the begging of each line. I did this by first cutting a return chacter, then doing a search and replace, pasting in the cut ‘return’ as the search criteria, andreplaceing it with the ‘return’ followed by “info symbol “. I use this hack a lot to modify the start or end of all the lines in a file.

Once done, I ran

gdb –command=~adyoung/bugs/myapp/backtrace.txt ./myapp core

A couple of gdb tricks

If you run a command with arguments in gdb like this

>run -a -b -c

The next time you go to run, gdb will assume you want the same arguments.   You can work around this by running:

>run —

Which tells the standard options to not process anything after the double dash as an option.

Also, If you have a stack track but not a complete core dump, you can find the line that corresponds with an address (say 0x80808881) in the trace  by using

info symbol 0x80808881