Dependency Licenses

The following script, when run on a RPM based system, tells you what licenses your application might trip by linking with other code.  You have to  set $MYEXE to som path to an executable.

for LIB in `ldd $MYEXE | sed -e ‘s!.*=>!!’ -e ‘s! (.*!!’` ; do rpmquery –whatprovides $LIB ; done | grep -v “any package” | sort -u | xargs rpmquery –qf %-30{NAME}:%{LICENSE}”\n

Here is a sample run for perl:

glibc                         :LGPL
perl                          :Artistic or GPL

Note that this is only for the linked files that are registered with RPM.

Small Scale High Performance Computing

At the top end of computing there is are the Supercomputers.  At the bottom end there are embedded devices.   In between, there are a wide array of types of computer systems.  Personal computers, workstations and servers are all really just a sliding scale of the same general set of technologies.  These systems are , more and more, the building blocks of the technologies higher up on the scale. Enterprise computing typically involves high-availability and high Input/Output (I/O) based systems.  Scientific and technical computing is similar, but high availability is not as important as performance.  Three of the variables that factor into system design are parallelization, running time and (disk) storage requirements. If a job is small enough that it can run on a single machine in a reasonable amount of time, it is usually best to leave it to do so.  Any speedup you would get in parallelizing the job and distributing the workload is offset by (Amdhals law) the serial portion of the job, the added overhead of parallelization, and the fact that you could run a different job on the other machine.  If your task is parallelizable, but is very storage intensive, you need a high speed disk interconnect.  Nowadays that means fiber channel.

Only if a job takes so long that it makes sense to parallelize, and that job does not require significant access to storage does it make sense to go to a traditional Beowulf cluster.  Although Infiniband does handle the interconnect for both network and storage access, the file systems themselves do not yet handle access by large clusters.

This is the point for which we need a new term:  storage bound, single system jobs that should be run on their own machine. Examples of this abound throughout science, engineering, enterprise, and government.  Potential terms for this are:Small Scale HPC, Single System HPC,  Storage Bound HPC, but none of them really roll of the tongue.

Faking out PAM Authentication

I am working on a server application that uses Pluggable Authentication Modules (PAM) for authentication support.  This application  must run as root. As part of development, people need to log in to this server.  I don’t want to give out the root password of my development machine to people.  My hack was to create a setup in /etc/pam.d/emo-auth that always allows the login to succeed, provided the account exists.  The emo-auth configuration is what the application looks up to authenticate network connections.

$ cat /chroot/etc/pam.d/emo-auth

account   required   pam_permit.so
auth      required pam_permit.so
session   required pam_permit.so

Now people login with root, and any password will allow them to get in.
Since this is only for development, this solution works fine, and does not require any code changes.

How to extract from tgz, rpm, and deb files

The most common way to move a bundle of files around in Linux is a combination of tar (tape archive), which appends all of the files together into a single large file,  and gzip, a compression utility.  These are often referred to as tarball. They will have an extension like tar.gz or tgz.  Sometimes bzip2 is used as the comprseeion utility, and they file will end with tar.bz2.

To see the list of files in the  tarball for sage, run:

tar -ztf  sage-2.10.2-linux-ubuntu64-opteron-x86_64-Linux.tar.gz

To extract the files, create a target directory and change to it, thenextract the file from it’s original location

mkdir /tmp/sage

cd /tmp/sage

tar -zxf  ~/Desktop/sage-2.10.2-linux-ubuntu64-opteron-x86_64-Linux.tar.gz

All of the files will be put into the current directory.  There is no rule that says that all the files in a tarball are under on subdirectory, so it really behooves you to do this in a empty directory.  That way you know all of the files you see post extraction are files from the tarball.

Debian uses a package management system called dpkg based on this technology.  The packages will end with .deb, but you can see what Linux thinks the  file type is by using the file command.  Here it is run on automake_1.10+nogfdl-1_all.deb:

 adyoung@adyoung-devd$ file ~/Desktop/automake_1.10+nogfdl-1_all.deb
/home/adyoung/Desktop/automake_1.10+nogfdl-1_all.deb: Debian binary package (format 2.0)

To See the list of included files, run:

dpkg –contents automake_1.10+nogfdl-1_all.deb

And to extract use the –extract command line parameter.  Note that you have to supply the target directory as well.

dpkg –extract automake_1.10+nogfdl-1_all.deb /tmp/deb/

Again, make sure the target directory is empty to avoid intermingling your own files and files from the package.

The letters RPM stand for Redhat Package Manager.  The file extension rpm is used for packages of software designed to be installed on a some distributions of GNU/Linux.   RPM is used on Redhat, SuSE, and distributions based off of these two major distributions, such as Fedora, CentOS, and  OpenSuSE.  RPMs are shipped in a format called cpio. This format has the advantage of allowing longer file names, and providing stroage and compression all in one utility and format.  However, RPMs are not exactly cpio format, and you have to run a converter first, before you can extract the files.  This converter is called rpm2cpio.  It reads the filename in as the first command line parameter, and outputs the cpio file to standard output.  So if you run it without redirecting output, you are going to spew binary data all over your terminal.  Better to redirect it into the cpio utility, with the command line switches -di.  These switches mean extract the files, and build any subdirectories required.  Again, run this in a clean directory:

rpm2cpio ../testware-e.x.p-00000.i386.rpm | cpio -di

Oracle ODBC Setup on Ubuntu (32 on 64)

First of all, I am running a 64 bit Ubuntu 7.04 but I need a 32 Bit Oracle for the application I am using. I have a 32 bit chroot setup. The setup for this is beyond the scope of this article. This version of Ubuntu ships with unixodbc version 2.2.11-13 . There is a symbol, SQLGetPrivateProfileStringW, defined in later versions that the Oracle 11g driver requires. This symbol is not defined in unixodbc version 2.2.11-13. Thus, you have to use the 10.2 Oracle Drivers.

I Downloaded 3 files from The Oracle tech download page for 32bit Linux: the instantclient, SQL Plus, and ODBC packages. I unzipped these in my directory ~/apps/oracle32 Which now looks like this:

adyoung@adyoung-laptop$ pwd
/home/adyoung/apps/oracle32/instantclient_10_2
adyoung@adyoung-laptop$ ls
classes12.jar libocci.so.10.1 libsqora.so.10.1 ojdbc14.jar
genezi libociei.so ODBC_IC_Readme_Linux.html sqlplus
glogin.sql libocijdbc10.so ODBCRelnotesJA.htm
libclntsh.so.10.1 libsqlplusic.so ODBCRelnotesUS.htm
libnnz10.so libsqlplus.so odbc_update_ini.sh

I created an entry in /chroot/etc/odbcinst.ini:

[Oracle 10g ODBC driver]
Description = Oracle ODBC driver for Oracle 10g
Driver = /home/adyoung/apps/oracle32/instantclient_10_2/libsqora.so.10.1
Setup =
FileUsage =
CPTimeout =
CPReuse =

And another in /chroot/etc/odbc.ini

[EMO]
Application Attributes = T
Attributes = W
BatchAutocommitMode = IfAllSuccessful
BindAsFLOAT = F
CloseCursor = F
DisableDPM = F
DisableMTS = T
Driver = Oracle 10g ODBC driver
DSN = EMO
EXECSchemaOpt =
EXECSyntax = T
Failover = T
FailoverDelay = 10
FailoverRetryCount = 10
FetchBufferSize = 64000
ForceWCHAR = F
Lobs = T
Longs = T
MetadataIdDefault = F
QueryTimeout = T
ResultSets = T
ServerName = 10.10.15.15/DRORACLE
SQLGetData extensions = F
Translation DLL =
Translation Option = 0
DisableRULEHint = T
UserID = adyoung
StatementCache=F
CacheBufferSize=20

Once again, DSN and IP Address have been changed to protect the guilty. To test the datasource, run:

sudo dchroot -d LD_LIBRARY_PATH=/home/adyoung/apps/oracle32/instantclient_10_2 DataManagerII

To just test sqlplus connectivity, from inside the chroot, run:

./sqlplus adyoung/adyoung@10.10.15.15/DRORACLE

Note that using the instant client, no TNSNAMES.ORA file is required.

ODBC Setup on Ubuntu/Debian

For the base config tools:

sudo apt-get install unixodbc-bin unixodbc odbcinst1debian1

For Postgres

sudo apt-get install odbc-postgresql

Use the template file provided to setup the odbc driver entries:

sudo odbcinst -i -d -f /usr/share/psqlodbc/odbcinst.ini.template

And this sets up the sample DSNs. (all one line)

 sudo odbcinst -i -s -l  -n adyoung-pg -f /usr/share/doc/odbc-postgresql/examples/odbc.ini.template

Then modify /etc/odbc.ini to suit your DB.

For MSSQL Server and Sybase:

 sudo apt-get install tdsodbc

sudo odbcinst -i -d -f /usr/share/doc/freetds-dev/examples/odbcinst.ini

Unfortunately, this does not have a sample ODBC setup template.