Hacking the Palm Pre from Fedora

My work machine is a Fedora 11 (F11) X86_64 system. The palm development SDK is distributed as a series of .deb packages, specifically targeted at an 32 bit Ubuntu 8 system. While I have the advantage of having a 32bit Debian system at home, so I was able to run through the setup process for development, ideally I would be able to attach to and control the Pre from my work machine.

The first step is to download the .deb files onto the F11 machine. I actually only needed the novacom deb, which in my case is novacom_1.0.38_i386.deb. Deb files are accessable using ar (happy talk like a pirate day!).

In a new and empty directory, run
ar -vxf ~/novacom_1.0.38_i386.deb

And you will see the three contained files:

control.tar.gz data.tar.gz debian-binary

Extract the data file using tar

tar -zxf data.tar.g

This will add a usr directory with the binaries in

usr/local/bin/novacom{d}

Novacom is a two piece effort: a daemon an a client. First make sure you can run the daemon.

First, lets see the file type:

file novacomd

novacomd: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.8, stripped

Let’s see what libraries it requires:

ldd novacomd
linux-gate.so.1 => (0x008f1000)
libpthread.so.0 => /lib/libpthread.so.0 (0x0078a000)
libusb-0.1.so.4 => not found
libc.so.6 => /lib/libc.so.6 (0x00110000)
/lib/ld-linux.so.2 (0x006ae000)

Note that the USB library is missing. I have it installed on my system, but only the 64 bit version. To get the 32 bit version, first, figure out what the 32 bit vversion is named.

yum search libusb

libusb.i586

And install

sudo yum install libusb.i586

F11 and the RHEL based approach for running 32bit apps on 64 makes this fairly easy to do. Unlike Debian based system which pretty much require you building a chroot if you are going to run a significant amount of 32 bit binaries, Red Hat based systems put 64 bit libraries into /usr/lib64 and 32 bit libraries int /usr/lib, so they don’t conflict. Now lddd shows we have everything:

ldd novacomd
linux-gate.so.1 => (0x00262000)
libpthread.so.0 => /lib/libpthread.so.0 (0x008a5000)
libusb-0.1.so.4 => /usr/lib/libusb-0.1.so.4 (0x00770000)
libc.so.6 => /lib/libc.so.6 (0x00263000)
/lib/ld-linux.so.2 (0x006ae00)

And we can now run this. Since it is talking straight to hardware, it insists on running as root:

./novacomd
[2009/9/22 11:40:48] novacomd version novacomd-62 starting…
[2009/9/22 11:40:48] novacomctl socket ready to accept
[2009/9/22 11:40:48] need to run as super user to access usb

so:

sudo ./novacomd
[2009/9/22 11:41:11] novacomd version novacomd-62 starting…
[2009/9/22 11:41:11] novacomctl socket ready to accept
[2009/9/22 11:41:11] sending rst
[2009/9/22 11:41:11] sending rst
[2009/9/22 11:41:11] sending rst
[2009/9/22 11:41:11] going online
[2009/9/22 11:41:11] novacom_register_device:188: dev ‘e851588c804e8caa722490a0314ce9782dd4d9a4’ via usb type castle-linux

No we turn our attention to the client piece.

file novacom
novacom: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.8, stripped
[ayoung@ayoung novacom]$ ldd novacom
linux-gate.so.1 =>  (0x00173000)
libc.so.6 => /lib/libc.so.6 (0x006d2000)
/lib/ld-linux.so.2 (0x006ae000)

So we are ready to run.  There is no novaterm in this deb.  Instead, you run novacom in terminal mode.  A little noted line in that I will make big here is:

./novacom $* -t open tty://0

Dogfooding

When I was contracting at Sun, someone sent out a memo using the phrase “eat our own dog food.”  Scott McNeally sent out a response to the effect that Sun did not make dog food, “We Fly our own airplanes.”

In the software world, the phrase has become so used that it has been verbed; If you work at a software company that uses its own products in house,  you are Dogfooding.

Since so much of my professional work has been on Red Hat Enterprise, I’ve tended to run Debian based systems for my desktop, to keep abreast of what is happening in both worlds.  At Red Hat, I’ve finally had reason to run Fedora as my desktop machine, and have been enjoying the experience.  The vast majority of the Software I run now is distributed by the company I work for, and is Open Source.  It really is an amazing feeling. I am now running Fedora 11 not only on my work laptop, but my wife gave me permission to blow away the Windows install on her eeepc and install it there as well.  Fedora makes a great Netbook OS.

However, one tenant of Software Development is to develop on the platform on which you are going to ship.  For Red Hat, that is RHEL5, and so I need access to a RHEL5 install, and in fact need both 32 and 64 bit, since the world has not completely moved to 64 bit yet.  I’ve used virtual machines in the past, but always from VMware.  Now I am running QEMU/KVM on my Laptop.  Like most things Linux-y, the command prompt method of controlling the virtual machine subsystem is a first class citizen:  I don’t need a visual console to start up a VM.  I realize this is old hat to some people, but It is new to me, and I am enjoying it.

That is the wonderful thing about the Open Source development model:  you very quickly take ownership of the software that is essential to you.  Whenever a user becomes a contributor, that software is no longer just something out there.  It has become personal.

Anyways. As I Fly the Red Hat virtualization airplane, I’ve learned a few things.  The GUI, Virtual Machine Manger, is great for getting over the learning curve.  The command line is virsh.  These tools are a  part of the libvirt project.  There is a command to start Qemu based vms directly, but his seems to bypass the libvirt infrastructure.  Rnning qemu-kvm allowed me to start a VM saved in /var/lib/libvirt/images, but was not able to talk to the kvm subsystem.  One thing that threw me was that connecting to the virtual shell and running the list command did not show my virtual machine;  by default, that only shows running virtual machines, and you need to add the –all option to see non-running VMs..important if you want to only run them occasionally as I do.  To connect to the system, run:

sudo virsh -c qemu:///system

There is also another URL qemu:///session that I am not yet familiar with.

Working with VMware, I was comfortable with the split of the info into a vmdk and vmx files for binary data and configuration information.  IN KVM/Qemu/libvirt land, the equivalent of the VMDK is  a boot image.  This seems right to me, in keeping with the fearless Unix way of not inventing a new technology if an existing one makes sense.  The Analogue of the vmx file is in /etc/libvirt/qemu/.

One thing I would like to get set up is bridged networking for my VMs to the corporate lan.  The base install takes the conservative view that the network should be confined to the local system.  I’ve seen some write ups to getting TAP interfaces set up to allow your virtual NICs to get packets to and from the physical NICs, but haven’t done that yet.  The configuration for the host local network can be viewed from the virtual machine manager, and it shows the range of DHCP address given out to the hosts.  It contains a wizard for adding new networks, but I am not sure if the VMware paradigm of a bridged network maps cleanly to the Linux view…I suspect not.  I see under advanced options when creating the VM that I can set the Network to bridged, but it doesn’t seem to find my DHCP server to PXE boot.  As an aside, I’d like to understand how this operates in an Ipv6 environment, as Uncle Sam is now dictating IPv6 for all new Software purchases.    So many things to learn!

Reading the links from a webpage

I needed to see the set of RPMs in a YUM repository. I wanted to do this as part of a larger script. To do this, I fetched the page via wget, and then applied an xsl transform on it using the command line tool xsltproc.

Here is how I called it:

wget -q -O – http://spacewalk.redhat.com/yum/0.5/Fedora/10/x86_64/os/Packages/ | xsltproc –html showhrefs.xslt –

And here is the xslt file showrefs.xslt

<?xml version=”1.0″ encoding=”UTF-8″?>
<xsl:stylesheet xmlns:xsl=”http://www.w3.org/1999/XSL/Transform” version=”1.0″>
<xsl:output method=”xml” indent=”yes”/>

<!–  shut off the default matchin rule –>
<xsl:template match=”text()” />

<!– print the href value of the hyperlinks –>
<xsl:template match=”a”>
<xsl:value-of select=”@href” />
<xsl:text >
</xsl:text>

</xsl:template>

</xsl:stylesheet>

Reading HTML docs on an ebook reader

The Calibre project is essential to me making full use of my SOny ebook reader. I recently wanted to pull down the HTML documentation for Red Hat Satellite server and load it to the reader. It was this simple:

wget -rL http://www.redhat.com/docs/manuals/satellite/Red_Hat_Network_Satellite-5.1.0/html/Installation_Guide/index.html

html2epub www.redhat.com/docs/manuals/satellite/Red_Hat_Network_Satellite-5.1.0/html/Installation_Guide/index.html

I probably should have used the -t option to set the title, as I had to rename the file from index.epub.

Transcribing Music using mpg123

A long time tradition for learning how to play like the great soloists is to learn their solos from recordings.  Ideally, you would listen to the recording slowed down.  However, on analog equipment, when you slow down the recording, you have to adjust the pitch back up, since you have effective dropped the frequency of the recorded notes.  With mp3 files, you have the advantage of having sampled the original recording into discrete chucks called frames.  Thus to slow down a recording, you can just tell your music player to double the number of times it plays each frame, and you have slowed down the playback without affecting the pitch.

Say you are a harmonica player and you want to learn John Popper’s solo at the beginning of Runaround.  You can do so by playing it back in mpg123.  Here’s the approach I would use.  Start by playing the song from the beginning:

mpg123 runaround.mp3

Now find the start of the solo.  To skip ahead, use the -k option to skip frames.  Try at about 1000.

mpg123 -k 1000 runaround.mp3

Now use a binary search:  If you hear the solo playing, go backwards, if you don’t go forwards.  At 1000 I hear it playing, so I go back to 500, then 250. At 250 I just hear the intro guitar, so I jump forward by 125 to 325 and so on until I figure out that the solo starts roughly at 270.  Now I slow down the playing by half:

mpg123 -h2 -k 1000 runaround.mp3

It is distorted as all hell, but I can make out the notes, and more importantly, match it on a piano, harmonica or saxophone.  I can slow it down even further with

mpg123 -h3 -k 1000 runaround.mp3

Which is as far as I would want to go.  Note that for older recordings, where the solo may be exclusively on the left or right track, you could add the -0 or -1 options to shut out more of the background noise.

As you progress, you want to move forward in the song.  Use the -v option to display frame information as you play.

OVER keyword in Oracle

I’m looking at the SQL Scripts that create and modify the database for VPXD. Specifically, I am looking at the oracle version stats[123] scripts. They included a query that uses the “OVER” keyword and I have not found a decent description of what this keyword does. This is the query:

SELECT
st.counter_id AS COUNTER_ID
,v_time_rowid AS TIME_ID
,ST.STAT_VAL AS STAT_VAL
,DENSE_RANK()

OVER (PARTITION BY ST.COUNTER_ID ORDER BY SM.SAMPLE_TIME DESC) AS DENSE_RANK
FROM vpx_sample_time1 sm,vpx_stat_counter sc, vpx_stat_def sd, vpx_hist_stat1 st
WHERE sm.rollup_counter=to_number(to_char(v_start_time_id)||to_char(v_cnt))
AND st.time_id = sm.time_id
AND st.counter_id=SC.COUNTER_ID
AND SC.STAT_ID=SD.ID
AND ((SC.DEVICE_ID =v_null_device_id and v_stat_rollup_level<3) or (v_stat_rollup_level>=3))
AND SD.STAT_LEVEL<=v_stat_rollup_level
AND SD.ROLLUP_TYPE=3)
WHERE DENSE_RANK=1

;

The one description of the Over command I have found has come from the Microsoft website:

“Determines the partitioning and ordering of the rowset before the associated window function is applied”

OK, so it is a union of two types of functionality: partitioning and ordering. The queries I am looking at use both. Let’s start with partitioning:

PARTITION BYDivides the result set into partitions. The window function is applied to each partition separately and computation restarts for each partition.

The window function in this place must somehow refer to DENSE_RANK. To me this looks like it is possibly trying to insert a subselect into the overall select. Here is how MSDN defines DENSE_RANK. I’d like to reiterate that these are Oracle functions I’m trying to understand, and Microsoft is doing all the work explaining them. When I tried executing this SQL Query in SQLPLUS via the Oracle 11 instant client, it balked:
SQL> 2 3 4 5 6 SQL> SP2-0734: unknown command beginning “OVER (PART…” – rest of line ignored.
SQL> SP2-0734: unknown command beginning “FROM vpx_…” – rest of line ignored. It parses fine when I do it as part of a plsql stored procedure.

Time to dig out the slightly larger guns: I picked up a copy of the SQL Cookbook by Anthony Moliaro from Our Friends in Sebastopol, CA. This covered both DENSE_RANK and the OVER keyword.

Over, it turns out, is one of a new set of functions added to SQL 2003 that the Standard calls Windowing functions. This means they operate on a subset of the results returned from the outside query. The best example I saw showed if you are selecting all employees from all departments, and next to each one you wanted to show how many employees were in that department, you could use count(*) with an over clause that specified PARTITION by department. This example is in the Cookbook.

DENSE_RANK () tells you how many rows come before the current row in some ordering scheme. The over clause is used to define that ordering scheme. If two rows have the same rank, they have equivalent values according to DENSE_RANK. For instance, if we have five people with five salaries, their DENSE_RANK of salaies will be 1 through five, highest to lowest. If the top two salaries were identical, top rank would return 1,1,2,3,4.

OK, so it looks like the original query wants to use DENSE_RANK on a subset of the returned rows of the query. The window will be defined by this portion of the clause: PARTITION BY ST.COUNTER_ID

And the order within this window will be determined by: ORDER BY SM.SAMPLE_TIME DESC

OK time to get serious about playing:

I’ve learned a little but since the above frustrations trying to run the stored procedure. Here’s what I do now:

run sqlplus. For me this is

/home/adyoung/apps/oracle/instantclient_11_1/sqlplus adyoung/adyoung@10.17.195.159/DRORACLE

Now make it so anything I print to the screen is visible:

set serveroutput on

Load my sql file in and execute it.

get myfile.sql

/

OK, I have a development method. To print debugging output:
DBMS_OUTPUT.PUT_LINE(‘Text Here’);
And we can debug. In the time honored tradition of always moving from success, I comment everything out but a simple output statement. In this case, I know that the real code uses a variable for the rollup level, so my File (minus commented out section, looks like this:
create or replace procedure stattest
is
v_stat_rollup_level int;
begin
v_stat_rollup_level := 2;
dbms_output.put_line(‘Value: ‘ || TO_CHAR(v_stat_rollup_level));
end stattest;
Which executes using the above method. Now I start adding in parts of the SELECT.
SELECT st.counter_id AS COUNTER_ID FROM vpx_hist_stat1 st;
Of course, it hates this. When I load it into sqlplus I get:
Warning: Procedure created with compilation errors.
Of course, in order to see these compilation errors, I have to jump through a couple more hoops:
SQL> set arraysize 1
SQL> set MAXDATA 6000
SQL> show errors procedure stattest
LINE/COL ERROR
——– —————————————————————–
6/1 PLS-00428: an INTO clause is expected in this SELECT statement
So I add a variable called maxcounter in the declare section and modify my select:
SELECT
max(st.counter_id) into maxcounter
FROM
vpx_hist_stat1 st;
dbms_output.put_line(‘Maxcounter: ‘ || TO_CHAR(maxcounter));
And I have a procedure that executes and does something. On to the rest of the select. After several iterations of trial and error, I have the SQL query wokring as a cursor. I’ve now figured out even a better way to work. I’ve modified my file like below, and can now cat it into the sqlplus interpreter:
set arraysize 1
set MAXDATA 6000
set serveroutput on

create or replace procedure stattest
is
v_stat_rollup_level int := 2;
maxcounter int := 0;

CURSOR MYCURSOR is SELECT
st.counter_id as COUNTER_ID,
st.STAT_VAL as STAT_VAL
FROM vpx_sample_time1 sm,
vpx_stat_counter sc,
vpx_stat_def sd,
vpx_hist_stat1 st

WHERE
st.time_id = sm.time_id
AND st.counter_id=SC.COUNTER_ID
AND SC.STAT_ID=SD.ID
AND SD.ROLLUP_TYPE=3
AND SD.STAT_LEVEL<=v_stat_rollup_level;

begin

open MYCURSOR;

FOR recname in MYCURSOR
LOOP
dbms_output.put_line(‘Value: ‘ || TO_CHAR(v_stat_rollup_level));
END LOOP;

dbms_output.put_line(‘Maxcounter: ‘ || TO_CHAR(maxcounter));

end stattest;

/

show errors procedure stattest

.

OK, I am going to bail here.  This post has been half written for a while, and is not going to get completed anytime soon.  I think I’d vote “Over” as something that would be useful in the next rev of postgres, if only for simplifying porting from Oracle.

Dependency Licenses

The following script, when run on a RPM based system, tells you what licenses your application might trip by linking with other code.  You have to  set $MYEXE to som path to an executable.

for LIB in `ldd $MYEXE | sed -e ‘s!.*=>!!’ -e ‘s! (.*!!’` ; do rpmquery –whatprovides $LIB ; done | grep -v “any package” | sort -u | xargs rpmquery –qf %-30{NAME}:%{LICENSE}”\n

Here is a sample run for perl:

glibc                         :LGPL
perl                          :Artistic or GPL

Note that this is only for the linked files that are registered with RPM.

How to extract from tgz, rpm, and deb files

The most common way to move a bundle of files around in Linux is a combination of tar (tape archive), which appends all of the files together into a single large file,  and gzip, a compression utility.  These are often referred to as tarball. They will have an extension like tar.gz or tgz.  Sometimes bzip2 is used as the comprseeion utility, and they file will end with tar.bz2.

To see the list of files in the  tarball for sage, run:

tar -ztf  sage-2.10.2-linux-ubuntu64-opteron-x86_64-Linux.tar.gz

To extract the files, create a target directory and change to it, thenextract the file from it’s original location

mkdir /tmp/sage

cd /tmp/sage

tar -zxf  ~/Desktop/sage-2.10.2-linux-ubuntu64-opteron-x86_64-Linux.tar.gz

All of the files will be put into the current directory.  There is no rule that says that all the files in a tarball are under on subdirectory, so it really behooves you to do this in a empty directory.  That way you know all of the files you see post extraction are files from the tarball.

Debian uses a package management system called dpkg based on this technology.  The packages will end with .deb, but you can see what Linux thinks the  file type is by using the file command.  Here it is run on automake_1.10+nogfdl-1_all.deb:

 adyoung@adyoung-devd$ file ~/Desktop/automake_1.10+nogfdl-1_all.deb
/home/adyoung/Desktop/automake_1.10+nogfdl-1_all.deb: Debian binary package (format 2.0)

To See the list of included files, run:

dpkg –contents automake_1.10+nogfdl-1_all.deb

And to extract use the –extract command line parameter.  Note that you have to supply the target directory as well.

dpkg –extract automake_1.10+nogfdl-1_all.deb /tmp/deb/

Again, make sure the target directory is empty to avoid intermingling your own files and files from the package.

The letters RPM stand for Redhat Package Manager.  The file extension rpm is used for packages of software designed to be installed on a some distributions of GNU/Linux.   RPM is used on Redhat, SuSE, and distributions based off of these two major distributions, such as Fedora, CentOS, and  OpenSuSE.  RPMs are shipped in a format called cpio. This format has the advantage of allowing longer file names, and providing stroage and compression all in one utility and format.  However, RPMs are not exactly cpio format, and you have to run a converter first, before you can extract the files.  This converter is called rpm2cpio.  It reads the filename in as the first command line parameter, and outputs the cpio file to standard output.  So if you run it without redirecting output, you are going to spew binary data all over your terminal.  Better to redirect it into the cpio utility, with the command line switches -di.  These switches mean extract the files, and build any subdirectories required.  Again, run this in a clean directory:

rpm2cpio ../testware-e.x.p-00000.i386.rpm | cpio -di

Reading C++ symbols in a binary

objdump -t will pull the raw symbols out of an elf binary, but it is mangled format like this

0000000000000000       F *UND*  0000000000000006              _ZNSt13basic_ostreamIwSt11char_traitsIwEElsEPFRS2_S3_E@@GLIBCXX_3.4

c++ file will translate this to a human readly string that maps to the rigianlfunction definition:

 adyoung@adyoung-devd$ echo _ZNSt13basic_ostreamIwSt11char_traitsIwEElsEPFRS2_S3_E@@GLIBCXX_3.4 | c++filt
std::basic_ostream<wchar_t, std::char_traits<wchar_t> >::operator<<(std::basic_ostream<wchar_t, std::char_traits<wchar_t> >& (*)(std::basic_ostream<wchar_t, std::char_traits<wchar_t> >&))@@GLIBCXX_3.4

To do this for an entire file, you want to print only the last column, a task custom made for awk:

 objdump -t   casttest | awk ‘{print $(NF)}’ | c++filt