Category Archives: Software

A favorite interview question of mine

Posted on by
Reply

Given the following API:

int bproc_numnodes() ;

void bproc_move(int nodenum);

long gettime();

Write a program that will tell me the average clock skew in a cluster.

I gave this problem to a lot of people back when I was at Penguin. The API and simple math functions really are all you need. No malloc, or socket calls. I did allow people to #include <stdio.h> for output.

If anyone reads this blog, and has an answer, post it to the comments section. I’ll post an answer with a full explanation in a bit.

Working on the Beowulf Process

Posted on by
Reply

I am currently listed as one of the maintainers of the BProc project on Sourceforge. Unfortunately, my current life has left me little enough time to do my job and be a father, so other projects fall by the wayside.

BPRoc portrays a cluster of computers a single system from an operating system perspective. A process running anywhere one the cluster shows up in the process tree on the head node. Signals sent on any machine were forwarded to the machine where the process was actually running. A process could voluntarily migrate from one machine to another. All of these techniques take place in the Linux Kernel. Maintaining this code requires understand of operating system concepts such as signal delivery, page table organization, dynamic library linking, as well as network programming. I’ve never had more fun coding.

Linux kernel development is done by taking a copy of the kode published at kernel.org and applying a file that contains the differences between how it looks at the start and how you want it to look at the end. This file is called a patch. THe major Linux distributions all have a version of the Linux Kernel that they select as a starting point, and then a series of patches that they apply to deal with issues they care about. For instance, I am running Red Hat Enterprise Linux 4 machine with a kernel version of 2.6.9-55.0.9. The 2.6.9 is the version that they got from kernel.org. The 55.0.9 indicates the number of major and minor upgrades they have made to that kernel. The number patches applied when last I looked was in the neighborhood of 200. All of the changes we applied to the Linux kernel was maintained in a single patch. As we maintained succeeding version of the kernel, we continued to generate newer versions of that patch. In addition to this code, we had a separate, and much larger, portion of code that was compiled into a binary format that could loaded into the Linux Kernel on demand. The majority of the code in the patch was merely hooks into the code that called out to the loadable kernel modules.

Penguin had branched from the Sourceforge BPRoc before I joined. As such, Sourceforge had already moved on to the 2.6 series Linux Kernel while we were still on the 2.4 series. This was a major difference in the code base, and there was little grounds for sharing. When we did finally start moving to the 2.6 code base, we had one Marketing requirement that the Sourceforge project did not: We needed to interoperate with the Linux Kernel shipped by RedHat for there Enterprise Linux product (RHEL). I spent a long time in research mode trying to make this happen. Two major decisions came out of this. First, PID masquerading had to go. Second, we needed to use binary patching in place of many of the source level patches.

 

Every process in an operating system has an integer process identifier (PID) that other processes and the kernel can use to access that process. A major mechanism in BProc was the ability to migrate a process from one physical machine to another. PID masquerading is a technique that ensures that the process identified does not have to change during migration. Instead, each process has two identifiers. The first is the ID as allocated on the head node, and is used when reporting information to the head node, other nodes, or user land functions. The second ID is the PID allocated on the local machine, and only used inside to local machines Kernel. When a function like getpid is called, the process identifier returned is the masqueraded PID, not the local PID. PID masquerading has both positive and negative implications. With PID masquerading, a given compute node can actually have two completely separate pools of processes that cannot communicate with each other. each of the pools of processes can be driven from a different head node. This allows the sharing of compute nodes between head nodes. A given machine can actually act as both a head node and a compute node. This was a requirement in early Beowulf clusters, but was no longer necessary by the time I worked on them. The negative impact of PID masquerading was the amount of code required to support it. Every PID reference in the Linux kernel had to be scrutinized for whether it should be a local or remote PID. If it needed to be translated, a hook was inserted that said “If the module is loaded, and this process is a masqueraded process, return the masqueraded PID, otherwise return the real PID.” This type of logic composed approximately a quarter of the BProc Linux Kernel patch. There was no practical way we could inject all of this code without source level patching the kernel.

 

 

 

 

 

Binary patching means changing the machine code on a running system. There are two assembly instructions we looked for to see if we could change code. They are CALL and JUMP. Actually, there are two types of jumps, long and short, and we can use either of them. We did analysis of the compiled Linux kernel for places with these instructions near our current set of hooks. The CALL instruction is what maps to a function call in C. In assembly it looks like CALL 0x00000000, where the zeros will be replaced by the linker or the loaded with an address in memory where the function resides. Once we know where the call operation takes place, we can replace the value with our own function. This technique is often used with malicious intent in virus and root kits, but really is not much different than how a debugger or many security software packages work. During the replacement process, we record the original value of the function, so that we can unload our module and return it to it’s original flow. The compiler will often use a JMP instruction in the place of a CALL instruction as an optimization called a “tail call.” All this means is that when the called function returns, instead of returning to the location it was called from, it continues up the call stack. I discussed this in the CS GRE problem set post.

One place that we had to hook to make this work was the function and structure that allocated the process identifiers. The function alloc_pidmap gets a PID from a bitmap. The bitmap is just a page of memory treated as an array of bytes. Bit zero of page[0] represents the PID 0, Bit 1 represents PID 1, and so on. If a given bit is set to 1, there exists a structure in memory that is using that process ID. In standard configuration, a page in Linux is 4k bytes. 1024*4*8=32768, which is the largest number that can be held in a 16 bit signed integer. PIDs have traditionally been 16 bit signed integers in Unix, and Linux. We used a couple tricks to help out here. On the Head node, we set all PIDs less than some threshold (we chose 1000) to be 1, indicating to the system that it should not allocate those pids. On compute nodes, we set all PIDs greater than the threshold to be 1. PIDs to be visible across the entire cluster were allocated on the head node. PIDs allocated for local work on the compute node would be guaranteed not to class with PIDs from the head node.

Aside:  recent versions of the Linux kernel have expanded PIDs to be 32bit signed integers.  At first it was tempting to expand the allowable PIDs, statically partition the PID space amongst the compute nodes, and allow local allocation of PIDs.  We origianlly pursued this approach, but rejected it for several reasons.  First, the Linux Kernel set an arbitrary limit of 4*1024*1024 on the number of PIDS.  We wanted to be able to support clusters of 1024.  This means that any node on the cluster had only 4*1024 PIDs to allocate.  Since the vast majority of PIDs were handed out on the head node anyway, we had to do some unbalanced scheme where the head node go something in the neighborhood of 16000 PIDs, leaving a very small pool to be handed out on each the compute nodes.  Additionally, a compute node crash erased all record of the PIDs that had been handed out on that machine.  Replacing a node meant rebuilding the pidmap from the existing process tree, a very error prone and time consuming activity.  Also, many applications still assumed a 16 bit PID, and we did not want to break those applications.

We found that there were several race conditions that hit us if we relied solely on the pidmap structure to control PIDs. This we ended up hooking  alloc_pidmap, checking for a compute node or head node, and checking that the returned PID was withing the appropriate range.  This type of code pathis frowned up in the mainline Linux kernel, but we found no noticable performance hit in our benchmark applications.

One benefit to this approach was that we could then slowly remove the PID Masquerading code.  We continued to track both the masqueraded and real PIDs, but they were assigned the same value.  Thus we never broke the system as we restructured.

Java Native Interface

Posted on by
Reply

The reason I left Java programming was due to the type of work I got in that field. However, I love the language. It provides the programmer with the right set of tools to focus on getting the task done. Memory management, while important, is not the primary focus of each and every programming task.

Don Knuth’s quote about premature optimization being the root of all evil often comes to mind. C is a great language for low level tasks, for embedding code on a really small platform, and for ensuring that you can understand the end binaries format and structure. One nice thing about Java is that it, too has a rigidly defined and easy to understand binary format. So, one approach to problem programming might be to code in Java to solve the problem, then optimize in C.

The language binding between Java and is the Java Native Interface (JNI). JNI makes use of the Java ‘native’ keyword. a Native method is one implemented external to the current Java class.

Here is a trivial example:

public class Sample

{

   private native String reverse(String prompt);   static {

      System.loadLibrary("reverse");

   }

public static void main(String[] args)

      {

         Sample jn=new Sample();

System.out.println("Got "+ jn.reverse("Some Data"));

      }

}

The ‘native’ line defines the method reverse. To resolve this, the jvm will look for a symbol: Java_Sample_reverse. From this we can infer that
parameter polymorphism is either not supported, or the jvm only resorts to name mangling when required to deal ambiguous situations.

the nexts steps are 1) run javac to convert the .java file to a .class file. and 2)run javah on the .class file to generate a .h file for the c library.

The function definition looks like this:

JNIEXPORT jstring JNICALL Java_Sample_reverse
(JNIEnv *, jobject, jstring);

This is within an extern “C” block for C++code, so we don’t get C++ semantics for the function implementation. This means you won’t be able to make a single set of objects to be shared between C+++ and Java, but this would be unwieldy no matter what.

For gcc The JNIEXPORT is resolved away to nothing. This compiler directive is to support Microsoft’s compiler which forces you to declare if a function is going to be visible outside of the scope of the current file, pretty much the opposite of ‘static’. JNICALL also resolves to nothing, but I suspect on a windows platform it would introduce code that specifies the pascal function calling convention for handling parameters. JNIEnv is a pointer to a pointer, and thus must be used like this :
(*env)->doSomething();
C++ Provides operator overloading of the -> so you can refer using this semantic instead:
env->doSomething();

Note that the string gets passed as a jstring. jni.h shows this to be a jobject. jobject is a struct with nothing inside it. Thus, once we go from java to C++, we are not supposed to know what exists on the other side except from reading the specification.

For this example, I have a function that reverses the string passed in. Here it is:

#include 

#include 

#include 

#include /*

 * Class:     MyJni

 * Method:    getLine

 * Signature: (Ljava/lang/String;)Ljava/lang/String;

 */

JNIEXPORT jstring JNICALL Java_Sample_reverse

  (JNIEnv * env, jobject obj, jstring str)

{

   jstring retval;

   char * msg;

   int i;

   int len;

const char * orig = (*env)->GetStringUTFChars(env, str,0);

len = strlen(orig);

msg = (char *)malloc(len+1);

/*disregard the null termination fro now; it will not be copied.*/

   len--;

for (i = 0; i <= len; ++i){

      msg[i] = orig[len-i];

   }

   msg[i]=0;

(*env)->ReleaseStringUTFChars(env, str, orig);

retval = (*env)->NewStringUTF(env, msg);

free(msg);

return retval;

}

Here is the Makefile I used to build it.

CFLAGS+=-I/usr/lib/jvm/java-6-sun-1.6.0.00/include -I/usr/lib/jvm/java-6-sun-1.6.0.00/include/linux
FILES=MyJni.h

all: libreverse.so Sample.class

.SUFFIXES : .c  .class .h .so .java

.class.h :
        javah $*

clean :
        rm -rf *.o *.so $(FILES) *~ *.class Sample.h

.java.class :
        javac $<

libreverse.so : Sample.h reverse.c
        gcc reverse.c -o $*.so -shared $(CFLAGS) -fPIC

test :
        LD_LIBRARY_PATH=. java Sample

$: make && make test
javac Sample.java
javah Sample
gcc reverse.c -o libreverse.so -shared -I/usr/lib/jvm/java-6-sun-1.6.0.00/include -I/usr/lib/jvm/java-6-sun-1.6.0.00/include/linux -fPIC
LD_LIBRARY_PATH=. java Sample
Got ataD emoS

rpm and deb commands: files to packages and back

Posted on by
1

These two commands tell you which package owns a particular file for an RPM or DEB based system respectively.

rpmquery –whatprovides <path-to-file>

dpkg-query –search <path-to-file>

(The short form for dpkg is -S)

To list the files owned by a package use:

rpmquery –list <package>

dpkg-query -L <package>

(This long form for debian is –listfiles )

A useful utility to find all the other files in a package for a given binary (in this case lsof) is:

rpmquery –list $(rpmquery –whatprovides $(which lsof ) )

Or

dpkg-query -L $(dpkg-query –search $( which lsof ) | sed ‘s!\:.*!!’ )

Projects that need IPv6 support

Posted on by
Reply

Our project uses a bunch of open source packages. I’ve been looking through them and this seems to be the current state:

  • Linux Kernel: Good to go in both 2.4 and 2.6
  • OpenPegasus CIM Broker: IPV6 support is underway, but not yet implemented.
  • SBLIM SFCBD IPV6 Support is built in, based on a compile time switch
  • OpenIPMI: IPMI Tool won’t accept a valid IPv6 address. This is a slightly different code source than the rest of the project, so it doesn’t mean that the rest of it won’t support IPv6.
  • OpenWSMAN
  • OpenSSL: Claims to be agnostic of the IP level. Since Open SSH is build on OpenSSL, and OpenSSH works, it works for at least a subset of it’s functionality.
  • OpenSSH: Connecting via IPv6 Works Confirmed for both ssh and scp.  scp is a pain.
  • OpenSLP: Seems to have IPv6 support, but it isvery recent.  It requires IPv6 multicast support.  Multicast has often been an after thought in switch implementations, so IPv6 multicast may have issues in the future.

Continuing support for IPv4 with a switch for IPv6

Posted on by
Reply

Although our product needs to support IPv6, it will be used by people in IPv4 mode for the near future. Since a call to the socket or bind system calls will fail if the underlying interface is not IPv6 enabled, we have to fall back to IPv6. So I am currently thinking we’ll have code like this:

int af_inet_version = AF_INET;

With code that can set that to AF_INET6 either read out of a config file or a command line argument.

Then later…

int rc = socket( af_inet_version, …);

And when calling bind, use af_inet_version to switch

Part of getting our product converted to IPv6 is project for reliable messaging. This code is no longer actively maintained. I’ve done a few simple greps through the code and agree with my co-worker who warned me that it is going to be quite tricky. Aside from the obvious calls to socket and bind, ISIS records addresses to be reused later. For example, In include/cl_inter.h the ioq structure contains

saddr io_address; /* Destination address */
saddr io_rcvaddr; /* Receive address (for statistics) */

where saddrs is a typedef in include/cl_typedefs.h

typedef struct sockaddr_in saddr;

I am thinking of an approach that would be to use a union:

struct sockaddress{
union{
struct sockaddr_in in;
struct sockaddr_in6 in6;
}addr;
};
struct sockaddress sin;
struct sockaddress pin;

switch(af_inet_version){
case AF_INET:
addrsize = sizeof(struct sockaddr_in);
sin.addr.in.sin_addr.s_addr = INADDR_ANY;
sin.addr.in.sin_port = htons(port);
sin.addr.in.sin_family = af_inet_version;
break;

case AF_INET6:
addrsize = sizeof(struct sockaddr_in6);
sin.addr.in6.sin6_addr = in6addr_any;
sin.addr.in6.sin6_port = htons(port);
sin.addr.in6.sin6_family = af_inet_version;
break;
}

I put the union inside a struct because I originally was going to put the AF_INET to AF_INET6 field into struct sockaddress. I may go back to that for the real code, and then I can support both IPv6 and IPv4 in a single system.

extracting files from an rpm

Posted on by
Reply

rpms are saved in a file format called cpio, with a few minor changes.  To convert an rpm to this format use rpm2cpio.  The cpio binary is responsible for manipulating these files.  So if you want to treat that rpm as just another archive, and extract the files in it, you can link a couple of tools.  Here’s the line:

rpm2cpio your.rpm | cpio -di

IPv6 Language Comparison

Posted on by
Reply

Language standard library support for IPv6.

What follows is an attempt to view the support for IPv6 in various
languages.

Java

Java has clean support for IPV6, and makes it easy to go between
IPv4 and IPv6 addresses. Example:

import java.net.*;public class IPTest{

private static void displayClasses(String host){
 		System.out.print("looking up host" +host+"tt");
 		try{
 			InetAddress[] address =	InetAddress.getAllByName(host);
 			System.out.print("[Success]:");
 			for (int i =0; i < address.length;i++){
 				System.out.println(address[i].getClass().getName());
 			}
 		}catch(UnknownHostException e){

System.out.println("[Unknown]");
 		}

}

public static void main(String[] args) {

displayClasses("fe80::218:8bff:fec4:284b");
 		displayClasses("fe80::218:8bff:fec4:284b/64");
 		displayClasses("00:18:8B:C4:28:4B");
 		displayClasses("::10.17.126.126");
 		displayClasses("10.17.126.126");
 		displayClasses("vmware.com");
 		displayClasses("adyoung-laptop");

}
 }

This code produces the following output

adyoung@adyoung-laptop$ java IPTest
 looking up hostfe80::218:8bff:fec4:284b         [Success]:java.net.Inet6Address
 looking up hostfe80::218:8bff:fec4:284b/64              [Unknown]
 looking up host00:18:8B:C4:28:4B                [Unknown]
 looking up host::10.17.126.126          [Success]:java.net.Inet6Address
 looking up host10.17.126.126            [Success]:java.net.Inet4Address
 looking up hostvmware.com               [Success]:java.net.Inet4Address
 looking up hostadyoung-laptop           [Success]:java.net.Inet4Address

C++

While C++ can always default to C for network support, I wanted to
see what existed in the C++ way of doing things. There is nothing in
the standard library for network support, and nothing pending in TR1.
The third party libary for Asynchronous I/O (asio) does support
IPv6. Boost has not accepted this package yet, but the acceptance
process appears to be underway. This package has a class for ip
address abstraction: asio::ip::address.

#include <iostream>
 #include <boost/array.hpp>
 #include <asio.hpp>using asio::ip::address;
 using namespace std;

void displayAddr(char * addr){
 	cout << "parsing addr " << addr;
 	try{
 		address::from_string(addr) ;
 		cout << "t[success]" ;
 	}catch(...){
 		cout << "t[Failed]";
 	}
 	cout << endl;
 }

int main(int argc, char* argv[]){
 	displayAddr("fe80::218:8bff:fec4:284b");
 	displayAddr("fe80::218:8bff:fec4:284b/64");
 	displayAddr("00:18:8B:C4:28:4B");
 	displayAddr("::10.17.126.126");
 	displayAddr("10.17.126.126");
 	displayAddr("vmware.com");
 	displayAddr("adyoung-laptop");
 	return 0;
 }

This produces the following output:

parsing addr fe80::218:8bff:fec4:284b   [success]
 parsing addr fe80::218:8bff:fec4:284b/64        [Failed]
 parsing addr 00:18:8B:C4:28:4B  [Failed]
 parsing addr ::10.17.126.126    [success]
 parsing addr 10.17.126.126      [success]
 parsing addr vmware.com [Failed]
 parsing addr adyoung-laptop     [Failed]

So the major distinction between this and the Java code is that the
Java code accepts hostnames, this only accepts well formed IP
Addresses.

Python

Python has IPv6 support built in to recent version.

#!/usr/bin/pythonimport socket

def displayAddr(addr):
 	try :
 		addr_info = socket.getaddrinfo(addr,"");
 		print "Parsing ", addr , "t[Succeeded]"
 	except socket.gaierror :
 		print "Parsing ", addr, "t[Failed]"

def main():
 	displayAddr("fe80::218:8bff:fec4:284b");
 	displayAddr("fe80::218:8bff:fec4:284b/64");
 	displayAddr("00:18:8B:C4:28:4B");
 	displayAddr("::10.17.126.126");
 	displayAddr("10.17.126.126");
 	displayAddr("vmware.com");
 	displayAddr("adyoung-laptop");

if __name__ == '__main__':
 	main()

This Code produces the following output

 adyoung@adyoung-laptop$ ./SockTest.py
 Parsing  fe80::218:8bff:fec4:284b       [Succeeded]
 Parsing  fe80::218:8bff:fec4:284b/64    [Failed]
 Parsing  00:18:8B:C4:28:4B      [Failed]
 Parsing  ::10.17.126.126        [Succeeded]
 Parsing  10.17.126.126  [Succeeded]
 Parsing  vmware.com     [Succeeded]
 Parsing  adyoung-laptop         [Succeeded]

So, like Java, hostnames are correctly parsed the same as IP
addressed.

PERL

Perl has two APIs that look promising: Socket and Net::IP.

#!/usr/bin/perl
 use strict;
 use Socket;
 sub displayAddr{
 	my ($addr) = @_;
 	my $host = gethostbyname ($addr);
 	if ($host){
 		print ("parsed ".$addr."n");
 	}else{
 		print ("Unable to parse ".$addr."n");
 	}
 }
 displayAddr("fec0::218:8bff:fe81:f81e");
 displayAddr("fe80::218:8bff:fe81:f81e");
 displayAddr("fe80::218:8bff:fec4:284b");
 displayAddr("fe80:0:0:0:0:0:0:0:0:0:0:0:218:8bff:fec4:284b");
 displayAddr("fe80::218:8bff:fec4:284b/64");
 displayAddr("00:18:8B:C4:28:4B");
 displayAddr("::10.17.126.126");
 displayAddr("10.17.126.126");
 displayAddr("vmware.com");
 displayAddr("adyoung-laptop");
 displayAddr("ip6-allhosts");
 displayAddr("ip6-localnet");

This produces:

adyoung@adyoung-laptop$ less ip-test.pl
 adyoung@adyoung-laptop$ ./ip-test.pl
 IP  : fec0:0000:0000:0000:0218:8bff:fe81:f81e Type: RESERVED
 IP  : fe80:0000:0000:0000:0218:8bff:fe81:f81e Type: LINK-LOCAL-UNICAST
 IP  : fe80:0000:0000:0000:0218:8bff:fec4:284b Type: LINK-LOCAL-UNICAST
 cannot parse fe80:0:0:0:0:0:0:0:0:0:0:0:218:8bff:fec4:284b
 cannot parse fe80::218:8bff:fec4:284b/64
 cannot parse 00:18:8B:C4:28:4B
 IP  : 0000:0000:0000:0000:0000:0000:0a11:7e7e Type: IPV4COMP
 IP  : 10.17.126.126 Type: PRIVATE
 cannot parse vmware.com
 cannot parse adyoung-laptop
 cannot parse ip6-allhosts
 cannot parse ip6-localnet

So it handles IPv4 and IPv6., but not host names.
The alternate API, Socket, is older. It doesn ot seem to have the
new Posix function getaddrinfo, so I treid the old gethostbyname:

#!/usr/bin/perl

use strict;
 use Socket;
 sub displayAddr{
 	my ($addr) = @_;
 	my $host = gethostbyname ($addr);
 	if ($host){
 		print ("parsed ".$addr."n");
 	}else{
 		print ("Unable to parse ".$addr."n");
 	}
 }
 displayAddr("fec0::218:8bff:fe81:f81e");
 displayAddr("fe80::218:8bff:fe81:f81e");
 displayAddr("fe80::218:8bff:fec4:284b");
 displayAddr("fe80:0:0:0:0:0:0:0:0:0:0:0:218:8bff:fec4:284b");
 displayAddr("fe80::218:8bff:fec4:284b/64");
 displayAddr("00:18:8B:C4:28:4B");
 displayAddr("::10.17.126.126");
 displayAddr("10.17.126.126");
 displayAddr("vmware.com");
 displayAddr("adyoung-laptop");
 displayAddr("ip6-allhosts");
 displayAddr("ip6-localnet");

This produced the following output:

Unable to parse fec0::218:8bff:fe81:f81e
 Unable to parse fe80::218:8bff:fe81:f81e
 Unable to parse fe80::218:8bff:fec4:284b
 Unable to parse fe80:0:0:0:0:0:0:0:0:0:0:0:218:8bff:fec4:284b
 Unable to parse fe80::218:8bff:fec4:284b/64
 Unable to parse 00:18:8B:C4:28:4B
 Unable to parse ::10.17.126.126
 parsed 10.17.126.126
 parsed vmware.com
 parsed adyoung-laptop
 Unable to parse ip6-allhosts
 Unable to parse ip6-localnet

It was able to handle IPv4 and domain names, but not IPv6. This was
on a system that had an IPv6 interface, so it was not the problem
shown in the straight C section

C#

This code exercizes the IPAddress and Dns classes.

using System;
 using System.Net;public class NetTest
 {
 	public static void Main(string[] args){
 		DisplayAddr("fec0::218:8bff:fe81:f81e");
 		DisplayAddr("fe80::218:8bff:fe81:f81e");
 		DisplayAddr("fe80::218:8bff:fec4:284b");
 		DisplayAddr("fe80:0:0:0:0:0:0:0:0:0:0:0:218:8bff:fec4:284b");
 		DisplayAddr("fe80::218:8bff:fec4:284b/64");
 		DisplayAddr("00:18:8B:C4:28:4B");
 		DisplayAddr("::10.17.126.126");
 		DisplayAddr("10.17.126.126");
 		DisplayAddr("vmware.com");
 		DisplayAddr("adyoung-laptop");
 		DisplayAddr("ip6-allhosts");
 		DisplayAddr("ip6-localnet");
 	}

public static void DisplayAddr(string host){
 		try{
 			IPAddress addr = IPAddress.Parse(host);
 			Console.WriteLine("addr has address:"
 					+ addr.ToString());
 		}catch(Exception ){
 			Console.WriteLine("unable to parse :" + host);
 			try{
 				IPHostEntry hostEntry = Dns.GetHostByName(host);

Console.WriteLine("addr has address:"
 						+ hostEntry.AddressList[0]
 						.ToString());
 			}catch(Exception ){
 				Console.WriteLine("Cannot get host from DNS:",host);
 			}
 		}
 	}
 }

This code produces the following output.

addr has address:fec0::218:8bff:fe81:f81e
 addr has address:fe80::218:8bff:fe81:f81e
 addr has address:fe80::218:8bff:fec4:284b
 unable to parse :fe80:0:0:0:0:0:0:0:0:0:0:0:218:8bff:fec4:284b
 Cannot get host from DNS:
 addr has address:fe80::218:8bff:fec4:284b
 unable to parse :00:18:8B:C4:28:4B
 Cannot get host from DNS:
 addr has address:::10.17.126.126
 addr has address:10.17.126.126
 unable to parse :vmware.com
 addr has address:10.19.249.99
 unable to parse :adyoung-laptop
 addr has address:10.17.124.70
 unable to parse :ip6-allhosts
 Cannot get host from DNS:
 unable to parse :ip6-localnet
 Cannot get host from DNS:

C# Handles both IPv4 and IPv6 address equally well. The IPAddress
class does not handle host names. The Dns service does an explicit
DNS lookup, not a call via the NSSwitch functions. This leaves a
hole for hosts declared via YP, /etc/hosts, LDAP, or other naming
services. I ran this under both Mono and Microsoft Visual Studio.
On MSVS, it called out that GetHostByName was deprecated, but the
replacement call, GetHostEntry, had the same behavior.

Straight C
(Posix)

The obvious method does not work:

#include <sys/types.h>
 #include <sys/socket.h>
 #include <netdb.h>
 #include <stdio.h>void displayAddr(char *addr){
         struct hostent * hostent;

hostent =gethostbyname(addr);
         if (hostent){
                 printf( "Parsing %s t[Succeeded]n", addr);
         }else{

printf( "Parsing %s t[Failed]n", addr);
         }
 }

int main(){
         displayAddr("fe80::218:8bff:fec4:284b");
         displayAddr("fe80::218:8bff:fec4:284b/64");
         displayAddr("00:18:8B:C4:28:4B");
         displayAddr("::10.17.126.126");
         displayAddr("10.17.126.126");
         displayAddr("vmware.com");
         displayAddr("adyoung-laptop");
         return 0;
 }

This code produces the following output:

adyoung@adyoung-laptop$ ./socktest
 Parsing fe80::218:8bff:fec4:284b        [Failed]
 Parsing fe80::218:8bff:fec4:284b/64     [Failed]
 Parsing 00:18:8B:C4:28:4B       [Failed]
 Parsing ::10.17.126.126         [Failed]
 Parsing 10.17.126.126   [Succeeded]
 Parsing vmware.com      [Succeeded]
 Parsing adyoung-laptop  [Succeeded]

Thus it does not deal with IPv6 addresses correctly. This seems to
be at odds with the man page which states:

The gethostbyname() function returns a structure of type hostent for the given  host  name.   Here name  is  either  a  host name, or an IPv4 address in standard dot notation, or an IPv6 address in colon (and possibly dot) notation.

The next attempt is to using the call specified in the porting
doc:getaddrinfo

#include <sys/types.h>
 #include <sys/socket.h>
 #include <netdb.h>
 #include <stdio.h>void displayAddr(char *addr){
 	struct addrinfo * addrinfo;
 	int rc = getaddrinfo(addr,NULL,NULL,&addrinfo);
 	if (0 ==  rc ){
 		printf( "Parsing %s t[Succeeded]n", addr);
 		freeaddrinfo(addrinfo);
 	}else{

printf( "Parsing %s t[Failed]n", addr);
 	}
 }

int main(){
 	displayAddr("fec0::218:8bff:fe81:f81e");
 	displayAddr("fe80::218:8bff:fe81:f81e");
 	displayAddr("fe80::218:8bff:fec4:284b");
 	displayAddr("fe80:0:0:0:0:0:0:0:0:0:0:0:218:8bff:fec4:284b");
 	displayAddr("fe80::218:8bff:fec4:284b/64");
 	displayAddr("00:18:8B:C4:28:4B");
 	displayAddr("::10.17.126.126");
 	displayAddr("10.17.126.126");
 	displayAddr("vmware.com");
 	displayAddr("adyoung-laptop");
 	return 0;
 }

This Code Worked differently Depending on whether the machine had an
IPv6 interface configured. Without and IPv6 Interface:

adyoung@adyoung-laptop$ ./getaddrinfo-test
 Parsing fec0::218:8bff:fe81:f81e        [Failed]
 Parsing fe80::218:8bff:fe81:f81e        [Failed]
 Parsing fe80::218:8bff:fec4:284b        [Failed]
 Parsing fe80:0:0:0:0:0:0:0:0:0:0:0:218:8bff:fec4:284b   [Failed]
 Parsing fe80::218:8bff:fec4:284b/64     [Failed]
 Parsing 00:18:8B:C4:28:4B       [Failed]
 Parsing ::10.17.126.126         [Failed]
 Parsing 10.17.126.126   [Succeeded]
 Parsing vmware.com      [Succeeded]
 Parsing adyoung-laptop  [Succeeded]

With an IPv6 interface.

-bash-3.00$ ./getaddrinfo-test
 Parsing fec0::218:8bff:fe81:f81e        [Succeeded]
 Parsing fe80::218:8bff:fe81:f81e        [Succeeded]
 Parsing fe80::218:8bff:fec4:284b        [Succeeded]
 Parsing fe80:0:0:0:0:0:0:0:0:0:0:0:218:8bff:fec4:284b   [Failed]
 Parsing fe80::218:8bff:fec4:284b/64     [Failed]
 Parsing 00:18:8B:C4:28:4B       [Failed]
 Parsing ::10.17.126.126         [Succeeded]
 Parsing 10.17.126.126   [Succeeded]
 Parsing vmware.com      [Succeeded]
 Parsing adyoung-laptop  [Failed]

The getaddrinfo function call gives us a way to determine the
correct family to use to connect to the host. If we add this to the
displayAddr function:

		switch( addrinfo->ai_family){
 			case AF_INET6:
 				printf( "socket family = AF_INET6n");
 				break;			case AF_INET:
 				printf( "socket family = AF_INETn");
 				break;
 		}

and request the resolution of a few more hosts:

	displayAddr("ip6-allhosts");
 	displayAddr("ip6-localnet");

We get:

Parsing fec0::218:8bff:fe81:f81e        [Succeeded]socket family = AF_INET6
 Parsing fe80::218:8bff:fe81:f81e        [Succeeded]socket family = AF_INET6
 Parsing fe80::218:8bff:fec4:284b        [Succeeded]socket family = AF_INET6
 Parsing fe80:0:0:0:0:0:0:0:0:0:0:0:218:8bff:fec4:284b   [Failed]
 Name or service not known
 Parsing fe80::218:8bff:fec4:284b/64     [Failed]
 Name or service not known
 Parsing 00:18:8B:C4:28:4B       [Failed]
 Name or service not known
 Parsing ::10.17.126.126         [Succeeded]socket family = AF_INET6
 Parsing 10.17.126.126   [Succeeded]socket family = AF_INET
 Parsing vmware.com      [Succeeded]socket family = AF_INET
 Parsing adyoung-laptop  [Succeeded]socket family = AF_INET
 Parsing ip6-allhosts    [Succeeded]socket family = AF_INET6
 Parsing ip6-localnet    [Succeeded]socket family = AF_INET6

So we can take the approach where the applications store hosts in a
free string format, presumably by host names, but perhaps by IPv4 or
IPv6 addresses, and we will use gethostinfo to decide how to connect.
For example, (without error handling)

void connectTo(char * host){
 	struct addrinfo * addrinfo;
 	struct protoent * protoent;
 	protoent = getprotobyname("tcp");
 	int rc = getaddrinfo(host,NULL,NULL,&addrinfo);
 	int sockfd = socket( addrinfo->ai_family,SOCK_STREAM,protoent->p_proto);
 }

Visual C++

The code for VC++ is very similar to posix, with slightly
different build requirments. Note the addition of the winsock
initialization code.

// nettest.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"
#include <stdio.h>
void displayAddr(char *addr){
        struct addrinfo * addrinfo;
        int rc = getaddrinfo(addr,NULL,NULL,&addrinfo);
        if (0 ==  rc ){
                printf( "Parsing %s t[Succeeded]", addr);
                switch( addrinfo->ai_family){
                                                case AF_INET6:
                                                        printf( "socket family = AF_INET6n");
                                                        break;

                                                case AF_INET:
                                                        printf( "socket family = AF_INETn");
                                                        break;
                }
                freeaddrinfo(addrinfo);
        }else{
                printf( "Parsing %s t[Failed]n", addr);
                printf("%sn",gai_strerror(rc));
        }
}

void connectTo(char * host){
        struct addrinfo * addrinfo;
        struct protoent * protoent;
        protoent = getprotobyname("tcp");
        int rc = getaddrinfo(host,NULL,NULL,&addrinfo);
        int sockfd = socket( addrinfo->ai_family,SOCK_STREAM,protoent->p_proto);
}
WSAData wsaData;
int _tmain(int argc, _TCHAR* argv[])
{
        int iResult;

        // Initialize Winsock
        iResult = WSAStartup(MAKEWORD(2,2), &wsaData);
        if (iResult != 0) {
                printf("WSAStartup failed: %dn", iResult);
                return 1;
        }

        displayAddr("fec0::218:8bff:fe81:f81e");
        displayAddr("fe80::218:8bff:fe81:f81e");
        displayAddr("fe80::218:8bff:fec4:284b");
        displayAddr("fe80:0:0:0:0:0:0:0:0:0:0:0:218:8bff:fec4:284b");
        displayAddr("fe80::218:8bff:fec4:284b/64");
        displayAddr("00:18:8B:C4:28:4B");
        displayAddr("::10.17.126.126");
        displayAddr("10.17.126.126");
        displayAddr("vmware.com");
        displayAddr("adyoung-laptop");
        displayAddr("ip6-allhosts");
        displayAddr("ip6-localnet");

        return 0;
}

The file stdafx.h contains the includes

#include <winsock2.h>
#include <ws2tcpip.h>

This produced the output:

Parsing fec0::218:8bff:fe81:f81e        [Succeeded]socket family = AF_INET6
Parsing fe80::218:8bff:fe81:f81e        [Succeeded]socket family = AF_INET6
Parsing fe80::218:8bff:fec4:284b        [Succeeded]socket family = AF_INET6
Parsing fe80:0:0:0:0:0:0:0:0:0:0:0:218:8bff:fec4:284b   [Failed]
N
Parsing fe80::218:8bff:fec4:284b/64     [Failed]
N
Parsing 00:18:8B:C4:28:4B       [Failed]
N
Parsing ::10.17.126.126         [Succeeded]socket family = AF_INET6
Parsing 10.17.126.126   [Succeeded]socket family = AF_INET
Parsing vmware.com      [Succeeded]socket family = AF_INET
Parsing adyoung-laptop  [Failed]
N
Parsing ip6-allhosts    [Succeeded]socket family = AF_INET6
Parsing ip6-localnet    [Succeeded]socket family = AF_INET6

Again, this only worked if there was an IPv6 interface configured on
the system. Also, the default localhost, allhosts, and localnet
names from Linux were not supported on Windows. These had been
declared in /etc/hosts. Once these entries were added to
c:\windows\system32\drivers\etc\hosts it worked correctly. The error
reporting function does not work, but it does resolve and link. It
merely prints out the letter ‘N’.

If you don’t have arrays…but you have awk

Posted on by
Reply

A small project I have in the embedded side of things requires keeping track of information per index, what one would normally do in an array. However, the ash shell from busy box does not support arrays. Busybox does support awk. The following function is used to increment, decrement, and reset and array of counters of single digits stored as characters in a string.

#mod_field takes three parameters: OP INDEX DATA

#OP is the opperation: inc, dec, reset

#INDEX is the offset into the string from 1 to 9

#DATA is the state stored as a string

#OUTPUT is DATA with DATA[INDEX] modified

mod_field(){
OP=$1
INDEX=$2
DATA=$3

echo $OP $INDEX $DATA | \
awk ‘
{
VAL=substr($3,$2,1);
FIRST=substr($3,0,$2-1);
TAIL=substr($3,$2+1) ;
}
$1 ~ /inc/ { VAL++; }
$1 ~ /dec/ { VAL–; }
$1 ~ /reset/{ VAL=0; }
END{ print FIRST VAL TAIL;}
‘
}

#Here is how to call it:
DATA=0000000000
DATA=`mod_field inc 3 $DATA `
echo $DATA

#This should produce 0010000000

DATA=`mod_field dec 3 $DATA `
echo $DATA

#And set DATA back to 0000000000
for i in 1 2 2 3 4 4 4 4 5 6 6 7 8 8 8 8 8 8 9
do
DATA=`mod_field inc $i $DATA `
done
echo $DATA
#Should produce 1214121610

for i in 1 2 2 3 4 4 4 4 5 6 6 7 8 8 8 8 8 8 9
do
DATA=`mod_field dec $i $DATA `
done
echo $DATA

#Should produce 0000000000

DATA=1111111111111
DATA=`mod_field reset 3 $DATA`
echo $DATA

#Should produce 0000000000