I’ve been porting our Active Directory based LDAP scripts to OpenLDAP.

Here’s what I have so far:

in /etc/openldap/slapd.conf


suffix                “dc=my-domain,dc=com”
rootdn                “cn=Manager,dc=my-domain,dc=com”

suffix                “dc=myproject,dc=company,dc=int”
rootdn                “cn=Manager,dc=myproject,dc=company,dc=int”

And added a password generated with:

slappasswd -s password

That looks like this:
rootpw                 {SSHA}qGjxdj5lesdqFmAJNk4Mn/c3uYULH06q

I have a “blow away the DB and restart” script that looks like this:
#  cat ~adyoung/bin/reset_ldap

/etc/init.d/ldap stop
rm -f /var/lib/ldap/*
cp /etc/openldap/DB_CONFIG.example /var/lib/ldap/DB_CONFIG
/etc/init.d/ldap start

I can insert things into the database with:

ldapadd  -D “cn=Manager, dc=myproject, dc=mycompany, dc=int” -x  -w mycompany -f my_schema.ldif

Note that the first thing inserted has to be the top level item itself:

dn: dc=myproject,dc=mycompany,dc=int
changetype: add
objectClass: top
objectClass: dcObject
objectClass: organization

I can query what objects are in this DB  by running

ldapsearch -LLL -x -b  ‘dc=myproject,dc=mycompany,dc=int’ ‘(objectclass=*)’

I’ve been converting out ldif files for the schema into schema files, as I can then test them by running the above script, which, amongst other things, runs slaptest.

When you insert an object into the LDAP DB, it has to have an objecttype.  Attribute types are simple values used to compose objects.  They are defined before the objectypes that use them.  Here is a sample in schema format:

attributetype ( NAME ‘project-IsGroup’
DESC ‘Whether a principal refers to a group or a user’
EQUALITY caseExactIA5Match

The number scheme is designed to be universally unique and is one of those things that has a portion assigned by a central server, and a portion defined by the end company.  The SYNTAX keyword references one of the syntax strings defined in this document:

The above attributetype definition uses ,’ , the syntax for IA5, a character set that is “not-quite-ascii”.  The EQUALITY keyword references a method that requires the input be validated by that syntax.  Our ldif files are sloppy, in that many of the attributetype definition use syntaxes other than the one above, but still specify EQUALITY types that are IA5 based.  Iy suspect this is a case of MS doing something deliberately broken….

Our objecttype definitions seem to be OK, although we reference a SUP (supertype) of container that doesn’t seem to be defined in the OpenLDAP schema.

LDAP Development Setup

To set up debugging on the slapd, in slapd.conf:

loglevel -1

in syslog.conf:

local4.* /var/log/slap.log

To create self signed certificates:  I created a makefile:

.SUFFIXES :.cert .csr .key

TARGET = host

all : ${TARGET}.cert

${TARGET}.key :
openssl genrsa -des3 -out $@ 1024
cp $@ $
openssl rsa -in $ -out $@
rm $

.key.csr :
openssl req -new -key $< -out $@

.csr.cert :
openssl x509 -req -days 365 -in $< -signkey $*.key -out $@

clean :
rm -rf *.cert *.csr *.key *~

Added to slapd.conf.  I was not able to get self signed certificates to work yet.

#TLSCACertificateFile /etc/openldap/certs/host.csr
TLSCertificateFile /etc/openldap/certs/host.cert
TLSCertificateKeyFile /etc/openldap/certs/host.key
TLSVerifyClient never

added to ~/.ldaprc


Sample code to test the connection. Does not do a query.

int result;
LDAP * ldap;
int version  = LDAP_VERSION3;
const char * host = “ldap://10=”;
int port = 389;
const char * binddn =  “cn=Administrator,dc=application,dc=company,dc=int”;
const char * bindpw =  “secret”;

result = ldap_initialize(&ldap , host);
if ( result  != LDAP_SUCCESS){
cerr <<  __LINE__<< ” failed ” <<ldap_err2string(result) << endl;

/* always default to LDAP V3 for TLS*/
result =  ldap_set_option(ldap, LDAP_OPT_PROTOCOL_VERSION, &version);
if ( result  != LDAP_SUCCESS){
cerr <<  __LINE__<< ” failed ” <<ldap_err2string(result) << endl;

result = ldap_start_tls_s( ldap, NULL, NULL );
if ( result  != LDAP_SUCCESS){
cerr <<  __LINE__<< ” failed ” <<ldap_err2string(result) << endl;

for (int failures=0; failures<10; failures++)
result = ldap_simple_bind_s(ldap, binddn, bindpw);
if (LDAP_SERVER_DOWN != result)

if (LDAP_SUCCESS != result)
cerr <<  “LDAP: ldap_simple_bind_s() failed ” <<ldap_err2string(result) << endl;

Command line to test TLS setup.   Returns many results in my setup:

ldapsearch -H “ldap://” -LLL -x -w secret -D “cn=Administrator,dc=application,dc=company,dc=int”  -b  ‘dc=application,dc=company,dc=int’ ‘(objectclass=*)’  -ZZ

My Ideal Technology Setup for work

“Since I’m dreaming, I’d like a pony” –Susie, in Calvin and Hobbes.

“I’m not just the President of the Hair Club for Men, I’m also a client.” –President of the Hair Club for Men

Not only do I write software, I use it. A whole bunch. I am a Linux guy, and when ever I end up in a situation where I have to work around a proprietary solution that just doesn’t make sense for what I am trying to do, it ads a point or two to my Diastolic. So here is my dream setup:

First of all, I’d like a company that doesn’t use Exchange. I want to be able to talk to my mail server, and my calendar server using standards based protocols. I want email based notifications of meeting. GMail does nice job of sending them via SMS. I don’t want to have to run Outlook to book a conference room. Is there such a solution out there? Probably. I haven’t looked, but I am sure I could find many. Let’s start with Exim for mail, and then find a decent Calendaring program that talks email and iCal. Squirrelmail is a clunky webmail client, so I hope we could find something better, but it still beats Exchanges webmail. Basically, I want to run the Mozilla tools on my desktop.  An I want to sync mail to my Palm based Cell phone without installin Goodlink.

Ubuntu Linux as the default developer install.  I have never been so happy with my desktop as I am now.  Debian package management rules, and Ubuntu desktop support is the best I’ve worked with in Linux.  Of course, I would probably be just as happy with Fedora, but haven’t used it in a while.  Most of my Red Hat work has been with Red Hat Enterprise and that is a fine, stable server architecture, but doesn’t suite my needs for a developer work station…too stable.

I want an internet proxy that allows me to talk to ports other than 80 and 443. Yes I realize this is a configuration issue. I want to be able to SSH in and out, and check in and out from public CVS, Subversion, and Git repositories. I want to be able to hit a website on port 8080.

For Revision Control, I would like to use Either Subversion or Git. Probably Git mixed with Quilt. Note that this is not just for software, but also document preparation. For document preparation: Open Office across the board. It does what I need for Presentations, Spreadsheets, and Word Processing.

A single unified indexing system for all of the companies information system.

All emails sent to public mailing lists should be indexed.

Blog sites for individual developers. Word Press seems to work nicely, but Drupal was good, too.  The more publically available the information in a development environemnt, the bettwer it supports community type development.

Decent IDE support for Refactoring. Refactoring support it baseline, regardless of language. I was incredibly productive using Eclipse for Java, but have not been able to get it to work well for our C++ projects. XRefactory looks promising, but I haven’t tried it yet. Slick Edit wasn’t able to handle our source setup, either. But whatever we chose, the team needs to support it.

A commitment to open source software.  There are few things more motivating to a developer than knowing that the effort they put into learning a code base will not get flushed when they leave the company…and you might even use this as a way to hire new talent, too.

Simple LDAP client code using deprecated API

This query returns all objects.

#include <stdio.h>

#define LDAP_DEPRECATED    1

#include <ldap.h>

const  char *base = “dc=application,dc=company,dc=com”;
char loginDN[] =”cn=Manager,dc=virtualcenter,dc=vmware,dc=int”;

#define TRACE()  printf(“%s:%d\n”, __FILE__,__LINE__)

/*TODO free memory*/
void handle_search_result(LDAP *ldap,  LDAPMessage *result){
char  *attr;
LDAPMessage *entry;
BerElement *ber;
char **values;

printf(“Entries =  %d\n”, ldap_count_entries( ldap, result ));

for (entry = ldap_first_entry(ldap, result);
entry != NULL;
entry = ldap_next_entry(ldap, result))
printf(“dn=(%s)\n”,ldap_get_dn( ldap, entry ));

for (attr = ldap_first_attribute(ldap, result, &ber);
attr != NULL;
attr = ldap_next_attribute(ldap, result, ber))

values = ldap_get_values(ldap, entry, attr);
int   i =0 ;
int len =  ldap_count_values(values);
for (i =0; i < len; ++i){
result = ldap_next_message(ldap, result);

int execute(){
LDAP *ldap;
char *uri = “ldap://localhost”;
int rc = ldap_initialize(&ldap, uri);
if (LDAP_SUCCESS != rc){
printf(“error:%s\n”,  ldap_err2string(rc));
return rc;

rc =  ldap_bind_s(ldap, loginDN, “vmware”, LDAP_AUTH_SIMPLE );
if (LDAP_SUCCESS != rc) goto error;

char *attrs;
LDAPMessage *res = 0;
rc =  ldap_search_s (ldap,base,LDAP_SCOPE_CHILDREN
if (LDAP_SUCCESS != rc) goto error;

printf(“number of results = %d\n”,ldap_count_messages(ldap,res));
int attid = 0;
char * attr;

handle_search_result(ldap,  res);

goto cleanup;
printf(“error:%s\n”,  ldap_err2string(rc));
return rc;

return  0;

int main(){

return execute() ? -1 : 0;


OpenLDAP API is somewhat hostile

After a few days beating my head against OpenLDAP’s C API, the only thing I can think is that the developers don’t want you to use it.

All of the simple functions, the functions that are in all of the sample code out there, has been deprecated:

  1. ldap_bind and it’s ilk are gone with the exception of ldap_sasl_bind and ldap_sasl_bind_s.
  2. ldap_init, and ldap_open are gone, although ldap_initialize still exists.

What seems to be going on is an attempt to push people to use SASL: The (not so) Simple Authentication and Security Layer that abstracts away the differences between Kerberos, Radius, and other full fledged authentication and authorization mechanisms.

The problem here is that the most common  application for using LDAP is for authenticating users.  It doesn’t matter that you “shouldn’t” do this according to the established literature.  If you are stuck talking to Active DIrectory, LDAP is your primary tool.

The LDAP_Result api makes sense from an object oriented perspective, but man, is that a lot of function calls just to iterate through a list.  For example, to process the effect of calling ldap_sasl_bind you have to call:

  1. ldap_result
  2. ldap_first_message
  3. ldap_parse_result
  4. ldap_free for each value allocated by ldap_parse_result
  5. ldap_next_message
  6. ldap_freemsg

The problem here is they are trying to be objected, but stopping short of actually returning vtables.  I suspect I could just look at the LDAPMessage structure and get the same result.

I think the choice of SASL is somewhat suspect as well.  I would much prefer it if they used PAM.  If you are going to make use of a flexible authentication mechanism, go with the dominant one.

I admit that I am not an expert on LDAP things, and that perhaps I am missing something.  But trying to get the C equivalent of

ldapsearch -H “ldap://localhost” -LLL -x -b  ‘dc=application,dc=domain,dc=com’ ‘(objectclass=*)’

Should not take several days.

Code to Convert a MAC address to IPv6

The following code will convert an ethernet MAC address to the comparable  IPv6 link only address.

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <stdarg.h>

int main(int argc, char** argv){

int addrlen = strlen(“0000:0000:0000:0000:0000:0000:0000:”);
char* out = malloc(addrlen);
char * outorig = out;

memset(out, 0, addrlen);

char* addr =    “00:0c:29:20:4e:e3″;

if (argc > 1){
addr = argv[1];
fprintf(stderr,”usage %s macaddr\n”,argv[0]);

int len = strlen(addr);

if (len > 18){
printf (“String too long\n”);

int i ;
int col_count = 0;
unsigned char current = 0;
for (i = 0; i < len; ++i){
char c = addr[i];

if (0 == c){
}else if (‘:’ == c){
switch( col_count ){
case 0:{
out += strlen(out);

unsigned short c2 = ( current | 0x02 );
if (c2 == current){
c2 = current & 0xcf;
out += strlen(out);
case 2:{
out += strlen(out);
out += strlen(out);

if (col_count % 2){
out += strlen(out);
current = 0;
}else if ((c >= ‘a’) && (c <= ‘f’)){
current *=16;
current += ( 10 + c – ‘a’);
}else if ((c >= ‘A’) && (c <= ‘F’)){
current *=16;
current += ( 10 + c – ‘a’);
}else if ((c >= ‘0’) && (c <= ‘9’)){
current *=16;
current += ( c – ‘0’);

out += strlen(out);
return 0;


Interview Question for Distributed Computing

This is an updated version of my interview question,  made non-bproc like.

Using only the following API:

  • printf(…)
  • int get_node_count() // number of compute nodes attached to the head node.  0 means no other nodes
  • int get_current_node()// 0 for the head node, 1-n for the compute nodes.
  • int remote_fork(int node) // like fork, but returns an fd to the child/parent process
  • void send_long_sync(int fd, long value)//send and wait, blocks until receipt
  • long recv_long_sync(int fd)//block until value is available
  • long gettime()

Calculate the average clock skew on the cluster.    Return 0 on success, and -1 on any failures.

Assume that all nodes are up and running.  This is a c subset of c++.  Each of these functions throw an exception upon failure.  rfork has the same semantics as fork:  when it returns, there are two copies of the program running, just on separate machines.  The  next line of code to execute will be the line immediately after the fork on both machines.  However, the returned value is not a process ID, and the parent process  does not need to wait for the remote processs to finish:  the child process is automatically reaped by init.

Faking out PAM Authentication

I am working on a server application that uses Pluggable Authentication Modules (PAM) for authentication support.  This application  must run as root. As part of development, people need to log in to this server.  I don’t want to give out the root password of my development machine to people.  My hack was to create a setup in /etc/pam.d/emo-auth that always allows the login to succeed, provided the account exists.  The emo-auth configuration is what the application looks up to authenticate network connections.

$ cat /chroot/etc/pam.d/emo-auth

account   required
auth      required
session   required

Now people login with root, and any password will allow them to get in.
Since this is only for development, this solution works fine, and does not require any code changes.


Don’t hit publish on the blog when you just want to save a draft.

Big Builds are Bad. Software should be developed and distributed in small packages. Linux is successful due to things like apt, yum, and yast.

Interface Specifications need to be more specific.  Just saying that something is a string is not really helpful if that something needs to conform to a pattern.

Programming and blogging requires sugar in the brain.

Interviews are tricky…on both sides of the table. Career fairs are worse.

C++ Has a lot of magic in it. Can we make type level programming more transparent?

Microsoft purchasing Yahoo would be good for Google, but bad for just about everyone else.

Being a Dad is really cool. Even when it sucks, it is great. Sometimes kids refuse to go to sleep. This leads to sleep deprivation, but also leads to really wonderful moments in rocking chair in the middle of the night.

Pool is a great Geek game. Lower left-hand English is neat.

Snowshoes are good off the trail. Not so good on the trail. If your going on the trail, take the cross country skis. Snowmobiles smell funny.

New Hampshire winter weather is still as brutal today as it was when I left the area in the early ’90s.

It is hard to sing a Jazzy version of Old MacDonald had a Farm.  It is harder to do after the tenth repetition while trying to get a child to fall asleep.
If you listen to Children’s CDs long enough, you will develop favorite children’s songs. I like the hippo song.

Is there really a difference between the Ethernet and SCSI protocols? I don’t know, but it would be fun to find out.

The compiler is your friend. Let it check your work for you.

Why write code on a white board if you have a computer available? Especially if you have an overhead projector?

Where do the local peregrine falcons sleep? Where would they be sleeping if we hadn’t built up the whole area?

If I could have a redo on which language to take as a Sophomore, I would probably would have liked to take Chinese. Russian and Arabic would also do. German was not a good choice for me.

If Bush Senior had insisted on pushing to Baghdad, it would have been my generation in this mess as opposed to the current set of junior officers. Instead of Haiti, I would have gone to Basra or something.

There are too many interesting topics in the world to pursue them all, or even a small fraction of them.

Every philosopher I’ve read, especially the ones I disagree with, ave said something that is valuable and true.

No matter how old you are, when you get together with your parents, you revert to teenager status.

This list should never see the light of day.

High Availability and dealing with failures

Time to use my pubic forum to muddle through some design issues I’m struggling to lay straight.

A Data center is made up of several objects: Servers ( computers, usually horizontal), racks(hold the computers), switches(a network device that connects two or more computers together), power sources, and cables (both electric and network cables, to include fiber optic for storage devices). A server in the data center can serve on or more roles: storage host, computation host, administration, or user interface. If an application is hosted in a data cetner, it is usually important enough that it requires some guarantee of availability. This application will resided on a computation host, and require access to other types of hosts. An email server stores the received messages in a storage host, probably connected to the computation host via fiber optics. It receives and sends messages via a network connection that goes to the outside world. It may also talk to a User Interface machine that runs a web server and an application that allows web access to email. If the computation host loses connectivity with either the public network or the storage host, it cannot process mail. If the web server loses connectivity to the mail server, certain users cannot access their mail.

There are many reasons that connectivity can fail. The major links in the chain are: OS failure, Network interface card (NIC) failure, bad cable, disconnected cable, bad switch, unplugged switch, switch turned off. Once you pass the switch, the same set of potential problems exist on to the other host. To increase reliability, a network will often have two switches, and each server will have two NICs, one plugged into each switch. The same set up goes for storage, although different technologies are used. As a general rule, you don’t want to have things running in hot standby mode. It is a waste of resources, and it doesn’t get tested until an emergency hits. Thus, the double network connectivity usually gets set up also as a way to double bandwidth. Now if one of the cables breaks, that server merely operates in a degraded mode. The second cable has been passing network traffic already, now it just gets all of it.

A typical data center has many machines. Typical server loads are very low, sometimes in the 1-3% range of overall capacity. Thus, if a machine fails, a data center often has plenty of servers that could absorb the load from the failed server. Figuring out how to cross load services in a data center has been a major topic in the IT world over the past decade. This goes by many names, one of which is grid computing. I’ll use that term myself here. There are several problems any grid system has to solve, but most can be clumped under the term application provisioning. This means getting all of the resources together that a given application requires so that they available on the computation host. These resources include the network and storage connections described above, as well as the executables, data files, licenses, and security permissions required to run the application.

When a host fails, some remote monitoring system needs to act. First, it needs to know that the host has failed. This is typically performed through a heartbeat sensor. This is a simple network communication sent by the computation host saying “I’m still alive.” Cue Mike McCready. When a heartbeat fails, the monitor needs to make sure that the application is up online somewhere as soon as possible. Now, the reason the heartbeat failed might have been because of a problem on the heartbeat network, and the application is actually up and running just fine. An advanced solution is to test the system through some alternative method. In the case of the email server, it may be to connect to the email port and send a sample message. This delays the restart of the application, but may minimize downtime.

Sometimes, two copies of the applications can’t run at the same time. In this case, you have to be sure that the original one is gone. To achieve this, you shut off the original server. This is called “Shoot the other node in the head.” or STONITH. Sometimes the word node is replaced with guy and you get STOGITH. If you do this incorrectly, you may take yourself down, a situation referred to as SMITH. Or you take all servers down, and this is called SEITH. But I digest…

Here’s the part that I am currently trying to decide. If an application depends on a resource, and that resource fails, you can bring the application up on a different server. It will take a non-trivial amount of time (estimate it a minute) to shut down the old instance and bring up the new instance. If, on the other hand, the disconnect is temporary, we can have minimal down time by just waiting for the network to come back up. If someone disconnects a cable by accident, that person can just plug the cable back in. If the network is redundant, removing one cable may result in degraded performance, but it may not.
If the failure is due to the switch being down, just selecting another host connected to the same switch will result in downtime and a situation that is no better than the original. If the problem is the storage host being down, there probably is nothing you can do to recover outside of human or divine intervention.

If a switch goes down but there is another set of hosts on a different switch, you can migrate applications to the new host. But you may end up overloading the new switch. This is referred to as the stampeding herd effect. If the lost switch is degrading performance for all applications dependent on that switch, you best strategy is to migrate a subset of applications to balance the load. After each application is moved, recheck the network traffic to determine if you’ve done enough. Or done too much.

A malfunctioning NIC or switch might manifest in intermittent connectivity.  In this case, you want to get the application off of the original server and on to a new server.  The problem is in distinguishing this from the case where the cable just got unplugged once, and then plugged back in.  From the server’s perspective, the network is gone, and then it is back.  This leads to a a lot of questions. What interval, and how many iterations do you let go by before you decide to bail from that server?  If you have redundancy, does the failing connection impact the user’s experience, or does proper routing ensure that they have seamless connectivity?