For authorization, we need a user name. For authentication, we need a tenancy and a set of roles that the user has in the tenancy.
Authorization cannot be for ever, so we need an expiration date/time. Here is a simple JSON representation of that data.

{"user": "ayoung",
"tenant": "coop-city",
"role": ["hallmonitor, groundskeeper"],
"expiry":"18JUL2012"
}

Note that it has the Roles listed in it. By cryptographically verifying this document, not only does it remove the need to validate the token with Keystone, but it removes the need to go back to keystone to fetch the roles. Thus, authentication and authorization information are linked in one document.

The following assumes you have a signing certificate. See the steps here to generate one if you need it.

To Sign auth_token.json run:

cmsutil -S -d alias -N ayoung -i auth_token.json -o auth_token.p7s

The signed message is put into auth_token.p7s, a file extension that alludes to the PKCS 7 standard.

This includes all of the certificates. Use a tool to view the contents in human readable form:

 /usr/lib64/nss/unsupported-tools/derdump -i auth_token.p7s

The output is big. To simply read the message back, use:

cmsutil  -D -i auth_token.p7s  -d alias/ -h1

This gives back the additional file, along with the validated signer:

SMIME: 	level=1.2; type=signedData; nsigners=1;
		signer0.id="Adam Young"; signer0.status=GoodSignature;
	level=1.1; type=data;
{"user": "ayoung",
"tenant": "coop-city",
"role": ["hallmonitor, groundskeeper"],
"expiry":"18JUL2012"
}

To simply check if the document is valid, drop the h1 argument. If to test it against an NSS database that does not have the CA cert in it, you get:

# cmsutil  -D -i auth_token.p7s  -d altdb

signer 0 status = SigningCertNotTrusted
cmsutil: problem decoding: Peer's Certificate issuer is not recognized.

To make this usable as a replacement for the current token scheme, we could use something like Base64 encoding:

[ayoung@ayoung cmstest]$ base64 auth_token.p7s
MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAaCAJIAEbXsidXNl
ciI6ICJheW91bmciLCAgCiJ0ZW5hbnQiOiAiY29vcC1jaXR5IiwgIAoicm9sZSI6IFsiaGFsbG1v
bml0b3IsIGdyb3VuZHNrZWVwZXIiXSwKImV4cGlyeSI6IjE4SlVMMjAxMiIKfQoAAAAAAACgggIG
MIICAjCCAWugAwIBAgICCSkwDQYJKoZIhvcNAQEFBQAwFDESMBAGA1UEAxMJTXkgSXNzdWVyMB4X
DTEyMDUxNTIwNDcxNFoXDTEyMDgxNTIwNDcxNFowUzELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAk1B
MREwDwYDVQQHEwhXZXN0Zm9yZDEPMA0GA1UEChMGUmVkSGF0MRMwEQYDVQQDEwpBZGFtIFlvdW5n
MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC7lNCEwdZMHIClZJ9f84R1tfgEjtZV2H+Nx1OR
eqeK+WYvVXSVJ61g/Vl3ponv8L/qRiqbuyP7Nfr7ogfM2OJuDOiWsyfHJSIS29NVOb5f6+oPw39Q
+fN7NSssTEr6UFz4d1mXl85iJXK+K7x5TGFQowA6po102YCj6tjesGPODQIDAQABoyQwIjAgBgNV
HSUBAf8EFjAUBggrBgEFBQcDBAYIKwYBBQUHAwIwDQYJKoZIhvcNAQEFBQADgYEAt105m7XYBugD
y+PXYq7R2XaAdyJGCBl4yq3Xz3qmfYmpP8wVTNy5j2dacuopq+W4DB0CeGo5+sYGAiRSgF8vNx6/
wMt3+EExdJO+IJl9QMRB6ooRzSMJAIH3b3jXOWln9TV6AHLHER8NWoNZq3moSYp4fO8tUVgerDKJ
7TdBP/IxggFaMIIBVgIBATAaMBQxEjAQBgNVBAMTCU15IElzc3VlcgICCSkwCQYFKw4DAhoFAKCB
lzAYBgkqhkiG9w0BCQMxCwYJKoZIhvcNAQcBMCMGCSqGSIb3DQEJBDEWBBRBECaXMyVVMoW2yonU
kWwriSqs5TApBgkrBgEEAYI3EAQxHDAaMBQxEjAQBgNVBAMTCU15IElzc3VlcgICCSkwKwYLKoZI
hvcNAQkQAgsxHKAaMBQxEjAQBgNVBAMTCU15IElzc3VlcgICCSkwDQYJKoZIhvcNAQEBBQAEgYAV
UCVEsgfo3SK5p5ZF98uWgoIjmyCksS7mULJMT/ZJZJqIyTVY6AmXWCgxzKOHqrzWCansSilYAYQr
aeEgCYjX9GvSxHFmCQQzYM2buzJr5GjoPZJ0osOulUzWX8Nzq5Y51DAcWtU8c+06Ezbu5q5YjBno
2fTEWwYcemr9m/SA7AAAAAAAAA==

The break is due to how it is displayed in the browser, but that is really one long stream of characters. The value can be sent just like the current tokens, in the -X-Auth-Token header. Now a remote service can validate the token by reversing the Base64 ENCODING AND using cmsutil or the corresponding APIs.

cat auth_token.p7s.base64  | base64 -d |  cmsutil  -D -i auth_token.p7s -d alias/

There are still a few wrinkles to iron out.

• The token is a little longer than it needs to be, due to the certificate being embedded. For Keystone, remote systems would be better off fetching the certificate once from the Keystone server, and just matching the serial number in the CMS token.
• The current scheme of using the token in the URL would probably not work very well with the long Base64 encoding above, but that system has other known issues, and is likely to be replaced shortly with a scheme that will instead put the token into the body of the document. However, even that step will be unnecessary with certificate based verification.
• Signing documents from Python currently requires calling out to additional processes. The Python-NSS maintainer is aware of the issue, and is looking into making a Python-appropriate set of bindings for the CMS calls in Python-NSS.
• Since there is no revocation of the tokens, their lifespan should be kept short. 10-30 minutes is probably appropriate. The default value will require some consensus
• Adding roles for a user will require getting an additional token. The old token will still be valid,but will not allow the user to do any actions that require the new role.
• The code that checks roles does not currently know about the token. It will require a small amount of code changes to make sure that the token is available to this code.
• While this scheme is backwards compatible with the current auth scheme, the reverse is not true. Short of checking the length of the token, there is no way to confirm that a token is a cryptographically signed document as opposed to the alpha-numeric cookie in current usage. One approach for transition may be to test reading the document, and, upon failure, fall back to the online protocol.

Update: Revocation will be discussed in an additional article.

In order for the message signature to be verifiable, we generate a pair of keys, one that is kept private, and one that is made public. Data encrypted with the private key can then be decrypted with only the corresponding public key. So, to sign a message, it is encrypted with the private key.

There are ways to make this more efficient, such as signing a hash of the message, but the principal is the same.

I am going to use the same set of cryptographic tools that Mozilla provides for securing web traffic. These tools fall under then name Network Security Services or NSS for short.

The first step is to provide a Certificate Authority (CA) Certificate. This is a certificate used to sign other certificates. This is actually supposed to be part of a chain. In the browser, there are a list of CA certs that are accepted by default. For our purposes here, I am going to self sign one. Figuring out how to set this up for your organization has been left as an exercise to the reader.

We need a place to put the certificate. For NSS, this is called an NSS database. I am going to create one in a directory I call alias, because that is the name of the NSS database used in Apache HTTPD. To generate:

mkdir alias
certutil -N -d ./alias/

Now create a self-signed CA certificate. Type the command:

certutil -S -s "CN=My Issuer" -n myissuer -x -t "C,C,C" -1 -2 -5 -m 1234 -d alias/

You will be prompted to type. This is used to generate entropy, or randomness, for the underlying cryptography.

Then you will be prompted to select aspects of the certificate. A menu of options like this:

		0 - Digital Signature
		1 - Non-repudiation
		2 - Key encipherment
		3 - Data encipherment
		4 - Key agreement
		5 - Cert signing key
		6 - CRL signing key
		Other to finish

Type each number in order. 0 1 2 3 4 5 6. The menu will reappear after each.

Yes, it is annoying. I didn’t write the tool.

then type any number to finish

So the number 9 works.

 9
Is this a critical extension [y/N]?
y
Is this a CA certificate [y/N]?
y
Enter the path length constraint, enter to skip [<0 for unlimited path]: >
Is this a critical extension [y/N]?
y

Not done yet: Next menu.

		0 - SSL Client
		1 - SSL Server
		2 - S/MIME
		3 - Object Signing
		4 - Reserved for future use
		5 - SSL CA
		6 - S/MIME CA
		7 - Object Signing CA
		Other to finish

For this we want 5 6 7 then again pick 9 to exit.

 > 9
Is this a critical extension [y/N]?
y

The security conscious folks out there are squirming in their seats reading this, because a self-signed CA cert is BAD under most circumstances. However, this is proof-of-concept type stuff. You always need to get a certificate signed by a CA…this is just the simple way to do it for development.

Now that we have a CA cert, we need to generate a Key. With NSS, you can generate the Key and the certificate signing request in one step:

 certutil -R -s "CN=Adam Young, O=RedHat , L=Westford, ST=MA, C=US" -p "617-555-1212" -o mycert.req -d alias

You see the same entropy generation prompt as before.

You can list the keys using:

certutil -K -d ./alias/

Which produces output along the lines of:

certutil: Checking token "NSS Certificate DB" in slot "NSS User Private Key and Certificate Services"
< 0> rsa      99f6d153a007f8fb2d79312d454749eb92e4db99   NSS Certificate DB:myissuer
< 1> rsa      b8472a65af1c2da900fb0eb953d9c278d615a580   NSS Certificate DB:ayoung

Note that there is one in there for the CA certificate we produced before.

To sign the key run the following:

 certutil -C -m 2345 -i mycert.req -o mycert.crt -c myissuer -d ./alias/ -6

Here is the menu you are presented:

		0 - Server Auth
		1 - Client Auth
		2 - Code Signing
		3 - Email Protection
		4 - Timestamp
		5 - OCSP Responder
		6 - Step-up
		Other to finish

The import options are 1 and 3, as we are going to use the same mechanism as we would use to sign email. The private and public keys are held in the NSS database. OK, we finally have a useful certificate and key pair that we can use to sign a document. To Sign auth_token.json run:

cmsutil -S -d alias -N ayoung -i signme.txt -o signed.p7s

The signed message is put into signed.p7s. The file extension alludes to the PKCS 7 standard that CMS grew out of.

Kerberos communicates over port 88 in order to secure tickets that will be used for authentication and (to a limited degree) authorization on other servers.  This communication is done using the Kerberos wire protocols.  First the user requests a ticket granting ticket (TGT).  Later, the user will use that TGT to get a ticket specific to the service they are trying to talk to.

The connection to get the TGT is a two way street.  There are multiple messages and back and forth as both sides performs steps necessary to prove to the other that they are indeed,  who they claim to be.  This cannot be done over straight HTTP without serious reworking,  but it can be done using a techonology called websockets.  With websockets,  the connection between client and server remains open, and HTTP becomes a tunneling protocol for TCP.

Thus, the two things we need to do to get a TGT are:

1.  Place the Key Distribution Center (KDC)  behind a websockets Proxy.
2. Create a modified version of the kinit program that can talk websockets,  or build a client side proxy.

There are other approaches, of course.  FreeIPA can request a TGT on the server on behalf of the user.  The user could download this TGT, and a Browser plugin could even stick it into the Kerberos Credentials Cache.  But this bypasses the Kerberos ideal of the users password never crossing the wire.

We could write the kinit code into the browser itself, using either a native extension or Javascript,  but it is a complicated enough of a protocol that forking from the C code would be a hefty maintenance effort.

The second step is trickier:  getting the service ticket.  This action is performed for the user automatically when visiting a web site that is protected by, for example, mod_auth_krb.  The web server responds to a request with a 401 and enough information that the browser knows to try Kerberos…by way of two wrapper libraries called SASL and GSSAPI in turn.  Unfortunately,  this is a pipeline that is pretty much welded shut:  without modifying  either the browser or the Kerberos libraries,  there is no way to automatically request and inject a service ticket on behalf of the user.

It would be possible for the user to do it explicitly, though.  That might be an OK stop-gap measure.  Again, a custom client of some sort could fetch the service ticket and push it into the Kerberos credential cache.

I think the most straight forward path would be to :

• Extend the Kerberos libraries to be able to talk Web Sockets as a tunnel for the Kerberos wire protocol.
• Allow a krb5.conf option that will specify that the KDC is reachable via websockets.  This would be something along the lines of an krb5.conf line

kdc = wss://kerberos.younglogic.com/kdc

• This value should also be available as a SRV record out of DNS:

NAME: _kerberos._wss

TYPE: SRV:

VALUE:wss://kerberos.younglogic.com/kdc

As a simple proof of concept,  a websocket proxy can run on the client, listening on localhost:88 and talking via Websockets to the KDC.  Then the krb5.conf file can point to the local proxy instead of the KDC Websocket proxy.

Websockets is a simple enough protocol that writing it into the Kerberos code base should not be prohibitive.  If desired,  it could be done as a separate shared object for the general consumption of the system.

It is not necessary to run the TGT fetching code over wss,  the SSL version of web sockets, as the handshake already sets up encryption.  But there is no drawback to running the whole thing over SSL and it will mask any KDC traffice that is not encrypted.

An additional advantage is that now kpasswd can be run on the same port as the rest of the KDC based services, just on a different URL.

Since Websockets is supported by all the major browsers,  making browser plugins to communicate with the KDC is viable.

There might well be other issues of scale:  protecting against DOS attacks and the like, but there is a path forward to a secure WebSSO.

To date,  my involvement with Openstack has focused primarily on Keystone.  Using that as a starting point,  I want to continue it in two directions: deeper work on the identity management side, and expending HTTPD support beyond Keystone.

First is working on the Public Key Infrastructure (PKI) alternative to the Keystone Token architecture. In addition to implementing that blueprint,  I have agreed to work with gyee and liemmn on making sure that the PKI work and the Key based authentication they are working on fit into an understandable narrative about the options available in Openstack.

Of course,  my PKI work assumes web server support for PKI,  which, shortest path, is using HTTPD.  There is  code from that example that I need to review and commit.  In order for it to be usable,  I need to set up and run a proof of concept system where Glance, Nova and Horizon all use Keystone  from HTTPD.

Once I’ve made that work,  I need to add support to devstack for it.

A follow on set of tasks will involve HTTPD-ifying  Nova and Glance.  To prep to site for this work,  we are trying to establish a n URL naming scheme for Openstack.

While testing the EPEL packages of Openstack, I noticed that we lacked support for the VNC viewer noVNC.  I am working with Anthony Young (no relation to me) to come up with a supportable packing format that will be accepted by the Debian and Fedora packaging standards.

Longer term,  I want to be able to serve the noVNC javascript from Apache HTTPD, and talk back to HTTPD as well.  There are two alternatives here:

1. Use an Apache Websockets module and do it all in native code
2. Add Websockets support to mod_wsgi.

The Google code approach uses mod_python. However, mod_python is not actively maintained, so I think this code is not going to work long haul. Making it work in mod_wsgi has hurdles to overcome.  This leads me to favor the native approach as of now.  I have the Apache websocket module built as an RPM, and will continue toward integrating the VNC proxy in C as an additional module.

I created an annotation with a processor that creates a new class, one the contains a single static instance of an array of the parameter names from the constructor.

All code on this page released under the same license as the OpenJDK Libraries: GPL with the classpath exception.

First, the annotation itself.

package com.younglogic.annote;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

@Target(ElementType.CONSTRUCTOR)
@Retention(RetentionPolicy.RUNTIME)
public @interface NamedParameters {

}

Next the annotation processor. This uses the javac API, which really should become part of the Base Java platform. In order to compile I added /usr/lib/jvm/java-1.7.0/lib/tools.jar to the classpath.

package com.younglogic.annote;

import java.io.BufferedWriter;
import java.io.IOException;
import java.util.Set;

import javax.annotation.processing.AbstractProcessor;
import javax.annotation.processing.RoundEnvironment;
import javax.annotation.processing.SupportedAnnotationTypes;
import javax.annotation.processing.SupportedSourceVersion;
import javax.lang.model.SourceVersion;
import javax.lang.model.element.Element;
import javax.lang.model.element.PackageElement;
import javax.lang.model.element.TypeElement;
import javax.tools.JavaFileObject;

import com.sun.tools.javac.code.Symbol.ClassSymbol;
import com.sun.tools.javac.code.Symbol.MethodSymbol;
import com.sun.tools.javac.code.Symbol.VarSymbol;

@SupportedAnnotationTypes(value = { "com.younglogic.annote.NamedParameters" })
@SupportedSourceVersion(SourceVersion.RELEASE_7)
public class NamedParametersAnnotationProcessor extends AbstractProcessor {

	@Override
	public boolean process(Set annotations,
			RoundEnvironment roundEnv) {
		for (TypeElement element : annotations) {
			for (Element constructor : roundEnv
					.getElementsAnnotatedWith(element)) {

				ClassSymbol classSymbol = (ClassSymbol) constructor
						.getEnclosingElement();
				PackageElement packageElement = (PackageElement) classSymbol
						.getEnclosingElement();

				JavaFileObject jfo;
				try {
					jfo = processingEnv.getFiler().createSourceFile(
							classSymbol.getQualifiedName() + "ParameterNames");
					BufferedWriter bw = new BufferedWriter(jfo.openWriter());
					bw.append("package ");
					bw.append(packageElement.getQualifiedName());
					bw.append(";");
					bw.newLine();
					bw.newLine();
					bw.append("public class " + classSymbol.getSimpleName()
							+ "ParameterNames {");
					bw.newLine();
					bw.append("    static String[] instance = {");
					MethodSymbol symbol = (MethodSymbol) constructor;
					int i = 0;
					for (VarSymbol param : symbol.getParameters()) {
						if (i++ > 0) {
							bw.append(",");
						}
						bw.newLine();
						bw.append("            \"" + param.name + "\"");
					}
					bw.newLine();
					bw.append("        };");
					bw.newLine();
					bw.append("    }");
					bw.newLine();
					bw.close();
				} catch (IOException e) {
					// TODO Auto-generated catch block
					e.printStackTrace();
				}

			}

		}
		return true;
	}
}

A sample class that uses this annotation. Note that this class is immutable.

package com.younglogic.annotetest;

import com.younglogic.annote.NamedParameters;

public class SampleClass {

	final String s1;
	final String s2;
	final Integer i1;

	@NamedParameters
	public SampleClass(String s1, String s2, Integer i1) {
		super();
		this.s1 = s1;
		this.s2 = s2;
		this.i1 = i1;
	}

	@Override
	public String toString() {

		return "SampleClass values are: s1=" + s1 + ", s2=" + s2 + ", i1=" + i1.toString();
	}
}

Now add some parameters to the compiler.

-processor is the processor we want to explicitly call
-processorpath tells it where to find the new annotation.
-s tells it where to put the generated code

 javac -cp ../NamedParameters/bin/  -processorpath ../NamedParameters/bin/ -s /tmp/generated/ -processor com.younglogic.annote.NamedParametersAnnotationProcessor   src/com/younglogic/annotetest/SampleClass.java

This will generate the source file: /tmp/generated/com/younglogic/annotetest/SampleClassParameterNames.java

package com.younglogic.annotetest;

public class SampleClassParameterNames {
    public static String[] instance = {
            "s1",
            "s2",
            "i1",
            "i2"
        };
    }

And compile it to /tmp/generated/com/younglogic/annotetest/SampleClassParameterNames.class

We can use that information to create new instances. I have a file SampleClass.properties with the values I want to use to create a new instance.

i1=5
i2=10
s1="first"
s2="next"

Read it in and populate a new instance dynamically:

package com.younglogic.annotetest;

import java.io.IOException;
import java.io.InputStream;
import java.lang.reflect.Constructor;
import java.lang.reflect.InvocationTargetException;
import java.util.Properties;

public class UseParamNames {

	public static void main(String[] args) throws IOException,
			InstantiationException, IllegalAccessException,
			IllegalArgumentException, InvocationTargetException, NoSuchMethodException, SecurityException {
		Properties properties = new Properties();

		InputStream instream = UseParamNames.class
				.getResourceAsStream("SampleClass.properties");
		properties.load(instream);

		Constructor c = getConstructor();

		Object[] initargs = new Object[SampleClassParameterNames.instance.length];
		int i = 0;
		for (String name : SampleClassParameterNames.instance) {
			Class[] argTypes = { String.class };

			String strvalue = properties.getProperty(name);
			String[] paramArgs = new String[1];
			paramArgs[0] = strvalue;

			initargs[i] = c.getParameterTypes()[i].getConstructor(argTypes)
					.newInstance(paramArgs);
			i++;

		}

		Object newInstance = c.newInstance(initargs);

		System.out.println(newInstance.toString());
	}

	private static Constructor getConstructor() {
		for (Constructor c : SampleClass.class.getConstructors()) {
			if (c.getAnnotations().length > 0) {
				return c;
			}

		}
		return null;
	}
}

Which, when run, produces the output:

SampleClass values are: s1="first", s2="next", i1=5

When creating a new Object in Java,  the parameter list for the constructor or factory provides the contract.  Unfortunately,  a key element of this contract is missing at run rime when it is needed most.  If a constructor takes multiple values of the same type,  the only way to distinguish between them are the names of the parameters.  For example,  the JDBC API to create a new connection takes three parameters: the username and password to use for the connection authentication and authorization, and the URL of the Datasource.  However, all three values are strings. In order to distinguish between them,  the user needs to know the order of the parameters:  is it URL first or UserID first?  

Building a new object from a persisted dictionary is common enough that there is a different API as well that takes a Properties object.  This variation loses all documentation about what is required to authenticate a connection: username and password.  Since the format of the URL is also Driver specific,  the user is thus relegated to digging through the documentation to figure out how to correctly create a Connection.  However, reading the documentation is not an option for an automated tool.

What Java lacks is a standardized and automated way to map a key-value pair collection to a function.  The mechanism should look like this.

private static Object createInstanceFromProperties(Method method, Properties properties)
throws InstantiationException, IllegalAccessException,
       InvocationTargetException, NoSuchMethodException {
    //This is the new call.
    String[] paramNames = method.getParameterNames();
    //Spacing is due to limitation of web display for angle bracketed expressions.
    Class < ? > [] argsTypes = method.getParameterTypes();

    Object arguments[] = new Object[argsTypes.length];
    for (int i = 0; i < paramNames.length; i += 1) {
        String val = properties.getProperty(paramNames[i]);
        if (argsTypes[i] == java.lang.String.class) {
	    arguments[i] = val;
        }else{
            arguments[i] = argsTypes[i].getConstructor(java.lang.String.class).newInstance(val);
	}
    }
    return method.invoke(null, arguments);
}

A simple variation can work with a Constructor object instead of a Method object. One assumption from the above code that should be explicit is that the framework using this call has enough information to determine which variation of the Method or Constructor to find.

I wouldn't ordinarily implement an API like this as Parallel arrays, but in this case, it is the least impact on the existing API. There is no "Parameter" object in the existing API.

Since existing compiled classes would not have provided the array of names, one version of Java should be provided as a transitory step which will allow the JDK to return a Null if the class does not provide the appropriate strings.

Update: in testing keystone, you need to change the username from demo to admin as per the export lines near the bottom of this page.

A prerequisite is to set up NSS HTTPD SSL support. We can use it in conjunction with Keystone to provide encrypted access for authentication and authorization.

I created a file to enable wsgi:

/var/www/cgi-bin/keystone/keystone.py

import os
from keystone import config
from paste import deploy
from keystone.common import logging
from keystone.common import utils
from keystone.common import wsgi

LOG = logging.getLogger(__name__)
CONF = config.CONF
config_files = ['/opt/stack/keystone/etc/keystone.conf']
CONF(config_files=config_files)

conf = CONF.config_file[0]
#name = 'admin'
name = os.path.basename(__file__)

if CONF.debug:
    CONF.log_opt_values(logging.getLogger(CONF.prog), logging.DEBUG)

options = deploy.appconfig('config:%s' % CONF.config_file[0])

application = deploy.loadapp('config:%s' % conf, name=name)

And I hardlinked it:

cd /var/www/cgi-bin/keystone/
ln  keystone.py admin
ln  keystone.py main

This is based on a Fedora devstack deployment, which checks out the code in /opt/stack/keystone. I ran against master from git, which as of this writing should be the same as the next Essex RC. I ran python setup.py install with this, which gave me the installed egg in

/usr/lib/python2.7/site-packages/keystone-2012.2-py2.7.egg.

Since eggs deployed this way are part of the default Python path, it is usable by WSGI applications in Apache.

To run against the Fedora RPMs, the difference would be to read in the config file /etc/keystone/keystone.conf instead.

I’ve changed /etc/httpd/conf.d/keystone.conf to

WSGIScriptAlias /main /var/www/cgi-bin/keystone/main
WSGIScriptAlias /admin /var/www/cgi-bin/keystone/admin


   NSSRequireSSL



   NSSRequireSSL

and restarted httpd. To test:

The most straightforward way to test Keystone is via the command line interface (CLI). When running via devstack, the normal approach is to source the environment file:

. openrc

which pulls in the localrc. This sets many values, including the URL to be used when the Keystone CLI talks to the server: OS_AUTH_URL. Typically this is set as some variation on the local hostname and the port 5000. It also sets the username to demo To test HTTPD, change this to:

export OS_AUTH_URL=https://$HOSTNAME/admin/v2.0
export OS_USERNAME=admin

However, running the command line client gave an error:

[ayoung@ayoungstack devstack]$ keystone user-list
No handlers could be found for logger "keystoneclient.client"
Invalid OpenStack Identity credentials

Looking in the access log, I saw the token request was succeeding (returned 200) but the second request failed with unauthorized (401).

One difference between Eventlet and the Fedora build of HTTPD is that each request in HTTPD is handled by a separate process. This is called prefork, and you can test a given install by running

[ayoung@ayoung keystone]$ httpd -l
Compiled in modules:
  core.c
  prefork.c
  http_core.c
  mod_so.c

To see the built in modules. The default way that Keystone stores Tokens is by using a key value pair storage. You see this in the keystone.conf file.

[token]
driver = keystone.token.backends.kvs.Token

By changing this to the SQL store, the tokens are now accessable by other processes, including the ones that will handle follow on requests to Keystone.

[token]
driver = keystone.token.backends.sql.Token

And now the keystone CLI works:

[ayoung@ayoungstack devstack]$ keystone user-list
+----------------------------------+---------+--------------------+--------+
|                id                | enabled |       email        |  name  |
+----------------------------------+---------+--------------------+--------+
| 6d42897928974d469a4223302e6bae4a | True    | admin@example.com  | admin  |
| 72f2941c7e3e40b292d50bd8b1662899 | True    | demo@example.com   | demo   |
| 92556c2d36de4f46a0a9916d96d2e793 | True    | glance@example.com | glance |
| e452d5f649ed489c8b84b8192e781263 | True    | nova@example.com   | nova   |
+----------------------------------+---------+--------------------+--------+

First, the application needs the additional directive in its /etc/httpd/conf.d specific file.

   NSSVerifyClient require

Did that work for you? NO…me neither…

OK, so actually, it was slightly more involved. First, I had to figure out how to turn on NSS debugging. That is done by editing the file

/etc/httpd/conf.d/nss.conf

and setting the value for

LogLevel debug

Once that worked, I saw messages in /var/log/httpd/error_log that looked like:

[Fri Mar 30 11:43:54 2012] [debug] nss_engine_kernel.c(309): Changed client verification type will force renegotiation
[Fri Mar 30 11:43:54 2012] [info] Requesting connection re-negotiation
[Fri Mar 30 11:43:54 2012] [debug] nss_engine_kernel.c(404): Performing full renegotiation: complete handshake protocol
[Fri Mar 30 11:43:54 2012] [debug] nss_engine_kernel.c(418): Re-negotation request failed: returned error -12176

So I enabled renegotiation. Again, editing /etc/httpd/conf.d/nss.conf

NSSRenegotiation on
NSSRequireSafeNegotiation on

And then client side certs worked for me. They probably won’t work for you, though.

I cheated.

In between, I set up a Dogtag PKI instance, generated a new server side certificate, a client side certificate for my browser, and installed the CA and Server side certificate in my web server.

I don’t think all of that was necessary.

A variation would have been to generate a client side certificate and sign it with the self-sign CA key that my web server was already using. The bottom line is that the browser and the server need to both trust the CAs used for the client and server certs.

Ok, so we’ve established a secure channel. This doesn’t mean much if we don’t know who the user is at the application level. The NSS layer follows the SSL approach of using the option: FakeBasicAuth to pass information from the Certificate through to the application. I uncommented the line:

NSSOptions +FakeBasicAuth +ExportCertData +CompatEnvVars +StrictRequire

You also need to tell NSS which value from the certificate should be passed as the user. In my case, it is the Subject:uid value, so I configured mine as:

NSSUserName SSL_CLIENT_S_DN_UID

The overall difference in the nss.conf file is:

--- /etc/httpd/conf.d/nss.conf.orig	2012-03-29 12:59:06.319470425 -0400
+++ /etc/httpd/conf.d/nss.conf	2012-03-30 16:21:24.836124822 -0400
@@ -17,7 +17,7 @@
 # Note: Configurations that use IPv6 but not IPv4-mapped addresses need two
 #       Listen directives: "Listen [::]:8443" and "Listen 0.0.0.0:443"
 #
-Listen 8443
+Listen 443

 ##
 ##  SSL Global Context
@@ -71,17 +71,17 @@
 #
 # Only renegotiate if the peer's hello bears the TLS renegotiation_info
 # extension. Default off.
-NSSRenegotiation off
+NSSRenegotiation on

 # Peer must send Signaling Cipher Suite Value (SCSV) or
 # Renegotiation Info (RI) extension in ALL handshakes.  Default: off
-NSSRequireSafeNegotiation off
+NSSRequireSafeNegotiation on

 ##
 ## SSL Virtual Host Context
 ##

-
+

 #   General setup for the virtual host
 #DocumentRoot "/etc/httpd/htdocs"
@@ -92,7 +92,7 @@
 # LogLevel is not inherited from httpd.conf.
 ErrorLog /etc/httpd/logs/error_log
 TransferLog /etc/httpd/logs/access_log
-LogLevel warn
+LogLevel debug

 #   SSL Engine Switch:
 #   Enable/Disable SSL for this virtual host.
@@ -198,7 +198,8 @@
 #   o OptRenegotiate:
 #     This enables optimized SSL connection renegotiation handling when SSL
 #     directives are used in per-directory context.
-#NSSOptions +FakeBasicAuth +ExportCertData +CompatEnvVars +StrictRequire
+NSSOptions +FakeBasicAuth +ExportCertData +CompatEnvVars +StrictRequire
+NSSUserName SSL_CLIENT_S_DN_UID
 
     NSSOptions +StdEnvVars
 

Now a little bit of Python/WSGI specifics
For my WSGI application, I add in a value that says “pass on the authentication info to the application”

WSGIPassAuthorization On

Then the WSGI app can access the value via:

class Application(BaseApplication):
    @webob.dec.wsgify
    def __call__(self, req):
    context['remote_user'] = req.environ.get('REMOTE_USER')

Here is what my file /etc/httpd/conf.d/keystone.conf looks like to protect the suburl “main”:

WSGIScriptAlias /main /var/www/wsgi/httpdmain.py
WSGIScriptAlias /admin /var/www/wsgi/httpdadmin.py

< Location "/main" >
   NSSRequireSSL
   NSSVerifyClient require
   WSGIPassAuthorization On
< / Location >

sudo yum install mod_nss

/etc/httpd/alias/ is populated already with ca and server cert self signed
/etc/httpd/conf.d/nss.conf already exists
change 8443 to 443 in two places

--- /etc/httpd/conf.d/nss.conf.orig	2012-03-29 12:59:06.319470425 -0400
+++ /etc/httpd/conf.d/nss.conf	2012-03-29 12:19:38.862721465 -0400
@@ -17,7 +17,7 @@
 # Note: Configurations that use IPv6 but not IPv4-mapped addresses need two
 #       Listen directives: "Listen [::]:8443" and "Listen 0.0.0.0:443"
 #
-Listen 8443
+Listen 443

 ##
 ##  SSL Global Context
@@ -81,7 +81,7 @@
 ## SSL Virtual Host Context
 ##

-
+

 #   General setup for the virtual host
 #DocumentRoot "/etc/httpd/htdocs"

Make sure your firewall is open on the HTTPS port. Add the following line in /etc/sysconfig/iptables

-A INPUT -m state --state NEW -m tcp -p tcp --dport 443 -j ACCEPT

before the statement

-A INPUT -j REJECT --reject-with icmp-host-prohibited

and restart the services

sudo systemctl restart iptables.service
sudo systemctl restart httpd.service

The documentation provides a lot more detail. Almost all of these steps are performed by the RPM install on F16 and later.

There are numerous uses for this style of booting.  A short list:

• Debugging the installation processes of software packages
• Running computationally intensive tasks on a large array of nodes
• Inventorying the hardware no new servers
• Deploying a light management framework for virtualization hypervisors

Here is a brief overview of the pieces needed to set this up for testing purposes on a workstation running KVM.

• Configure a local network for PXE traffic
• Install a machine to server as the PXE server
• Install and Configure DHCPD
• Install and Configure TFTP
• create a rootfs
• Install HTTPD
• get a boot image
• PXE Boot a Blank VM

The first piece is done on the workstation itself, and is just overhead for doing this ins a completely virtualized environment. This is the analogue of using a private secured backbone for administrative backbone when using physical machines. Configuring the network used to take editing the Network XML file, but now you can deselect the DHCP checkbox in virt-manager when creating a new network. Here is what mine generated in /var/lib/libvirt/network/nodhcp.xml.

  nodhcp
  ec819c42-b4b0-1577-363e-1c5918a3ee29
  
  
  
  

You could save that in /root and do

 sudo virsh network-create /root/nodhcp.xml

This one has a bridge defined, which is probably not what you would want for your server, as this network should not see the outside world. Still the device is useful if you want to run wireshark on it to debug booting issues.

Create the server as a VM, and provide it Network interface on both a publicly accessible as well as the Private network defined above. Install Fedora (I used F17 Alpha) as per normal.

Install the Server pieces

yum install syslinux tftp-server  dhcp  dracut-network  httpd httpd-tools apr apr-util apr-util-lda

Configure DHCPD by editing /etc/dhcp/dhcpd.conf

allow booting;
allow bootp;

subnet 172.22.33.0 netmask 255.255.255.0 {
        option routers                  172.22.33.1;
        option subnet-mask              255.255.255.0;
        option domain-search              "younglogic.com";
        option domain-name-servers       172.22.33.1;
        option time-offset              -18000;     # Eastern Standard Time
        range 172.22.33.100 172.22.33.140;
}

class "pxeclients" {
   match if substring(option vendor-class-identifier, 0, 9) = "PXEClient";
   next-server server-ip;
   filename "pxelinux.0";

Set up TFTP.

edit /etc/xinetd.d/tftp and set

	disable			= no

Copy over the Kernel

cp /boot/vmlinuz-3.3.0-0.rc6.git0.2.fc17.x86_64 /var/lib/tftpboot/vmlinuz-3.3.0-0.rc6.git0.2.fc17.x86_64

Dracut is a great tool for people that need to control the boot procedure of their machines. It has a module called Livenet that we use to specifying that the root file system should be downloaded an unpacked in a directory. In order for that to work, we need the dracut-network package specified on the yum line above. THen, we need to generate a new initial ramdisk (initrd) with the livenet module enabled.

dracut -v  --force --add "livenet"  /var/lib/tftpboot/initramfs-kernel-3.3.0-0.rc6.git0.2.fc17.x86_64.img 3.3.0-0.rc6.git0.2.fc17.x86_64

Syslinux is our boot loader. Syslinux is more than a tool to run an installer (although it does that quite well). Here, it is a program that is configured much as for a kickstart, but it will instead switch to running the kernel instead of Anaconda.

cp  /usr/share/syslinux/pxelinux.0 /var/lib/tftpboot
mkdir /var/lib/tftpboot/pxelinux.cfg/

The configuration for syslinux is in /var/lib/tftpboot/pxelinux.cfg/default and looks like this

default f17

serial --unit=0 --speed=9600
terminal --timeout=5 serial console

label f17
  kernel vmlinuz-3.3.0-0.rc6.git0.2.fc17.x86_64
  append initrd=initramfs-kernel-3.3.0-0.rc6.git0.2.fc17.x86_64.img console=tty0  rd.shell rd.debug root=live:http://172.22.33.2/squashfs.img   rw

The most interesting value here is root=live:http://172.22.33.2/squashfs.img which is what activates the Dracut module in the initrd to download the rootfs via http and expand it into the new root directory.

I know that a couple of these values are incorrect, and I will update when I have them straight, but this will work.

Once everything is in place, you need to make sure SELinux is happy. Otherwise, the Kernel will deny the tftp network service access to the files it needs to serve. Of course, you could always disable SELinux, but lets try to do things right To see what the SELinux context for the newly copied file is:

ls -Z /var/lib/tftpboot/vmlinuz-3.3.0-0.rc6.git0.2.fc17.x86_64
rwxr-xr-x. root root unconfined_u:object_r:tftpdir_rw_t:s0 /var/lib/tftpboot/vmlinuz-3.3.0-0.rc6.git0.2.fc17.x86_64

To see what it should be:

[root@f17stack ~]# matchpathcon /var/lib/tftpboot/
/var/lib/tftpboot	system_u:object_r:tftpdir_rw_t:s0

To set the whole directory to the correct context

 chcon -Rv --type=tftpdir_rw_t  --user=system_u  /var/lib/tftpboot/

There is probably a more correct way to do this, something along the lines of

restorecon -R -v /var/lib/tftpboot/

But that doesn’t seem to set the user.

At this point you can restart both services. Make sure that dhcpd runs at machine reboot as well.

/bin/systemctl enable  dhcpd.service
/bin/systemctl restart xinetd.service
/bin/systemctl restart dhcpd.service

To test that things are running correctly:

[root@f17stack tmp]# cd /tmp
[root@f17stack tmp]# getenforce
Enforcing
[root@f17stack tmp]# tftp localhost -c get initramfs-kernel-3.3.0-0.rc6.git0.2.fc17.x86_64.img
[root@f17stack tmp]# ls initramfs-kernel-3.3.0-0.rc6.git0.2.fc17.x86_64.img
initramfs-kernel-3.3.0-0.rc6.git0.2.fc17.x86_64.img

Make sure Apache is running and will run on boot:

[root@f17stack tmp]# systemctl enable  httpd.service
[root@f17stack tmp]# systemctl start httpd.service

The final step is to setup a bootable image. In this case, we use the same thing that the live CD uses, which is packaged inside of a squashfs image.

I did this on my desktop to keep from having to have too big a file on my Server VM. As it is, this is still close to a 588MB image.

sudo mount -o loop  ~/Downloads/Fedora-16-x86_64-Live-Desktop.iso /mnt
scp /mnt/LiveOS/squashfs.img root@f17stack.ayoung.boston.devel.redhat.com:/var/www/html

And again, on the server, lets check SELinux

[root@f17stack tmp]# ls -Z /var/www/html/
f16/          index.html    squashfs.img
[root@f17stack tmp]# ls -Z /var/www/html/squashfs.img
-r--r--r--. root root unconfined_u:object_r:httpd_sys_content_t:s0 /var/www/html/squashfs.img

And make SELinux Happy:

chcon -Rv --type=httpd_sys_content_t  --user=system_u  /var/www/html/

I have to admit, I had already gotten the system to relable the file by the time I tested this, I suspect upon a reboot, so this last step might not be necessary.

Now, open the firewall to let dhcp, tftp, and http traffic through. This can be done via the following rules added to /etc/sysconfig/iptables

-A INPUT -m state --state NEW -m tcp -p tcp --dport 68 -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --dport 68 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 69 -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --dport 69 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT

And restarting iptables:

systemctl start iptables.service

Now Boot the second VM. If all goes well, it will Get its IP address:

There will likely be a delay as it downloads the 588 install image:

And when it is done, you are at Firstboot:

Thanks to Will Woods for cluing me in about squashfs and livenet, Harald Hoyer for general Dracut help, and Ray Strode for help with debugging why my console wouldn’t echo.