Openstack Keystone in HTTPD

After calling for Keystone to migrate to HTTPD, several people asked me if I would show how this can be done. Here are the steps.

Update 2; See a more recent article here: which is based on the info below. The below article goes into more detail. This is a moving target, and I would recommend reading both articles.

Update: in testing keystone, you need to change the username from demo to admin as per the export lines near the bottom of this page.

A prerequisite is to set up NSS HTTPD SSL support. We can use it in conjunction with Keystone to provide encrypted access for authentication and authorization.

I created a file to enable wsgi:


import os
from keystone import config
from paste import deploy
from keystone.common import logging
from keystone.common import utils
from keystone.common import wsgi

LOG = logging.getLogger(__name__)
CONF = config.CONF
config_files = ['/opt/stack/keystone/etc/keystone.conf']

conf = CONF.config_file[0]
#name = 'admin'
name = os.path.basename(__file__)

if CONF.debug:
    CONF.log_opt_values(logging.getLogger(CONF.prog), logging.DEBUG)

options = deploy.appconfig('config:%s' % CONF.config_file[0])

application = deploy.loadapp('config:%s' % conf, name=name)

And I hardlinked it:

cd /var/www/cgi-bin/keystone/
ln admin
ln main

This is based on a Fedora devstack deployment, which checks out the code in /opt/stack/keystone. I ran against master from git, which as of this writing should be the same as the next Essex RC. I ran python install with this, which gave me the installed egg in


Since eggs deployed this way are part of the default Python path, it is usable by WSGI applications in Apache.

To run against the Fedora RPMs, the difference would be to read in the config file /etc/keystone/keystone.conf instead.

I’ve changed /etc/httpd/conf.d/keystone.conf to

WSGIScriptAlias /main /var/www/cgi-bin/keystone/main
WSGIScriptAlias /admin /var/www/cgi-bin/keystone/admin



and restarted httpd. To test:

The most straightforward way to test Keystone is via the command line interface (CLI). When running via devstack, the normal approach is to source the environment file:

. openrc

which pulls in the localrc. This sets many values, including the URL to be used when the Keystone CLI talks to the server: OS_AUTH_URL. Typically this is set as some variation on the local hostname and the port 5000. It also sets the username to demo To test HTTPD, change this to:

export OS_AUTH_URL=https://$HOSTNAME/admin/v2.0 
export OS_USERNAME=admin

However, running the command line client gave an error:

[ayoung@ayoungstack devstack]$ keystone user-list
No handlers could be found for logger "keystoneclient.client"
Invalid OpenStack Identity credentials

Looking in the access log, I saw the token request was succeeding (returned 200) but the second request failed with unauthorized (401).

One difference between Eventlet and the Fedora build of HTTPD is that each request in HTTPD is handled by a separate process. This is called prefork, and you can test a given install by running

[ayoung@ayoung keystone]$ httpd -l
Compiled in modules:

To see the built in modules. The default way that Keystone stores Tokens is by using a key value pair storage. You see this in the keystone.conf file.

driver = keystone.token.backends.kvs.Token

By changing this to the SQL store, the tokens are now accessable by other processes, including the ones that will handle follow on requests to Keystone.

driver = keystone.token.backends.sql.Token

And now the keystone CLI works:

[ayoung@ayoungstack devstack]$ keystone user-list
|                id                | enabled |       email        |  name  |
| 6d42897928974d469a4223302e6bae4a | True    |  | admin  |
| 72f2941c7e3e40b292d50bd8b1662899 | True    |   | demo   |
| 92556c2d36de4f46a0a9916d96d2e793 | True    | | glance |
| e452d5f649ed489c8b84b8192e781263 | True    |   | nova   |

14 thoughts on “Openstack Keystone in HTTPD

  1. I set up all the things and when testing with command line client as suggested within the article I have such error:
    $ keystone user-list
    No handlers could be found for logger “keystoneclient.client”
    Authorization Failed: An unexpected error prevented the server from fulfilling your request. (OperationalError) attempt to write a readonly database u’INSERT INTO token (id, expires, extra) VALUES (?, ?, ?)’
    My apache2 web server is running as root – could it be a reason? I checked owner of the keystone.db and it’s good, file permissions also looks good. The keystone service is run as ‘user’ user not root and if I use http mode everything works very well. I tried also to use suexec in apache to run cgi script under keystone user ‘user’ but it didn’t helped. Any clues?

  2. I just saw the same thing as well, but with Devstack, with the error coming from Horizon.

    HTTPD should not be running as root. If you run it with the init script/ssytemd, it should be running as the user ‘nobody’ or ‘apache’ depending on your setup. Mine runs as apache.

    That said, it does sound like a permissions problem. Perhaps the issue is that the mod_wsgi call to the DB layer is coming from a user not trusted by MySQL.

  3. OK I ran apache2 as ‘user’ user and the problem with read only DB disappeared and keystone works perfectly over ssl. Thanks.
    But now I try to configure it with Swift all over ssl. Did you try it maybe? When I tried “swift stat – v” command I can see in logs that token was successfully retrieved from keystone over ssl but later swift isn’t able to talk with keystone properly over ssl via apache. In apache logs I can see that there is a request for resource with path “/var/www/v2.0” which is not found of course.
    When I tried normal http mode (without apache in between) everything works well. I know it is not a subject of this article but maybe you have some experience in it?

  4. I installed Openstack with Devstack.
    But, in compute node, “fail” happens.
    Repeated words are “Invalid OpenStack Identity credentials”.
    I want to know Why.

    + [[ 0 -ne 0 ]]
    + [[ =~ openvz ]]
    + KERNEL=
    + RAMDISK=
    + UNPACK=
    + case “$IMAGE_FNAME” in
    + ‘[‘ cirros-0.3.0-x86_64-uec ‘!=’ cirros-0.3.0-x86_64-uec.tar.gz ‘]’
    + IMAGE_NAME=cirros-0.3.0-x86_64-uec
    + xdir=/home/glory/devstack/files/images/cirros-0.3.0-x86_64-uec
    + rm -Rf /home/glory/devstack/files/images/cirros-0.3.0-x86_64-uec
    + mkdir /home/glory/devstack/files/images/cirros-0.3.0-x86_64-uec
    + tar -zxf /home/glory/devstack/files/cirros-0.3.0-x86_64-uec.tar.gz -C /home/glory/devstack/files/images/cirros-0.3.0-x86_64-uec
    ++ for f in ‘”$xdir/”*-vmlinuz*’ ‘”$xdir/”aki-*/image’
    ++ ‘[‘ -f /home/glory/devstack/files/images/cirros-0.3.0-x86_64-uec/cirros-0.3.0-x86_64-vmlinuz ‘]’
    ++ echo /home/glory/devstack/files/images/cirros-0.3.0-x86_64-uec/cirros-0.3.0-x86_64-vmlinuz
    ++ break
    ++ true
    + KERNEL=/home/glory/devstack/files/images/cirros-0.3.0-x86_64-uec/cirros-0.3.0-x86_64-vmlinuz
    ++ for f in ‘”$xdir/”*-initrd*’ ‘”$xdir/”ari-*/image’
    ++ ‘[‘ -f /home/glory/devstack/files/images/cirros-0.3.0-x86_64-uec/cirros-0.3.0-x86_64-initrd ‘]’
    ++ echo /home/glory/devstack/files/images/cirros-0.3.0-x86_64-uec/cirros-0.3.0-x86_64-initrd
    ++ break
    ++ true
    + RAMDISK=/home/glory/devstack/files/images/cirros-0.3.0-x86_64-uec/cirros-0.3.0-x86_64-initrd
    ++ for f in ‘”$xdir/”*.img’ ‘”$xdir/”ami-*/image’
    ++ ‘[‘ -f /home/glory/devstack/files/images/cirros-0.3.0-x86_64-uec/cirros-0.3.0-x86_64-blank.img ‘]’
    ++ echo /home/glory/devstack/files/images/cirros-0.3.0-x86_64-uec/cirros-0.3.0-x86_64-blank.img
    ++ break
    ++ true
    + IMAGE=/home/glory/devstack/files/images/cirros-0.3.0-x86_64-uec/cirros-0.3.0-x86_64-blank.img
    + [[ -z cirros-0.3.0-x86_64-uec ]]
    + ‘[‘ ” = bare ‘]’
    + KERNEL_ID=
    + ‘[‘ -n /home/glory/devstack/files/images/cirros-0.3.0-x86_64-uec/cirros-0.3.0-x86_64-vmlinuz ‘]’
    ++ glance –os-auth-token d769e9c1d8014781a90504473550aae7 –os-image-url image-create –name cirros-0.3.0-x86_64-uec-kernel –public –container-format aki –disk-format aki
    ++ grep ‘ id ‘
    ++ get_field 2
    ++ read data
    Request returned failure status.
    Invalid OpenStack Identity credentials.
    + KERNEL_ID=
    + ‘[‘ -n /home/glory/devstack/files/images/cirros-0.3.0-x86_64-uec/cirros-0.3.0-x86_64-initrd ‘]’
    ++ grep ‘ id ‘
    ++ glance –os-auth-token d769e9c1d8014781a90504473550aae7 –os-image-url image-create –name cirros-0.3.0-x86_64-uec-ramdisk –public –container-format ari –disk-format ari
    ++ get_field 2
    ++ read data
    Request returned failure status.
    Invalid OpenStack Identity credentials.
    + glance –os-auth-token d769e9c1d8014781a90504473550aae7 –os-image-url image-create –name cirros-0.3.0-x86_64-uec –public –container-format ami –disk-format ami
    Request returned failure status.
    Invalid OpenStack Identity credentials.
    ++ failed
    ++ local r=1
    ++ set +o xtrace failed: full log in /opt/stack/logs/

  5. hi i got the following error
    No handlers could be found for logger “keystoneclient.client”
    Authorization Failed: Unable to communicate with identity service: [Errno 111] Connection refused. (HTTP 400)
    what may be its reason

  6. Sounds like maybe you are talking on the wrong port. Could also be firewall. If you are going against default HTTP, it should be port 443, not 5000 or 35357

  7. I want to know after these steps, is keystone api also moved to apache? I want to generate token use keystone-api ,and after these steps is the url ‘http://localhost:5000/admin/v2.0/tokens ?

  8. 1. I checked out keystone source from GIT.

    2. Copied keystone/httpd/wsgi-keystone to /etc/httpd/conf.d/wsgi-keystone

    3. Copied keystone/httpd/ to /usr/local/www/wsgi-scripts/

    4. Inside /etc/httpd/conf.d/wsgi-keystone modified following lines :-
    WSGIScriptAlias /keystone/main /usr/local/www/wsgi-scripts/
    WSGIScriptAlias /keystone/admin /usr/local/www/wsgi-scripts/
    So that we directly use the (WSGI Application)

    5. Restarted httpd.
    6. The /etc/httpd/conf/httpd.conf is left unmodified
    7. Try running curl -vs fails with error

  9. im getting “Target WSGI script ‘/var/www/cgi-bin/keystone/main’ cannot be loaded as Python module”
    while configuring federation for keystone any help

    [info] [client] mod_wsgi (pid=25248, process=’keystone’, application=’|’): Loading WSGI script ‘/var/www/cgi-bin/keystone/main’.
    [error] [client] mod_wsgi (pid=25248): Target WSGI script ‘/var/www/cgi-bin/keystone/main’ cannot be loaded as Python module.
    [error] [client] mod_wsgi (pid=25248): Exception occurred processing WSGI script ‘/var/www/cgi-bin/keystone/main’.
    [error] [client] Traceback (most recent call last):
    [error] [client] File “/var/www/cgi-bin/keystone/main”, line 21, in
    [error] [client] config.configure()
    [error] [client] File “/usr/lib/python2.7/dist-packages/keystone/common/”, line 694, in configure
    [error] [client] help=’Do not monkey-patch threading system modules.’))
    [error] [client] File “/usr/lib/python2.7/dist-packages/oslo/config/”, line 1579, in __inner
    [error] [client] result = f(self, *args, **kwargs)
    [error] [client] File “/usr/lib/python2.7/dist-packages/oslo/config/”, line 1731, in register_cli_opt
    [error] [client] raise ArgsAlreadyParsedError(“cannot register CLI option”)
    [error] [client] ArgsAlreadyParsedError: arguments already parsed: cannot register CLI option

  10. Hi , this is a very nice guide on how to configure the wsgi server for keystone. Would these configuration be valid for Ubuntu on Kilo ? Thanks.

  11. Roughly, but Ubuntu puts things in different locations, names things slightly differently, and thus you need to make the Ubuntu specific changes. I don’t know what those are, but many people do…ask around.

Leave a Reply

Your email address will not be published. Required fields are marked *