If you are getting a message of  “Invalid Openstack Identity Credentials” when talking to one of the OpenStack services, it might be due to a recent changeover from UUID tokens to PKI tokens. Here are some troubleshooting tips.

The PKI based token validation scheme relies on two certificates from Keystone that are fetched via HTTP and stored in a local directory.  The location for this directory is specified by the configuration variable signing_dir.  In your services configuration file is a section that looks like this:

[keystone_authtoken]
signing_dir = /var/cache/glance/api
auth_uri = http://127.0.0.1:5000/
auth_host = 127.0.0.1
auth_port = 35357
auth_protocol = http
admin_tenant_name = service
admin_user = glance
revocation_cache_time = 1

If your service does not have this stanza, the defaults are specified in the file keystoneclient/middleware/auth_token.py. If no value is specified for this directory it defaults to a secure temporary directory. Initialization code for the service checks that the directory exists and is a writable directory. If it does not, it attempts to create it. If this fails, the whole service fails to start. However, it often succeeds, but later problems get in the way.

The first thing to check is that the signing_dir does, in fact, exist. If it does, check for the presence of the certificate files inside there:

$ ls -la /var/cache/glance/api/
total 24
drwx------. 2 ayoung root   4096 Jul 22 10:58 .
drwxr-xr-x. 4 root   root   4096 Nov  7  2012 ..
-rw-r-----. 1 ayoung ayoung 1424 Jul 22 10:58 cacert.pem
-rw-r-----. 1 ayoung ayoung   15 Jul 22 10:58 revoked.pem
-rw-r-----. 1 ayoung ayoung 4518 Jul 22 10:58 signing_cert.pem

There should be three files in this directory. Two are certificates, and the third is the token revocation list. If they are not present, that is a sign that your service cannot fetch them from Keystone. To troubleshoot, first attempt to talk to Keystone to make sure it is serving out the files:

curl http://localhost:35357/v2.0/certificates/signing

That should fetch over the signing certificate, the start of which which looks like:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=Unset, L=Unset, O=Unset, CN=www.example.com
        Validity
            Not Before: Jul 22 14:57:31 2013 GMT
            Not After : Jul 20 14:57:31 2023 GMT
        Subject: C=US, ST=Unset, O=Unset, CN=www.example.com

Note the expiration dates of the certificate:

            Not Before: Jul 22 14:57:31 2013 GMT
            Not After : Jul 20 14:57:31 2023 GMT

The token revocation list is updated frequently (once a minute) but the certificates are not. One possible problem is that the certificates are the wrong files, or are garbage. You can remove all three of these files and execute another command against your server, as they are fetched on demand.

Your Keystone Log should show the access of the certificate files. You might have to turn up your logging levels: set debug = True and verbose = True in your Keystone config file and restart the Keystone server.

(keystone.common.wsgi): 2013-07-24 12:18:11,461 DEBUG wsgi __call__ arg_dict: {}
(access): 2013-07-24 12:18:11,462 INFO core __call__ 127.0.0.1 - - [24/Jul/2013:16:18:11 +0000] "GET http://localhost:35357/v2.0/certificates/signing HTTP/1.0" 200 4518

If the files are not present in your directory after this, it is likely one of three issues:

1. Your service is misconfigured and cannot talk to Keystone. Check the auth_port and auth_host values and make sure you can talk to that service via curl as above
2. Your signing dir is not writable. chmod it so that the service (POSIX) user can write to it. Check via su and touch
3. SELinux policy is denying access to the directory

SELinux troubles often come from using Fedora/RHEL based packages, but choosing configuration options that do not match the standard policy. setenforce permissive and see if that makes a difference. If so, you probably need to relabel the directory. If you are using a directory under /var/cache/ try:

sudo restorecon -R /var/cache/

If you are not using a /var/cache subdir, you should be. Modify the signing_dir configuration value for your service and restart.

Set back to setenforce enforcing in order to confirm you changes have solved the problem.

If your certificates are being fetched on demand, the PKI validation is working properly. Most likely, the token from Keystone is not valid for the operation you are attempting to perform, and your user needs a different role for the operation.

UPDATE:

revocation_cache_time defaults to 0 in some older builds. It is worthwhile setting this value to some non-trivial value. 60 makes sense, as that means to cache token revocation for one minute; a reasonable duration, yet still conservative value. If set to 0, the file is not cached at all, and will be requested on every token validation. This will increase load on the Keystone server.

Note: Special thanks to Eoghan Glynn for editing.