Official TripleO Keystone Images

My recent forays into running containerized Keystone images have been based on a Centos base image with RPMs installed on top of it. But TripleO does not run this way; it runs via containers. Some notes as I look into them.

The official containers for TripleO are currently hosted on docker.com. The Keystone page is here:

Don’t expect the docker pull command posted on that page to work. I tried a comparable one with podman and got:

$ podman pull tripleomaster/centos-binary-keystone
Trying to pull docker.io/tripleomaster/centos-binary-keystone...
  manifest unknown: manifest unknown
Trying to pull registry.fedoraproject.org/tripleomaster/centos-binary-keystone...

And a few more lines of error output. Thanks to Emilien M, I was able to get the right command:

$ podman pull tripleomaster/centos-binary-keystone:current-tripleo
Trying to pull docker.io/tripleomaster/centos-binary-keystone:current-tripleo...
Getting image source signatures
...
Copying config 9e85172eba done
Writing manifest to image destination
Storing signatures
9e85172eba10a2648ae7235076ada77b095ed3da05484916381410135cc8884c

Since I did this as a normal account, and not as root, the image does not get stored under /var, but instead goes somewhere under $HOME/.local. If I type

$ podman images
REPOSITORY                                       TAG               IMAGE ID       CREATED        SIZE
docker.io/tripleomaster/centos-binary-keystone   current-tripleo   9e85172eba10   2 days ago     904 MB

I can see the short form of the hash starting with 9e85. I copy that to use to match the subdir under ls /home/ayoung/.local/share/containers/storage/overlay-image

ls /home/ayoung/.local/share/containers/storage/overlay-images/9e85172eba10a2648ae7235076ada77b095ed3da05484916381410135cc8884c/

If I cat that file, I can see all of the layers that make up the image itself.

Trying a naive: podman run docker.io/tripleomaster/centos-binary-keystone:current-tripleo I get an error that shows just how kolla-centric this image is:

$ podman run docker.io/tripleomaster/centos-binary-keystone:current-tripleo
+ sudo -E kolla_set_configs
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
ERROR:__main__:Unexpected error:
Traceback (most recent call last):
  File "/usr/local/bin/kolla_set_configs", line 412, in main
    config = load_config()
  File "/usr/local/bin/kolla_set_configs", line 294, in load_config
    config = load_from_file()
  File "/usr/local/bin/kolla_set_configs", line 282, in load_from_file
    with open(config_file) as f:
IOError: [Errno 2] No such file or directory: '/var/lib/kolla/config_files/config.json'

So I read the docs. Trying to fake it with:

$ podman run -e KOLLA_CONFIG='{}'   docker.io/tripleomaster/centos-binary-keystone:current-tripleo
+ sudo -E kolla_set_configs
INFO:__main__:Validating config file
ERROR:__main__:InvalidConfig: Config is missing required "command" key

When running with TripleO, The config files are generated from Heat Templates. The values for the config.json come from here.
This gets me slightly closer:

podman run  -e KOLLA_CONFIG_STRATEGY=COPY_ONCE   -e KOLLA_CONFIG='{"command": "/usr/sbin/httpd"}'   docker.io/tripleomaster/centos-binary-keystone:current-tripleo

But I still get an error of “no listening sockets available, shutting down” even if I try this as Root. Below is the whole thing I tried to run.

$ podman run   -v $PWD/fernet-keys:/var/lib/kolla/config_files/src/etc/keystone/fernet-keys   -e KOLLA_CONFIG_STRATEGY=COPY_ONCE   -e KOLLA_CONFIG='{ "command": "/usr/sbin/httpd", "config_files": [ { "source": "/var/lib/kolla/config_files/src/etc/keystone/fernet-keys", "dest": "/etc/keystone/fernet-keys", "owner":"keystone", "merge": false, "perm": "0600" } ], "permissions": [ { "path": "/var/log/kolla/keystone", "owner": "keystone:keystone", "recurse": true } ] }'  docker.io/tripleomaster/centos-binary-keystone:current-tripleo

Lets go back to simple things. What is inside the container? We can peek using:

$
podman run  docker.io/tripleomaster/centos-binary-keystone:current-tripleo ls

Basically, we can perform any command that will not last longer than the failed kolla initialization. No Bash prompts, but shorter single line bash commands work. We can see that mysql is uninitialized:

 podman run  docker.io/tripleomaster/centos-binary-keystone:current-tripleo cat /etc/keystone/keystone.conf | grep "connection ="
#connection = <None>

What about those config files that the initialization wants to copy:

podman run  docker.io/tripleomaster/centos-binary-keystone:current-tripleo ls /var/lib/kolla/config_files/src/etc/httpd/conf.d
ls: cannot access /var/lib/kolla/config_files/src/etc/httpd/conf.d: No such file or directory

So all that comes from external to the container, and is mounted at run time.

$ podman run  docker.io/tripleomaster/centos-binary-keystone:current-tripleo cat /etc/passwd  | grep keystone
keystone:x:42425:42425::/var/lib/keystone:/usr/sbin/nologin

Which owns the config and the log files.

$ podman run  docker.io/tripleomaster/centos-binary-keystone:current-tripleo ls -la /var/log/keystone
total 8
drwxr-x---. 2 keystone keystone 4096 Dec 17 08:28 .
drwxr-xr-x. 6 root     root     4096 Dec 17 08:28 ..
-rw-rw----. 1 root     keystone    0 Dec 17 08:28 keystone.log
$ podman run  docker.io/tripleomaster/centos-binary-keystone:current-tripleo ls -la /etc/keystone
total 128
drwxr-x---. 2 root     keystone   4096 Dec 17 08:28 .
drwxr-xr-x. 2 root     root       4096 Dec 19 16:30 ..
-rw-r-----. 1 root     keystone   2303 Nov 12 02:15 default_catalog.templates
-rw-r-----. 1 root     keystone 104220 Dec 14 01:09 keystone.conf
-rw-r-----. 1 root     keystone   1046 Nov 12 02:15 logging.conf
-rw-r-----. 1 root     keystone      3 Dec 14 01:09 policy.json
-rw-r-----. 1 keystone keystone    665 Nov 12 02:15 sso_callback_template.html
$ podman run  docker.io/tripleomaster/centos-binary-keystone:current-tripleo cat /etc/keystone/policy.json
{}

Yes, policy.json is empty.

Lets go back to the config file. I would rather not have to pass in all the config info as an environment variable each time. If I run as root, I can use the podman bind-mount option to relabel it:

 podman run -e KOLLA_CONFIG_FILE=/config.json  -e KOLLA_CONFIG_STRATEGY=COPY_ONCE   -v $PWD/config.json:/config.json:z   docker.io/tripleomaster/centos-binary-keystone:current-tripleo

This eventually fails with the error message “no listening sockets available, shutting down” Which seems to be due to the lack of the httpd.conf entries for keystone:

# podman run -e KOLLA_CONFIG_FILE=/config.json  -e KOLLA_CONFIG_STRATEGY=COPY_ONCE   -v $PWD/config.json:/config.json:z   docker.io/tripleomaster/centos-binary-keystone:current-tripleo  ls /etc/httpd/conf.d
auth_mellon.conf
auth_openidc.conf
autoindex.conf
README
ssl.conf
userdir.conf
welcome.conf

The clue seems to be in the Heat Templates. There are a bunch of files that are expected to be in /var/lib/kolla/config_files/src in side the container. Here’s my version of the WSGI config file:

Listen 5000
Listen 35357
 
ServerSignature Off
ServerTokens Prod
TraceEnable off
 
ErrorLog /var/log/kolla/keystone/apache-error.log"
<IfModule log_config_module>
    CustomLog /var/log/kolla/keystone/apache-access.log" common
</IfModule>
 
LogLevel info
 
<Directory "/usr/bin">
    <FilesMatch "^keystone-wsgi-(public|admin)$">
        AllowOverride None
        Options None
        Require all granted
    </FilesMatch>
</Directory>
 
 
<VirtualHost *:5000>
    WSGIDaemonProcess keystone-public processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP} python-path=/usr/lib/python2.7/site-packages
    WSGIProcessGroup keystone-public
    WSGIScriptAlias / /usr/bin/keystone-wsgi-public
    WSGIApplicationGroup %{GLOBAL}
    WSGIPassAuthorization On
    <IfVersion >= 2.4>
      ErrorLogFormat "%{cu}t %M"
    </IfVersion>
    ErrorLog "/var/log/kolla/keystone/keystone-apache-public-error.log"
    LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b %D \"%{Referer}i\" \"%{User-Agent}i\"" logformat
    CustomLog "/var/log/kolla/keystone/keystone-apache-public-access.log" logformat
</VirtualHost>
 
<VirtualHost *:35357>
    WSGIDaemonProcess keystone-admin processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP} python-path=/usr/lib/python2.7/site-packages
    WSGIProcessGroup keystone-admin
    WSGIScriptAlias / /usr/bin/keystone-wsgi-admin
    WSGIApplicationGroup %{GLOBAL}
    WSGIPassAuthorization On
    <IfVersion >= 2.4>
      ErrorLogFormat "%{cu}t %M"
    </IfVersion>
    ErrorLog "/var/log/kolla/keystone/keystone-apache-admin-error.log"
    LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b %D \"%{Referer}i\" \"%{User-Agent}i\"" logformat
    CustomLog "/var/log/kolla/keystone/keystone-apache-admin-access.log" logformat
</VirtualHost>

So with a directory structure like this:

C[root@ayoungP40 kolla]find src/ -print
src/
src/etc
src/etc/keystone
src/etc/keystone/fernet-keys
src/etc/keystone/fernet-keys/1
src/etc/keystone/fernet-keys/0
src/etc/httpd
src/etc/httpd/conf.d
src/etc/httpd/conf.d/wsgi-keystone.conf

And a Kolla config.json file like this:

{
   "command": "/usr/sbin/httpd",
   "config_files": [
        {
              "source": "/var/lib/kolla/config_files/src/etc/keystone/fernet-keys",
              "dest": "/etc/keystone/fernet-keys",
              "merge": false,
              "preserve_properties": true
        },{
              "source": "/var/lib/kolla/config_files/src/etc/httpd/conf.d",
              "dest": "/etc/httpd/conf.d",
              "merge": false,
              "preserve_properties": true
        },{  
              "source": "/var/lib/kolla/config_files/src/*",
              "dest": "/",
              "merge": true,
              "preserve_properties": true
        }
    ],
    "permissions": [
	    {
            "path": "/var/log/kolla/keystone",
            "owner": "keystone:keystone",
            "recurse": true
        }
    ]
}

I can run Keystone like this:

podman run -e KOLLA_CONFIG_FILE=/config.json  -e KOLLA_CONFIG_STRATEGY=COPY_ONCE   -v $PWD/config.json:/config.json:z   -v $PWD/src:/var/lib/kolla/config_files/src:z  docker.io/tripleomaster/centos-binary-keystone:current-tripleo

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.