My recent forays into running containerized Keystone images have been based on a Centos base image with RPMs installed on top of it. But TripleO does not run this way; it runs via containers. Some notes as I look into them.
The official containers for TripleO are currently hosted on docker.com. The Keystone page is here:
Don’t expect the docker pull command posted on that page to work. I tried a comparable one with podman and got:
$ podman pull tripleomaster/centos-binary-keystone Trying to pull docker.io/tripleomaster/centos-binary-keystone... manifest unknown: manifest unknown Trying to pull registry.fedoraproject.org/tripleomaster/centos-binary-keystone... |
And a few more lines of error output. Thanks to Emilien M, I was able to get the right command:
$ podman pull tripleomaster/centos-binary-keystone:current-tripleo Trying to pull docker.io/tripleomaster/centos-binary-keystone:current-tripleo... Getting image source signatures ... Copying config 9e85172eba done Writing manifest to image destination Storing signatures 9e85172eba10a2648ae7235076ada77b095ed3da05484916381410135cc8884c |
Since I did this as a normal account, and not as root, the image does not get stored under /var, but instead goes somewhere under $HOME/.local. If I type
$ podman images REPOSITORY TAG IMAGE ID CREATED SIZE docker.io/tripleomaster/centos-binary-keystone current-tripleo 9e85172eba10 2 days ago 904 MB |
I can see the short form of the hash starting with 9e85. I copy that to use to match the subdir under ls /home/ayoung/.local/share/containers/storage/overlay-image
ls /home/ayoung/.local/share/containers/storage/overlay-images/9e85172eba10a2648ae7235076ada77b095ed3da05484916381410135cc8884c/ |
If I cat that file, I can see all of the layers that make up the image itself.
Trying a naive: podman run docker.io/tripleomaster/centos-binary-keystone:current-tripleo I get an error that shows just how kolla-centric this image is:
$ podman run docker.io/tripleomaster/centos-binary-keystone:current-tripleo + sudo -E kolla_set_configs INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json ERROR:__main__:Unexpected error: Traceback (most recent call last): File "/usr/local/bin/kolla_set_configs", line 412, in main config = load_config() File "/usr/local/bin/kolla_set_configs", line 294, in load_config config = load_from_file() File "/usr/local/bin/kolla_set_configs", line 282, in load_from_file with open(config_file) as f: IOError: [Errno 2] No such file or directory: '/var/lib/kolla/config_files/config.json' |
So I read the docs. Trying to fake it with:
$ podman run -e KOLLA_CONFIG='{}' docker.io/tripleomaster/centos-binary-keystone:current-tripleo + sudo -E kolla_set_configs INFO:__main__:Validating config file ERROR:__main__:InvalidConfig: Config is missing required "command" key |
When running with TripleO, The config files are generated from Heat Templates. The values for the config.json come from here.
This gets me slightly closer:
podman run -e KOLLA_CONFIG_STRATEGY=COPY_ONCE -e KOLLA_CONFIG='{"command": "/usr/sbin/httpd"}' docker.io/tripleomaster/centos-binary-keystone:current-tripleo |
But I still get an error of “no listening sockets available, shutting down” even if I try this as Root. Below is the whole thing I tried to run.
$ podman run -v $PWD/fernet-keys:/var/lib/kolla/config_files/src/etc/keystone/fernet-keys -e KOLLA_CONFIG_STRATEGY=COPY_ONCE -e KOLLA_CONFIG='{ "command": "/usr/sbin/httpd", "config_files": [ { "source": "/var/lib/kolla/config_files/src/etc/keystone/fernet-keys", "dest": "/etc/keystone/fernet-keys", "owner":"keystone", "merge": false, "perm": "0600" } ], "permissions": [ { "path": "/var/log/kolla/keystone", "owner": "keystone:keystone", "recurse": true } ] }' docker.io/tripleomaster/centos-binary-keystone:current-tripleo |
Lets go back to simple things. What is inside the container? We can peek using:
$ podman run docker.io/tripleomaster/centos-binary-keystone:current-tripleo ls |
Basically, we can perform any command that will not last longer than the failed kolla initialization. No Bash prompts, but shorter single line bash commands work. We can see that mysql is uninitialized:
podman run docker.io/tripleomaster/centos-binary-keystone:current-tripleo cat /etc/keystone/keystone.conf | grep "connection =" #connection = <None> |
What about those config files that the initialization wants to copy:
podman run docker.io/tripleomaster/centos-binary-keystone:current-tripleo ls /var/lib/kolla/config_files/src/etc/httpd/conf.d ls: cannot access /var/lib/kolla/config_files/src/etc/httpd/conf.d: No such file or directory |
So all that comes from external to the container, and is mounted at run time.
$ podman run docker.io/tripleomaster/centos-binary-keystone:current-tripleo cat /etc/passwd | grep keystone keystone:x:42425:42425::/var/lib/keystone:/usr/sbin/nologin |
Which owns the config and the log files.
$ podman run docker.io/tripleomaster/centos-binary-keystone:current-tripleo ls -la /var/log/keystone total 8 drwxr-x---. 2 keystone keystone 4096 Dec 17 08:28 . drwxr-xr-x. 6 root root 4096 Dec 17 08:28 .. -rw-rw----. 1 root keystone 0 Dec 17 08:28 keystone.log $ podman run docker.io/tripleomaster/centos-binary-keystone:current-tripleo ls -la /etc/keystone total 128 drwxr-x---. 2 root keystone 4096 Dec 17 08:28 . drwxr-xr-x. 2 root root 4096 Dec 19 16:30 .. -rw-r-----. 1 root keystone 2303 Nov 12 02:15 default_catalog.templates -rw-r-----. 1 root keystone 104220 Dec 14 01:09 keystone.conf -rw-r-----. 1 root keystone 1046 Nov 12 02:15 logging.conf -rw-r-----. 1 root keystone 3 Dec 14 01:09 policy.json -rw-r-----. 1 keystone keystone 665 Nov 12 02:15 sso_callback_template.html $ podman run docker.io/tripleomaster/centos-binary-keystone:current-tripleo cat /etc/keystone/policy.json {} |
Yes, policy.json is empty.
Lets go back to the config file. I would rather not have to pass in all the config info as an environment variable each time. If I run as root, I can use the podman bind-mount option to relabel it:
podman run -e KOLLA_CONFIG_FILE=/config.json -e KOLLA_CONFIG_STRATEGY=COPY_ONCE -v $PWD/config.json:/config.json:z docker.io/tripleomaster/centos-binary-keystone:current-tripleo |
This eventually fails with the error message “no listening sockets available, shutting down” Which seems to be due to the lack of the httpd.conf entries for keystone:
# podman run -e KOLLA_CONFIG_FILE=/config.json -e KOLLA_CONFIG_STRATEGY=COPY_ONCE -v $PWD/config.json:/config.json:z docker.io/tripleomaster/centos-binary-keystone:current-tripleo ls /etc/httpd/conf.d
auth_mellon.conf
auth_openidc.conf
autoindex.conf
README
ssl.conf
userdir.conf
welcome.conf |
The clue seems to be in the Heat Templates. There are a bunch of files that are expected to be in /var/lib/kolla/config_files/src in side the container. Here’s my version of the WSGI config file:
Listen 5000
Listen 35357
ServerSignature Off
ServerTokens Prod
TraceEnable off
ErrorLog /var/log/kolla/keystone/apache-error.log"
<IfModule log_config_module>
CustomLog /var/log/kolla/keystone/apache-access.log" common
</IfModule>
LogLevel info
<Directory "/usr/bin">
<FilesMatch "^keystone-wsgi-(public|admin)$">
AllowOverride None
Options None
Require all granted
</FilesMatch>
</Directory>
<VirtualHost *:5000>
WSGIDaemonProcess keystone-public processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP} python-path=/usr/lib/python2.7/site-packages
WSGIProcessGroup keystone-public
WSGIScriptAlias / /usr/bin/keystone-wsgi-public
WSGIApplicationGroup %{GLOBAL}
WSGIPassAuthorization On
<IfVersion >= 2.4>
ErrorLogFormat "%{cu}t %M"
</IfVersion>
ErrorLog "/var/log/kolla/keystone/keystone-apache-public-error.log"
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b %D \"%{Referer}i\" \"%{User-Agent}i\"" logformat
CustomLog "/var/log/kolla/keystone/keystone-apache-public-access.log" logformat
</VirtualHost>
<VirtualHost *:35357>
WSGIDaemonProcess keystone-admin processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP} python-path=/usr/lib/python2.7/site-packages
WSGIProcessGroup keystone-admin
WSGIScriptAlias / /usr/bin/keystone-wsgi-admin
WSGIApplicationGroup %{GLOBAL}
WSGIPassAuthorization On
<IfVersion >= 2.4>
ErrorLogFormat "%{cu}t %M"
</IfVersion>
ErrorLog "/var/log/kolla/keystone/keystone-apache-admin-error.log"
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b %D \"%{Referer}i\" \"%{User-Agent}i\"" logformat
CustomLog "/var/log/kolla/keystone/keystone-apache-admin-access.log" logformat
</VirtualHost> |
So with a directory structure like this:
C[root@ayoungP40 kolla]find src/ -print src/ src/etc src/etc/keystone src/etc/keystone/fernet-keys src/etc/keystone/fernet-keys/1 src/etc/keystone/fernet-keys/0 src/etc/httpd src/etc/httpd/conf.d src/etc/httpd/conf.d/wsgi-keystone.conf |
And a Kolla config.json file like this:
{ "command": "/usr/sbin/httpd", "config_files": [ { "source": "/var/lib/kolla/config_files/src/etc/keystone/fernet-keys", "dest": "/etc/keystone/fernet-keys", "merge": false, "preserve_properties": true },{ "source": "/var/lib/kolla/config_files/src/etc/httpd/conf.d", "dest": "/etc/httpd/conf.d", "merge": false, "preserve_properties": true },{ "source": "/var/lib/kolla/config_files/src/*", "dest": "/", "merge": true, "preserve_properties": true } ], "permissions": [ { "path": "/var/log/kolla/keystone", "owner": "keystone:keystone", "recurse": true } ] } |
I can run Keystone like this:
podman run -e KOLLA_CONFIG_FILE=/config.json -e KOLLA_CONFIG_STRATEGY=COPY_ONCE -v $PWD/config.json:/config.json:z -v $PWD/src:/var/lib/kolla/config_files/src:z docker.io/tripleomaster/centos-binary-keystone:current-tripleo |