Podman login to a secured registry

I took the container registry I ran via podman and put it behind an Apache HTTPD instance secured with mod_ssl. Now when I try to log in to it, I get:

error authenticating creds for “nuzleaf.home.younglogic.net”: error pinging docker registry nuzleaf.home.younglogic.net: invalid status code from registry 403 (Forbidden)

Here’s my debugging notes.

To get more information, I rerun the loogin command with the debug flag:

podman login  -u ayoung  -p `cat password`   nuzleaf.home.younglogic.net   --log-level debug
<!-- /wp:html -->
<!-- wp:paragraph -->
<p>And I see this in the output:</p>
<!-- /wp:paragraph -->
<!-- wp:html -->
<pre lang="bash">DEBU[0000] Credentials not found 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/nuzleaf.home.younglogic.net

If I try with the flag –tls-verify=false, things work:

$ podman login  -u ayoung  -p `cat password`   nuzleaf.home.younglogic.net      --tls-verify=false 
WARNING! Using --password via the cli is insecure. Please consider using --password-stdin
Login Succeeded!

Yes, that warning bothers me. One issue at a time.

And…there is this: https://github.com/golang/go/issues/40521  Which shos that the TLS libraray in Go does not support the client authentication mechanism in TLS 1.3.

I’m going to leave the stuff below this in the page, as it was good debugging, but the short of it is that it won’t work until Go updates, and then we rebuild podman.

Looking at the Man page for containers-certs.d(5) I see the following:

A certs directory can contain one or more files with the following extensions:
· *.crt files with this extensions will be interpreted as CA certificates

Hmmm…my machines are both ipa clients, and there should be no further certificate non-sense. I am assuming here the podman code is following the pattern from Docker. Let me try putting the IPA server CA cert into this directory.

curl -O https://nuzleaf.home.younglogic.net/ipa/config/ca.crt
sudo mkdir -p /etc/docker/certs.d/nuzleaf.home.younglogic.net
sudo mv ca.crt /etc/docker/certs.d/nuzleaf.home.younglogic.net/

While this does not solve my problem, it does have some effect. Rerunning with tls checking on, and debugging, I now see:

DEBU[0000] Returning credentials from /run/user/1000/containers/auth.json 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/nuzleaf.home.younglogic.net 
DEBU[0000]  crt: /etc/docker/certs.d/nuzleaf.home.younglogic.net/ca.crt

Since I manage the server as well as the client, I can go over there to see what is happening. I use tail -f on /var/log/httpd/error_log to see the error specific to the login. It produces this output:

[Tue Sep 01 12:13:32.801607 2020] [ssl:error] [pid 131048:tid 140649565263616] [client] AH: verify client post handshake
[Tue Sep 01 12:13:32.801681 2020] [ssl:error] [pid 131048:tid 140649565263616] [client] AH10158: cannot perform post-handshake authentication
[Tue Sep 01 12:13:32.801710 2020] [ssl:error] [pid 131048:tid 140649565263616] SSL Library Error: error:14268117:SSL routines:SSL_verify_client_post_handshake:extension not received

Looking at the proxy config file in the /etc/httpd/conf.d/registry-proxy.conf:

ProxyRequests Off
<LocationMatch "^/v2">
    SSLOptions +StdEnvVars +ExportCertData +StrictRequire +OptRenegotiate
    SSLVerifyClient optional
    ProxyPassMatch http://localhost:4000
    ProxyPassReverse http://localhost:4000

So it looks like it is trying to validate the client certificate. Let’s issue one and see what happens. I’m going to create a service principal for the client machine, request a cert for it, and use that cert to authenticate when I call to the registry.

The man page for containers-certs.d has the following layout specified:

             /etc/containers/certs.d/    <- Certificate directory
              └── my-registry.com:5000    <- Hostname:port
                 ├── client.cert          <- Client certificate
                 ├── client.key           <- Client key
                 └── ca.crt               <- Certificate authority that signed the registry certificate

Here is how I requested the certificate. Note the my user is an admin on the IPA server. I also need to set the SE Linux context on the target directory so certmonger can write the key and certificate files.

$ kinit 
Password for ayoung@HOME.YOUNGLOGIC.NET: 
$ ipa service-add containerregistry/yamask.home.younglogic.net@HOME.YOUNGLOGIC.NET
Added service "containerregistry/yamask.home.younglogic.net@HOME.YOUNGLOGIC.NET"
  Principal name: containerregistry/yamask.home.younglogic.net@HOME.YOUNGLOGIC.NET
  Principal alias: containerregistry/yamask.home.younglogic.net@HOME.YOUNGLOGIC.NET
  Managed by: yamask.home.younglogic.net
$ sudo semanage fcontext -a   -t cert_t /etc/docker/certs.d/nuzleaf.home.younglogic.net/
$ sudo restorecon /etc/docker/certs.d/nuzleaf.home.younglogic.net/
$ sudo ipa-getcert request -f /etc/docker/certs.d/nuzleaf.home.younglogic.net/client.cert -k /etc/docker/certs.d/nuzleaf.home.younglogic.net/client.key --principal=containerregistry/yamask.home.younglogic.net@HOME.YOUNGLOGIC.NET

Lets take a look at the files to see if there is anytthing there

$ ls -lZ /etc/docker/certs.d/nuzleaf.home.younglogic.net/
total 12
-rw-rw-r--. 1 ayoung ayoung unconfined_u:object_r:user_home_t:s0 1667 Sep  1 12:05 ca.crt
-rw-------. 1 root   root   system_u:object_r:cert_t:s0          1996 Sep  1 12:40 client.cert
-rw-------. 1 root   root   system_u:object_r:cert_t:s0          1704 Sep  1 12:40 client.key

And lets try again:

$ podman login  -u ayoung  -p `cat password`   nuzleaf.home.younglogic.net   --log-level debug
DEBU[0000]  cert: /etc/docker/certs.d/nuzleaf.home.younglogic.net/client.cert 
ERRO[0000] error authenticating creds for "nuzleaf.home.younglogic.net": error creating new docker client: open /etc/docker/certs.d/nuzleaf.home.younglogic.net/client.cert: permission denied

Closer. So it is trying to read something….these next two steps are not secure, but included for debugging. Specifically, the key file should NEVER be made world writable.

$ sudo chmod a+r /etc/docker/certs.d/nuzleaf.home.younglogic.net/client.key
$ sudo chmod a+r /etc/docker/certs.d/nuzleaf.home.younglogic.net/client.cert

No love: still get a 403 from the server. However, I do notice this:

DEBU[0000] Returning credentials from /run/user/1000/containers/auth.json

And in there I do, infact, see an entry for my server

$ cat /run/user/1000/containers/auth.json
	"auths": {
		"nuzleaf.home.younglogic.net": {
			"auth": "YXlvdW5nOldvbGZob3VuZHM0LTI3"
		"nuzleaf.home.younglogic.net:4000": {
			"auth": "YXlvdW5nOldvbGZob3VuZHM0LTI3"

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.