I took the container registry I ran via podman and put it behind an Apache HTTPD instance secured with mod_ssl. Now when I try to log in to it, I get:
error authenticating creds for “nuzleaf.home.younglogic.net”: error pinging docker registry nuzleaf.home.younglogic.net: invalid status code from registry 403 (Forbidden)
Here’s my debugging notes.
To get more information, I rerun the loogin command with the debug flag:
podman login -u ayoung -p `cat password` nuzleaf.home.younglogic.net --log-level debug <pre> <!-- /wp:html --> <!-- wp:paragraph --> <p>And I see this in the output:</p> <!-- /wp:paragraph --> <!-- wp:html --> <pre lang="bash">DEBU[0000] Credentials not found DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/nuzleaf.home.younglogic.net |
If I try with the flag –tls-verify=false, things work:
$ podman login -u ayoung -p `cat password` nuzleaf.home.younglogic.net --tls-verify=false WARNING! Using --password via the cli is insecure. Please consider using --password-stdin Login Succeeded! |
Yes, that warning bothers me. One issue at a time.
And…there is this: https://github.com/golang/go/issues/40521Â Which shos that the TLS libraray in Go does not support the client authentication mechanism in TLS 1.3.
I’m going to leave the stuff below this in the page, as it was good debugging, but the short of it is that it won’t work until Go updates, and then we rebuild podman.
Looking at the Man page for containers-certs.d(5) I see the following:
A certs directory can contain one or more files with the following extensions:
· *.crt files with this extensions will be interpreted as CA certificates
Hmmm…my machines are both ipa clients, and there should be no further certificate non-sense. I am assuming here the podman code is following the pattern from Docker. Let me try putting the IPA server CA cert into this directory.
curl -O https://nuzleaf.home.younglogic.net/ipa/config/ca.crt sudo mkdir -p /etc/docker/certs.d/nuzleaf.home.younglogic.net sudo mv ca.crt /etc/docker/certs.d/nuzleaf.home.younglogic.net/ |
While this does not solve my problem, it does have some effect. Rerunning with tls checking on, and debugging, I now see:
DEBU[0000] Returning credentials from /run/user/1000/containers/auth.json DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/nuzleaf.home.younglogic.net DEBU[0000] crt: /etc/docker/certs.d/nuzleaf.home.younglogic.net/ca.crt |
Since I manage the server as well as the client, I can go over there to see what is happening. I use tail -f on /var/log/httpd/error_log to see the error specific to the login. It produces this output:
[Tue Sep 01 12:13:32.801607 2020] [ssl:error] [pid 131048:tid 140649565263616] [client 192.168.123.116:41734] AH: verify client post handshake [Tue Sep 01 12:13:32.801681 2020] [ssl:error] [pid 131048:tid 140649565263616] [client 192.168.123.116:41734] AH10158: cannot perform post-handshake authentication [Tue Sep 01 12:13:32.801710 2020] [ssl:error] [pid 131048:tid 140649565263616] SSL Library Error: error:14268117:SSL routines:SSL_verify_client_post_handshake:extension not received |
Looking at the proxy config file in the /etc/httpd/conf.d/registry-proxy.conf:
ProxyRequests Off <LocationMatch "^/v2"> SSLOptions +StdEnvVars +ExportCertData +StrictRequire +OptRenegotiate SSLVerifyClient optional ProxyPassMatch http://localhost:4000 ProxyPassReverse http://localhost:4000 </LocationMatch> |
So it looks like it is trying to validate the client certificate. Let’s issue one and see what happens. I’m going to create a service principal for the client machine, request a cert for it, and use that cert to authenticate when I call to the registry.
The man page for containers-certs.d has the following layout specified:
/etc/containers/certs.d/ <- Certificate directory └── my-registry.com:5000 <- Hostname:port ├── client.cert <- Client certificate ├── client.key <- Client key └── ca.crt <- Certificate authority that signed the registry certificate |
Here is how I requested the certificate. Note the my user is an admin on the IPA server. I also need to set the SE Linux context on the target directory so certmonger can write the key and certificate files.
$ kinit Password for ayoung@HOME.YOUNGLOGIC.NET: $ ipa service-add containerregistry/yamask.home.younglogic.net@HOME.YOUNGLOGIC.NET -------------------------------------------------------------------------------- Added service "containerregistry/yamask.home.younglogic.net@HOME.YOUNGLOGIC.NET" -------------------------------------------------------------------------------- Principal name: containerregistry/yamask.home.younglogic.net@HOME.YOUNGLOGIC.NET Principal alias: containerregistry/yamask.home.younglogic.net@HOME.YOUNGLOGIC.NET Managed by: yamask.home.younglogic.net $ sudo semanage fcontext -a -t cert_t /etc/docker/certs.d/nuzleaf.home.younglogic.net/ $ sudo restorecon /etc/docker/certs.d/nuzleaf.home.younglogic.net/ $ sudo ipa-getcert request -f /etc/docker/certs.d/nuzleaf.home.younglogic.net/client.cert -k /etc/docker/certs.d/nuzleaf.home.younglogic.net/client.key --principal=containerregistry/yamask.home.younglogic.net@HOME.YOUNGLOGIC.NET |
Lets take a look at the files to see if there is anytthing there
$ ls -lZ /etc/docker/certs.d/nuzleaf.home.younglogic.net/ total 12 -rw-rw-r--. 1 ayoung ayoung unconfined_u:object_r:user_home_t:s0 1667 Sep 1 12:05 ca.crt -rw-------. 1 root root system_u:object_r:cert_t:s0 1996 Sep 1 12:40 client.cert -rw-------. 1 root root system_u:object_r:cert_t:s0 1704 Sep 1 12:40 client.key |
And lets try again:
$ podman login -u ayoung -p `cat password` nuzleaf.home.younglogic.net --log-level debug ... DEBU[0000] cert: /etc/docker/certs.d/nuzleaf.home.younglogic.net/client.cert ERRO[0000] error authenticating creds for "nuzleaf.home.younglogic.net": error creating new docker client: open /etc/docker/certs.d/nuzleaf.home.younglogic.net/client.cert: permission denied |
Closer. So it is trying to read something….these next two steps are not secure, but included for debugging. Specifically, the key file should NEVER be made world writable.
$ sudo chmod a+r /etc/docker/certs.d/nuzleaf.home.younglogic.net/client.key $ sudo chmod a+r /etc/docker/certs.d/nuzleaf.home.younglogic.net/client.cert |
No love: still get a 403 from the server. However, I do notice this:
DEBU[0000] Returning credentials from /run/user/1000/containers/auth.json |
And in there I do, infact, see an entry for my server
$ cat /run/user/1000/containers/auth.json { "auths": { "nuzleaf.home.younglogic.net": { "auth": "YXlvdW5nOldvbGZob3VuZHM0LTI3" }, "nuzleaf.home.younglogic.net:4000": { "auth": "YXlvdW5nOldvbGZob3VuZHM0LTI3" } } } |
`podman` looks for certificates under `/etc/containers/certs.d` by default. More here:
https://docs.podman.io/en/latest/markdown/podman-login.1.html
Most of the time you only require the Certificate Authority (CA) certificate of the _registry_ in that folder.
I found that using `docker`’s default location troublesome. On more recent versions it the podman location takes precedent.