I had a handful of machines enrolled in a demo cluster. About half of them got shut down, and now I can’t SSH into them via Kerberos tickets. Here is my debugging notebook.

Tail -f the file tail -f /var/log/krb5kdc.log on the ipa server.

Start by doing a kdestroy on my home machine, and then

kinit ayoung@OPENSTACK.FREEIPA.ORG

I see this in the IPA server.

Apr 25 22:38:56 ipa.openstack.freeipa.org krb5kdc[5728](info): AS_REQ (7 etypes {18 17 16 23 1 3 2}) 10.10.59.141: NEEDED_PREAUTH: ayoung@OPENSTACK.FREEIPA.ORG for krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG, Additional pre-authentication required
Apr 25 22:39:00 ipa.openstack.freeipa.org krb5kdc[5728](info): AS_REQ (7 etypes {18 17 16 23 1 3 2}) 10.10.59.141: ISSUE: authtime 1366929540, etypes {rep=18 tkt=18 ses=18}, ayoung@OPENSTACK.FREEIPA.ORG for krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG
Apr 25 22:39:01 ipa.openstack.freeipa.org krb5kdc[5729](info): AS_REQ (6 etypes {18 17 16 23 25 26}) 192.168.0.61: NEEDED_PREAUTH: keystone@OPENSTACK.FREEIPA.ORG for krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG, Additional pre-authentication required
Apr 25 22:39:01 ipa.openstack.freeipa.org krb5kdc[5729](info): AS_REQ (6 etypes {18 17 16 23 25 26}) 192.168.0.61: ISSUE: authtime 1366929541, etypes {rep=18 tkt=18 ses=18}, keystone@OPENSTACK.FREEIPA.ORG for krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG

Now try to hit the web UI with my browser by pointing it at:

https://ipa.openstack.freeipa.org/ipa/ui/

Klist shows no ticket…I probably need to log out first to forget the form based auth. Click log out and see a page that says:

You have been logged out
Return to main page.

Returning to the main page should do a negotiate. Lets see… nope

OK, just to be sure, I go through the browser config steps again. Then head back to the main page: and it works. Looking in the log, the interesting entries are:

Apr 25 22:44:44 ipa.openstack.freeipa.org krb5kdc[5728](info): TGS_REQ (7 etypes {18 17 16 23 1 3 2}) 10.10.59.141: ISSUE: authtime 1366929540, etypes {rep=18 tkt=18 ses=18}, ayoung@OPENSTACK.FREEIPA.ORG for HTTP/ipa.openstack.freeipa.org@OPENSTACK.FREEIPA.ORG

This shows it getting a ticket for the web UI and then klist shows

Valid starting     Expires            Service principal
04/25/13 18:39:00  04/26/13 18:39:00  krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG
	renew until 04/26/13 18:42:53
04/25/13 18:44:44  04/26/13 18:39:00  HTTP/ipa.openstack.freeipa.org@
	renew until 04/26/13 18:42:53
04/25/13 18:44:44  04/26/13 18:39:00  HTTP/ipa.openstack.freeipa.org@OPENSTACK.FREEIPA.ORG
	renew until 04/26/13 18:42:53

OK, on to those failing machines.

ssh -vv pg.openstack.freeipa.org

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug1: Next authentication method: gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic

...

Received disconnect from 10.16.16.125: 2: Too many authentication failures for ayoung

Nothing in the krb5kdc.log for that transaction, but I see that I now have a ticket for pg…must have gotten it from a prior attempt. Kdestroy, kinit and try again.

See this

Apr 25 22:50:43 ipa.openstack.freeipa.org krb5kdc[5728](info): TGS_REQ (7 etypes {18 17 16 23 1 3 2}) 10.10.59.141: ISSUE: authtime 1366930206, etypes {rep=18 tkt=18 ses=18}, ayoung@OPENSTACK.FREEIPA.ORG for host/pg.openstack.freeipa.org@OPENSTACK.FREEIPA.ORG

OK, I have a Host ticket. Same response from the server. I can connect to the pg server via an ssh keypair, so I have a backdoor to debug. ssh in as root and: let me see if there is an sshd log.

Apr 25 22:58:30 pg sshd[6115]: Invalid user ayoung from 10.10.59.141
Apr 25 22:58:30 pg sshd[6115]: input_userauth_request: invalid user ayoung [preauth]
Apr 25 22:58:30 pg sshd[6115]: Disconnecting: Too many authentication failures for ayoung [preauth]

Hmm. Invalid user. Sounds like a getent failure of some sort.

Is sssd running?

systemctl status sssd.service
...
 Active: active (running) since Mon 2013-04-22 14:25:39 UTC; 3 days ago

Yep. OK, what about nsswitch setup?

passwd:     files sss

That looks right.  Should check in /etc/passwrd and then talk to sss, which should talk to IPA.  Lets see if that is the case....nothing in 
/var/log/sssd/sssd_ssh.log
/var/log/sssd/sssd.log
/var/log/sssd/sssd_nss.log

How about /var/log/secure? Same as the sshd log.

 ping ipa
ping: unknown host ipa\

AHA! Rebooting did a new dhcp request and probably overwrote my /etc/resolve.conf file….lets look:

[root@pg ~]# cat /etc/resolv.conf 
# Generated by NetworkManager
domain novalocal
search novalocal
nameserver 192.168.0.3

My internal was 192.168.0.45 for IPA….OK, we have at least one culprit. Change it to:

[root@pg ~]# cat /etc/resolv.conf 
# Generated by NetworkManager
domain openstack.freeipa.org
search openstack.freeipa.org
domain novalocal
nameserver 192.168.0.45

And now

[root@pg ~]# getent passwd ayoung
ayoung:*:1615800005:1615800005:Adam Young:/home/ayoung:/bin/sh

So…here is the fix: add the following to /etc/dhcp/dhclient.conf

interface "eth0" {
           supersede domain-name  "openstack.freeipa.org";
           supersede domain-search  "openstack.freeipa.org";
           supersede domain-name-servers 192.168.0.45;
       }

And the resolv.conf data survives a reboot.

First hack at a script to open the ports needed by FreeIPA. On Fedora 18, running Firewall D, I ran the below script. Comments and corrections welcome.

Continue reading →