I had a handful of machines enrolled in a demo cluster. About half of them got shut down, and now I can’t SSH into them via Kerberos tickets. Here is my debugging notebook.
Tail -f the file tail -f /var/log/krb5kdc.log on the ipa server.
Start by doing a kdestroy on my home machine, and then
kinit ayoung@OPENSTACK.FREEIPA.ORG
I see this in the IPA server.
Apr 25 22:38:56 ipa.openstack.freeipa.org krb5kdc[5728](info): AS_REQ (7 etypes {18 17 16 23 1 3 2}) 10.10.59.141: NEEDED_PREAUTH: ayoung@OPENSTACK.FREEIPA.ORG for krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG, Additional pre-authentication required
Apr 25 22:39:00 ipa.openstack.freeipa.org krb5kdc[5728](info): AS_REQ (7 etypes {18 17 16 23 1 3 2}) 10.10.59.141: ISSUE: authtime 1366929540, etypes {rep=18 tkt=18 ses=18}, ayoung@OPENSTACK.FREEIPA.ORG for krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG
Apr 25 22:39:01 ipa.openstack.freeipa.org krb5kdc[5729](info): AS_REQ (6 etypes {18 17 16 23 25 26}) 192.168.0.61: NEEDED_PREAUTH: keystone@OPENSTACK.FREEIPA.ORG for krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG, Additional pre-authentication required
Apr 25 22:39:01 ipa.openstack.freeipa.org krb5kdc[5729](info): AS_REQ (6 etypes {18 17 16 23 25 26}) 192.168.0.61: ISSUE: authtime 1366929541, etypes {rep=18 tkt=18 ses=18}, keystone@OPENSTACK.FREEIPA.ORG for krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG
Now try to hit the web UI with my browser by pointing it at:
https://ipa.openstack.freeipa.org/ipa/ui/
Klist shows no ticket…I probably need to log out first to forget the form based auth. Click log out and see a page that says:
You have been logged out
Return to main page.
Returning to the main page should do a negotiate. Lets see… nope
OK, just to be sure, I go through the browser config steps again. Then head back to the main page: and it works. Looking in the log, the interesting entries are:
Apr 25 22:44:44 ipa.openstack.freeipa.org krb5kdc[5728](info): TGS_REQ (7 etypes {18 17 16 23 1 3 2}) 10.10.59.141: ISSUE: authtime 1366929540, etypes {rep=18 tkt=18 ses=18}, ayoung@OPENSTACK.FREEIPA.ORG for HTTP/ipa.openstack.freeipa.org@OPENSTACK.FREEIPA.ORG
This shows it getting a ticket for the web UI and then klist shows
Valid starting Expires Service principal 04/25/13 18:39:00 04/26/13 18:39:00 krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG renew until 04/26/13 18:42:53 04/25/13 18:44:44 04/26/13 18:39:00 HTTP/ipa.openstack.freeipa.org@ renew until 04/26/13 18:42:53 04/25/13 18:44:44 04/26/13 18:39:00 HTTP/ipa.openstack.freeipa.org@OPENSTACK.FREEIPA.ORG renew until 04/26/13 18:42:53
OK, on to those failing machines.
ssh -vv pg.openstack.freeipa.org debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: Next authentication method: gssapi-keyex debug1: No valid Key exchange context debug2: we did not send a packet, disable method debug1: Next authentication method: gssapi-with-mic debug2: we sent a gssapi-with-mic packet, wait for reply debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug2: we sent a gssapi-with-mic packet, wait for reply debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug2: we sent a gssapi-with-mic packet, wait for reply debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug2: we sent a gssapi-with-mic packet, wait for reply debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic ... Received disconnect from 10.16.16.125: 2: Too many authentication failures for ayoung
Nothing in the krb5kdc.log for that transaction, but I see that I now have a ticket for pg…must have gotten it from a prior attempt. Kdestroy, kinit and try again.
See this
Apr 25 22:50:43 ipa.openstack.freeipa.org krb5kdc[5728](info): TGS_REQ (7 etypes {18 17 16 23 1 3 2}) 10.10.59.141: ISSUE: authtime 1366930206, etypes {rep=18 tkt=18 ses=18}, ayoung@OPENSTACK.FREEIPA.ORG for host/pg.openstack.freeipa.org@OPENSTACK.FREEIPA.ORG
OK, I have a Host ticket. Same response from the server. I can connect to the pg server via an ssh keypair, so I have a backdoor to debug. ssh in as root and: let me see if there is an sshd log.
Apr 25 22:58:30 pg sshd[6115]: Invalid user ayoung from 10.10.59.141 Apr 25 22:58:30 pg sshd[6115]: input_userauth_request: invalid user ayoung [preauth] Apr 25 22:58:30 pg sshd[6115]: Disconnecting: Too many authentication failures for ayoung [preauth]
Hmm. Invalid user. Sounds like a getent failure of some sort.
Is sssd running?
systemctl status sssd.service ... Active: active (running) since Mon 2013-04-22 14:25:39 UTC; 3 days ago
Yep. OK, what about nsswitch setup?
passwd: files sss That looks right. Should check in /etc/passwrd and then talk to sss, which should talk to IPA. Lets see if that is the case....nothing in /var/log/sssd/sssd_ssh.log /var/log/sssd/sssd.log /var/log/sssd/sssd_nss.log
How about /var/log/secure? Same as the sshd log.
ping ipa ping: unknown host ipa\
AHA! Rebooting did a new dhcp request and probably overwrote my /etc/resolve.conf file….lets look:
[root@pg ~]# cat /etc/resolv.conf # Generated by NetworkManager domain novalocal search novalocal nameserver 192.168.0.3
My internal was 192.168.0.45 for IPA….OK, we have at least one culprit. Change it to:
[root@pg ~]# cat /etc/resolv.conf # Generated by NetworkManager domain openstack.freeipa.org search openstack.freeipa.org domain novalocal nameserver 192.168.0.45
And now
[root@pg ~]# getent passwd ayoung ayoung:*:1615800005:1615800005:Adam Young:/home/ayoung:/bin/sh
So…here is the fix: add the following to /etc/dhcp/dhclient.conf
interface "eth0" {
supersede domain-name "openstack.freeipa.org";
supersede domain-search "openstack.freeipa.org";
supersede domain-name-servers 192.168.0.45;
}
And the resolv.conf data survives a reboot.