Troubleshooting a FreeIPA install:

I had a handful of machines enrolled in a demo cluster. About half of them got shut down, and now I can’t SSH into them via Kerberos tickets. Here is my debugging notebook.

Tail -f the file tail -f /var/log/krb5kdc.log on the ipa server.

Start by doing a kdestroy on my home machine, and then

kinit ayoung@OPENSTACK.FREEIPA.ORG

I see this in the IPA server.

Apr 25 22:38:56 ipa.openstack.freeipa.org krb5kdc[5728](info): AS_REQ (7 etypes {18 17 16 23 1 3 2}) 10.10.59.141: NEEDED_PREAUTH: ayoung@OPENSTACK.FREEIPA.ORG for krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG, Additional pre-authentication required
Apr 25 22:39:00 ipa.openstack.freeipa.org krb5kdc[5728](info): AS_REQ (7 etypes {18 17 16 23 1 3 2}) 10.10.59.141: ISSUE: authtime 1366929540, etypes {rep=18 tkt=18 ses=18}, ayoung@OPENSTACK.FREEIPA.ORG for krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG
Apr 25 22:39:01 ipa.openstack.freeipa.org krb5kdc[5729](info): AS_REQ (6 etypes {18 17 16 23 25 26}) 192.168.0.61: NEEDED_PREAUTH: keystone@OPENSTACK.FREEIPA.ORG for krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG, Additional pre-authentication required
Apr 25 22:39:01 ipa.openstack.freeipa.org krb5kdc[5729](info): AS_REQ (6 etypes {18 17 16 23 25 26}) 192.168.0.61: ISSUE: authtime 1366929541, etypes {rep=18 tkt=18 ses=18}, keystone@OPENSTACK.FREEIPA.ORG for krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG

Now try to hit the web UI with my browser by pointing it at:


https://ipa.openstack.freeipa.org/ipa/ui/

Klist shows no ticket…I probably need to log out first to forget the form based auth. Click log out and see a page that says:

You have been logged out
Return to main page.

Returning to the main page should do a negotiate. Lets see… nope

OK, just to be sure, I go through the browser config steps again. Then head back to the main page: and it works. Looking in the log, the interesting entries are:

Apr 25 22:44:44 ipa.openstack.freeipa.org krb5kdc[5728](info): TGS_REQ (7 etypes {18 17 16 23 1 3 2}) 10.10.59.141: ISSUE: authtime 1366929540, etypes {rep=18 tkt=18 ses=18}, ayoung@OPENSTACK.FREEIPA.ORG for HTTP/ipa.openstack.freeipa.org@OPENSTACK.FREEIPA.ORG

This shows it getting a ticket for the web UI and then klist shows

Valid starting     Expires            Service principal
04/25/13 18:39:00  04/26/13 18:39:00  krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG
	renew until 04/26/13 18:42:53
04/25/13 18:44:44  04/26/13 18:39:00  HTTP/ipa.openstack.freeipa.org@
	renew until 04/26/13 18:42:53
04/25/13 18:44:44  04/26/13 18:39:00  HTTP/ipa.openstack.freeipa.org@OPENSTACK.FREEIPA.ORG
	renew until 04/26/13 18:42:53

OK, on to those failing machines.

ssh -vv pg.openstack.freeipa.org

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug1: Next authentication method: gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic

...

Received disconnect from 10.16.16.125: 2: Too many authentication failures for ayoung

Nothing in the krb5kdc.log for that transaction, but I see that I now have a ticket for pg…must have gotten it from a prior attempt. Kdestroy, kinit and try again.

See this

Apr 25 22:50:43 ipa.openstack.freeipa.org krb5kdc[5728](info): TGS_REQ (7 etypes {18 17 16 23 1 3 2}) 10.10.59.141: ISSUE: authtime 1366930206, etypes {rep=18 tkt=18 ses=18}, ayoung@OPENSTACK.FREEIPA.ORG for host/pg.openstack.freeipa.org@OPENSTACK.FREEIPA.ORG

OK, I have a Host ticket. Same response from the server. I can connect to the pg server via an ssh keypair, so I have a backdoor to debug. ssh in as root and: let me see if there is an sshd log.

Apr 25 22:58:30 pg sshd[6115]: Invalid user ayoung from 10.10.59.141
Apr 25 22:58:30 pg sshd[6115]: input_userauth_request: invalid user ayoung [preauth]
Apr 25 22:58:30 pg sshd[6115]: Disconnecting: Too many authentication failures for ayoung [preauth]

Hmm. Invalid user. Sounds like a getent failure of some sort.

Is sssd running?

systemctl status sssd.service
...
 Active: active (running) since Mon 2013-04-22 14:25:39 UTC; 3 days ago

Yep. OK, what about nsswitch setup?

passwd:     files sss

That looks right.  Should check in /etc/passwrd and then talk to sss, which should talk to IPA.  Lets see if that is the case....nothing in 
/var/log/sssd/sssd_ssh.log
/var/log/sssd/sssd.log
/var/log/sssd/sssd_nss.log

How about /var/log/secure? Same as the sshd log.

 ping ipa
ping: unknown host ipa\

AHA! Rebooting did a new dhcp request and probably overwrote my /etc/resolve.conf file….lets look:

[root@pg ~]# cat /etc/resolv.conf 
# Generated by NetworkManager
domain novalocal
search novalocal
nameserver 192.168.0.3

My internal was 192.168.0.45 for IPA….OK, we have at least one culprit. Change it to:

[root@pg ~]# cat /etc/resolv.conf 
# Generated by NetworkManager
domain openstack.freeipa.org
search openstack.freeipa.org
domain novalocal
nameserver 192.168.0.45

And now

[root@pg ~]# getent passwd ayoung
ayoung:*:1615800005:1615800005:Adam Young:/home/ayoung:/bin/sh

So…here is the fix: add the following to /etc/dhcp/dhclient.conf


interface "eth0" {
           supersede domain-name  "openstack.freeipa.org";
           supersede domain-search  "openstack.freeipa.org";
           supersede domain-name-servers 192.168.0.45;
       }

And the resolv.conf data survives a reboot.

IPTables rules for FreeIPA

I end up editing this so much, figure I’d post it here for all to use.  This is the standard IPtables config file augmented with those rules required to let through the protocols supported by FreeIPA

# Firewall configuration written by system-config-firewall
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT

#TCP ports for FreeIPA
-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 443  -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 389 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 636 -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 88  -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 464  -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 53  -j ACCEPT

#UDP ports for FreeIPA
-A INPUT -m state --state NEW -m udp -p udp --dport 88 -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --dport 464 -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --dport 123 -j ACCEPT
-A INPUT -m state --state NEW -m udp -p udp --dport 53 -j ACCEPT

-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

Trusts and Role Based Access Control for Open Stack

Bearer tokens are vulnerable to replay attacks. OK, so what are our options? Something where the user proves, via cryptography that they have the right to actually use the token. It doesn’t matter if it is X509, Kerberos, or something we cook up ourselves, it is going to resolve to proving you have the right to use that token.

If tokens must be validated by the owner, we effectively break the ability of Open Stack to hand around bearer tokens to get work done. We are going to have to get a lot of stuff right in order to keep from breaking things. Fortunately, we now have the tools to work around this, and to better secure an OpenStack system: Trusts and Role Based Access Control.
Continue reading