Category Archives: Networking

Reproducing an Open vSwitch Bridge Configuration

Posted on by
2

In the previous post, I described the setup for installing FreeIPA on a VM parallel to the undercloud VM setup by Tripleo Quickstart. The network on the undercloud VM has been setup up by Ironic and Neutron to listen on a network defined for the overcloud. I want to reproduce this on a second machine that is not enrolled in the undercloud. How can I reproduce the steps?

Continue reading →

Firewalld rule for Minecraft Server

Posted on by
1

My sons play Minecraft.  I recently decided to let them play head to head on the same server.  Aside from the financial aspect (I had to buy a second account) it was fairly straightforward running the server.  The one thing that tripped me up was a firewall rule that prevented a remote client machine from connecting to the server.  Fix was pretty simple.

Continue reading →

Who holds the keys to the Kingdom

Posted on by
Reply

During the years I worked as a Web application developer, it seemed like every application had its own authentication mechanism. An application developer is thinking in terms of the domain model for their application whether it be eCommerce, Systems management, photography, or weblogs. Identity Management is a cross cutting concern, and it is hard to get right. Why, then, do so many applications have “user” tables in their databases?
Continue reading →

Troubleshooting a FreeIPA install:

Posted on by
Reply

I had a handful of machines enrolled in a demo cluster. About half of them got shut down, and now I can’t SSH into them via Kerberos tickets. Here is my debugging notebook.

Tail -f the file tail -f /var/log/krb5kdc.log on the ipa server.

Start by doing a kdestroy on my home machine, and then 

kinit ayoung@OPENSTACK.FREEIPA.ORG

I see this in the IPA server.

Apr 25 22:38:56 ipa.openstack.freeipa.org krb5kdc[5728](info): AS_REQ (7 etypes {18 17 16 23 1 3 2}) 10.10.59.141: NEEDED_PREAUTH: ayoung@OPENSTACK.FREEIPA.ORG for krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG, Additional pre-authentication required
Apr 25 22:39:00 ipa.openstack.freeipa.org krb5kdc[5728](info): AS_REQ (7 etypes {18 17 16 23 1 3 2}) 10.10.59.141: ISSUE: authtime 1366929540, etypes {rep=18 tkt=18 ses=18}, ayoung@OPENSTACK.FREEIPA.ORG for krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG
Apr 25 22:39:01 ipa.openstack.freeipa.org krb5kdc[5729](info): AS_REQ (6 etypes {18 17 16 23 25 26}) 192.168.0.61: NEEDED_PREAUTH: keystone@OPENSTACK.FREEIPA.ORG for krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG, Additional pre-authentication required
Apr 25 22:39:01 ipa.openstack.freeipa.org krb5kdc[5729](info): AS_REQ (6 etypes {18 17 16 23 25 26}) 192.168.0.61: ISSUE: authtime 1366929541, etypes {rep=18 tkt=18 ses=18}, keystone@OPENSTACK.FREEIPA.ORG for krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG

Now try to hit the web UI with my browser by pointing it at:

https://ipa.openstack.freeipa.org/ipa/ui/

Klist shows no ticket…I probably need to log out first to forget the form based auth. Click log out and see a page that says:

You have been logged out
Return to main page.

Returning to the main page should do a negotiate. Lets see… nope

OK, just to be sure, I go through the browser config steps again. Then head back to the main page: and it works. Looking in the log, the interesting entries are:

Apr 25 22:44:44 ipa.openstack.freeipa.org krb5kdc[5728](info): TGS_REQ (7 etypes {18 17 16 23 1 3 2}) 10.10.59.141: ISSUE: authtime 1366929540, etypes {rep=18 tkt=18 ses=18}, ayoung@OPENSTACK.FREEIPA.ORG for HTTP/ipa.openstack.freeipa.org@OPENSTACK.FREEIPA.ORG

This shows it getting a ticket for the web UI and then klist shows

Valid starting     Expires            Service principal
04/25/13 18:39:00  04/26/13 18:39:00  krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG
	renew until 04/26/13 18:42:53
04/25/13 18:44:44  04/26/13 18:39:00  HTTP/ipa.openstack.freeipa.org@
	renew until 04/26/13 18:42:53
04/25/13 18:44:44  04/26/13 18:39:00  HTTP/ipa.openstack.freeipa.org@OPENSTACK.FREEIPA.ORG
	renew until 04/26/13 18:42:53

OK, on to those failing machines. 

ssh -vv pg.openstack.freeipa.org

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: gssapi-keyex
debug1: No valid Key exchange context
debug2: we did not send a packet, disable method
debug1: Next authentication method: gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug2: we sent a gssapi-with-mic packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic

...

Received disconnect from 10.16.16.125: 2: Too many authentication failures for ayoung

Nothing in the krb5kdc.log for that transaction, but I see that I now have a ticket for pg…must have gotten it from a prior attempt. Kdestroy, kinit and try again.

See this

Apr 25 22:50:43 ipa.openstack.freeipa.org krb5kdc[5728](info): TGS_REQ (7 etypes {18 17 16 23 1 3 2}) 10.10.59.141: ISSUE: authtime 1366930206, etypes {rep=18 tkt=18 ses=18}, ayoung@OPENSTACK.FREEIPA.ORG for host/pg.openstack.freeipa.org@OPENSTACK.FREEIPA.ORG

OK, I have a Host ticket. Same response from the server. I can connect to the pg server via an ssh keypair, so I have a backdoor to debug. ssh in as root and: let me see if there is an sshd log. 

Apr 25 22:58:30 pg sshd[6115]: Invalid user ayoung from 10.10.59.141
Apr 25 22:58:30 pg sshd[6115]: input_userauth_request: invalid user ayoung [preauth]
Apr 25 22:58:30 pg sshd[6115]: Disconnecting: Too many authentication failures for ayoung [preauth]

Hmm. Invalid user. Sounds like a getent failure of some sort.

Is sssd running?

systemctl status sssd.service
...
 Active: active (running) since Mon 2013-04-22 14:25:39 UTC; 3 days ago

Yep. OK, what about nsswitch setup?

passwd:     files sss

That looks right.  Should check in /etc/passwrd and then talk to sss, which should talk to IPA.  Lets see if that is the case....nothing in 
/var/log/sssd/sssd_ssh.log
/var/log/sssd/sssd.log
/var/log/sssd/sssd_nss.log

How about /var/log/secure? Same as the sshd log.

 ping ipa
ping: unknown host ipa\

AHA! Rebooting did a new dhcp request and probably overwrote my /etc/resolve.conf file….lets look:

[root@pg ~]# cat /etc/resolv.conf 
# Generated by NetworkManager
domain novalocal
search novalocal
nameserver 192.168.0.3

My internal was 192.168.0.45 for IPA….OK, we have at least one culprit. Change it to:

[root@pg ~]# cat /etc/resolv.conf 
# Generated by NetworkManager
domain openstack.freeipa.org
search openstack.freeipa.org
domain novalocal
nameserver 192.168.0.45

And now

[root@pg ~]# getent passwd ayoung
ayoung:*:1615800005:1615800005:Adam Young:/home/ayoung:/bin/sh

So…here is the fix: add the following to /etc/dhcp/dhclient.conf


interface "eth0" {
           supersede domain-name  "openstack.freeipa.org";
           supersede domain-search  "openstack.freeipa.org";
           supersede domain-name-servers 192.168.0.45;
       }

And the resolv.conf data survives a reboot.

Latency

Posted on by
Reply

(To the tune of Yesterday, With apologies to all four Beatles and most sys admins)

Latency
It’s the signature of HPC
that is why it’s running endlessly
your process gates on Latency

This one, well
is embarrassingly parallel
that is why its running fast as hell
the render farm works just as well

Why’s it running slow
don’t you know the bottleneck
demands for some commands
will dictate your architect ect ecture

Here’s the scoop
This one’s nothing more than data soup
that you’re running through an endless loop
You probably should try Hadoop

(This might be the only one I tag as both Lyrics and Networking)