Since running the Cyrus SASL sample server and client was not too bad, I figured I would see what happened when I tried to secure it using Kerberos.
Table of contents
Mechanisms
I’m going to run this on a system that has been enrolled as a FreeIPA client, so I start with a known good Kerberos setup.
To see the list of mechanisms available, run
sasl2-shared-mechlist
I have the following available.
Available mechanisms: GSS-SPNEGO,GSSAPI,DIGEST-MD5,CRAM-MD5,ANONYMOUS Library supports: ANONYMOUS,CRAM-MD5,EXTERNAL,DIGEST-MD5,GSSAPI,GSS-SPNEGO
For Kerberos, I want to use GSSAPI.
Lets do this the hard way, by trial and error. First, run the server, telling it to use the GSSAPI mechanism
/usr/bin/sasl2-sample-server -p 1789 -h localhost -s hello -m GSSAPI
Then run the client in another terminal:
sasl2-sample-client -s hello -p 1789 -m GSSAPI localhost
Which includes the following in the output:
starting SASL negotiation: generic failure
SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)
closing connection
Kerberos
I need a Kerberos TGT in order to get a service ticket. Use kinit
$ kinit admin Password for admin@AYOUNG-DELL-T1700.TEST:
This time the error message is:
starting SASL negotiation: generic failure
SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server rcmd/localhost@AYOUNG-DELL-T1700.TEST not found in Kerberos database)
I notice two things, here:
- The service needs to be in the Kerberos servers directory.
- the service name should match the hostname.
If I rerun the command using the FQDN of the server, I can see the service name as expected:
$ sasl2-sample-client -s hello -p 1789 -m GSSAPI undercloud.ayoung-dell-t1700.testreceiving capability list... ... SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server hello/undercloud.ayoung-dell-t1700.test@AYOUNG-DELL-T1700.TEST not found in Kerberos database) closing connection
So I tried to create the service in the ipa server:
ipa service-add Principal: hello/overcloud.ayoung-dell-t1700.test@AYOUNG-DELL-T1700.TEST ipa: ERROR: Host does not have corresponding DNS A/AAAA record [stack@overcloud ~]$ ipa service-find
Strange error, I don’t understand, as the Host does have an A record.
Work around it with Force:
ipa service-add --force hello/undercloud.ayoung-dell-t1700.test@AYOUNG-DELL-T1700.TEST
Success:
------------------------------------------------------------------------------ Added service "hello/undercloud.ayoung-dell-t1700.test@AYOUNG-DELL-T1700.TEST" ------------------------------------------------------------------------------ Principal: hello/undercloud.ayoung-dell-t1700.test@AYOUNG-DELL-T1700.TEST Managed by: undercloud.ayoung-dell-t1700.test
OK, lets try running this again.
sasl2-sample-client -s hello -p 1789 -m GSSAPI ... SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (KDC has no support for encryption type)
Keytabs
OK, I’m going to guess that this is because my remote service can’t deal with the Kerberos service tickets it is getting. Since the service tickets are for the principal: hello/undercloud.ayoung-dell-t1700.test@AYOUNG-DELL-T1700.TEST it needs to be able to decrypt requests using a key meant for this principal.
Fetch a keytab for that principal, and put it in a place where the GSSAPI libraries can access it automatically. This place is:
/var/kerberos/krb5/user/{uid}
Where {uid} is the numeric UID for a users. In this case, the users name is stack and I can find the numeric UID value using getent.
KRB5_KTNAME=/var/kerberos/krb5/user/1000/client.keytab
ipa-getkeytab -p hello/undercloud.ayoung-dell-t1700.test@AYOUNG-DELL-T1700.TEST -k client.keytab -s identity.ayoung-dell-t1700.test Keytab successfully retrieved and stored in: client.keytab $ getent passwd stack stack:x:1000:1000::/home/stack:/bin/bash $ sudo mkdir /var/kerberos/krb5/user/1000 $ sudo chown stack:stack /var/kerberos/krb5/user/1000 $ mv client.keytab /var/kerberos/krb5/user/1000
Restart the server process, try again, and the log is interesting. Here is the full client side trace.
$ sasl2-sample-client -s hello -p 1789 -m GSSAPI undercloud.ayoung-dell-t1700.test
receiving capability list... recv: {6}
GSSAPI
GSSAPI
please enter an authorization id: admin
using mechanism GSSAPI
send: {6}
GSSAPI
send: {1}
Y
send: {655}
`[82][2][8B][6][9]*[86]H[86][F7][12][1][2][2][1][0]n[82][2]z0[82][2]v[A0][3][2][1][5][A1][3][2][1][E][A2][7][3][5][0] [0][0][0][A3][82][1][82]a[82][1]~0[82][1]z[A0][3][2][1][5][A1][18][1B][16]AYOUNG-DELL-T1700.TEST[A2]503[A0][3][2][1][3][A1],0*[1B][5]hello[1B]!undercloud.ayoung-dell-t1700.test[A3][82][1] 0[82][1][1C][A0][3][2][1][12][A1][3][2][1][1][A2][82][1][E][4][82][1][A]T[DD][F8]B[F4][B4]5[D]`[A3]![EE][19]-NN[8E][F5][B7]{O,#[91][A4]}[86]k[D5][EE]vL[E4]&[6][3][A][1C][91][A5][A7][88]j[D1][A3][82][EC][A][D6][CB][F3]9[C][13]#[94][86]d+[B8]V[B7]C^[C6][A8][16][D1]r[E4][0][B9][2][2]&2[E5]Y~[C1]\([BA]x}[17][BC][D][FC][D5][CA][CA]h[E4][A1][81].[15][17]?[CA][A][8B]}[1C]l[F0][D9][E8][96]3<+[84][E7]q.[8E][D5][6][1C]p[E6][6]v[B0][84]5[9][B7]w[D6]3[B8][E3][5]T[BF][92][AA][D5][B3][[83]X[C0]:[BA]V[E5]{>[A5]T[F6]j[CB]p[BF]][EF][E1][91][ED][C][F3]Y[4]x[8E][C2]H[E7][14]#9[EE]5[B3]=[FA][80][DD][93][EF]3[0]q~22[6]I<[EB][F9]V[D1][9D][A8][A6]:[CE]u[AE]-l[D3]"[D7][FE]iB[84][E0]]B[E][C8]U[E][FD][D2]=[F2][97][88][D3][DA]j[B4][FA][16][D1]^CE2?[9F][89]^A[E9][AF][1A]5[99][CE][7][AF]M[1A][A][CB]^[E1][BA]f[7]-n<[F8]8![A4][81][DA]0[81][D7][A0][3][2][1][12][A2][81][CF][4][81][CC][91][F0][A]D[91][F6][FA][F4][B9][13][DF]d|[F4]Y[DF][9E]M[A2]f[11][15]x[C5]-|Qt[F4]nL>@[F4][18][FF],[F6][B5]F6[EC]+[C3]V[F1][81][97][E2][1D]i[4]wD&[9A]V[CE][A1][16][D7]4[E0]C[B]O[D1]v[DD][E9][84]lW[DA]%[F6]v[93]<m"SAfiF[8E][[95]"[CC][D2]4[FA]_[FB]i[E7][D4]M[AE][5][82][FF][D7][0][8C]6[8D][B0]3[F8][E3][B4]P[9C][9E][A2]`[7]U[F7][1D]zub[E0]([A9]P>[AE]f[1A][B1][80][A0]}s[EA][D1]Zk[FF]n_S[9E]rK[E5]n [85]#[DB][FF][B3][E2][19];[F5][E2][8A]>2[E5][A4][81][E8]z[9D][E3][BC][C8][87][F]:[81]7[C9]ix[1E]5[15])[8D][9D][C7][DB][13][98][97][C7]C[6]q[D2][C1][ED][B3]:[E0]
waiting for server reply...
authentication failed
closing connection
On the server side, it looks similar, but ends like this:
starting SASL negotiation: generic failureclosing connection
It is not a GSSAPI error this time. To dig deeper, I’m going to look at the source code on the server side.
Debugging
I’ll shortcut a few steps. Install both gdb and the debugInfo for the sample code:
sudo yum install gdb sudo debuginfo-install cyrus-sasl-devel-2.1.26-20.el7_2.x86_64
Note that the version might change for the debuginfo.
The source code is included with the debuginfo rpm:
$ rpmquery --list cyrus-sasl-debuginfo-2.1.26-20.el7_2.x86_64 | grep server.c /usr/src/debug/cyrus-sasl-2.1.26/lib/server.c /usr/src/debug/cyrus-sasl-2.1.26/sample/server.c
Looking at the server code at line 267 I see:
if (r != SASL_OK && r != SASL_CONTINUE) {
saslerr(r, “starting SASL negotiation”);
fputc(‘N’, out); /* send NO to client */
fflush(out);
return -1;
}
Let’s put a breakpoint at line 255 above it and see what is happening. Here is the session for setting up the breakpoint:
$ gdb /usr/bin/sasl2-sample-server ... (gdb) break 255 Breakpoint 1 at 0x2557: file server.c, line 255. (gdb) run -h undercloud.ayoung-dell-t1700.test -p 1789 -m GSSAPI
Running the client code gets as far as prompting for the please enter an authorization id: admiyo
This is suspect. We’ll come back to it in a moment.
Back on the server, now, we see the breakpoint has been hit.
Breakpoint 1, mysasl_negotiate (in=0x55555575c150, out=0x55555575c390, conn=0x55555575a6e0)
at server.c:255
255 if(buf[0] == 'Y') {
Missing separate debuginfos, use: debuginfo-install keyutils-libs-1.5.8-3.el7.x86_64 libdb-5.3.21-19.el7.x86_64 libselinux-2.2.2-6.el7.x86_64 nss-softokn-freebl-3.16.2.3-14.2.el7_2.x86_64 openssl-libs-1.0.1e-51.el7_2.7.x86_64 pcre-8.32-15.el7_2.1.x86_64 xz-libs-5.1.2-12alpha.el7.x86_64 zlib-1.2.7-15.el7.x86_64
We might need some other RPMS if we want to step deeper through the code, but for now, let’s keep on here.
(gdb) print buf
$1 = "Y", '\000' ...
(gdb) n
257 len = recv_string(in, buf, sizeof(buf));
(gdb) n
recv: {655}
`[82][2][8B][6][9]*[86]H[86][F7][12][1][2][2][1][0]n[82][2]z0[82][2]v[A0][3][2][1][5][A1][3][2][1][E][A2][7][3][5][0] [0][0][0][A3][82][1][82]a[82][1]~0[82][1]z[A0][3][2][1][5][A1][18][1B][16]AYOUNG-DELL-T1700.TEST[A2]503[A0][3][2][1][3][A1],0*[1B][5]hello[1B]!undercloud.ayoung-dell-t1700.test[A3][82][1] 0[82][1][1C][A0][3][2][1][12][A1][3][2][1][1][A2][82][1][E][4][82][1][A]T[DD][F8]B[F4][B4]5[D]`[A3]![EE][19]-NN[8E][F5][B7]{O,#[91][A4]}[86]k[D5][EE]vL[E4]&[6][3][A][1C][91][A5][A7][88]j[D1][A3][82][EC][A][D6][CB][F3]9[C][13]#[94][86]d+[B8]V[B7]C^[C6][A8][16][D1]r[E4][0][B9][2][2]&2[E5]Y~[C1]\([BA]x}[17][BC][D][FC][D5][CA][CA]h[E4][A1][81].[15][17]?[CA][A][8B]}[1C]l[F0][D9][E8][96]3<+[84][E7]q.[8E][D5][6][1C]p[E6][6]v[B0][84]5[9][B7]w[D6]3[B8][E3][5]T[BF][92][AA][D5][B3][[83]X[C0]:[BA]V[E5]{>[A5]T[F6]j[CB]p[BF]][EF][E1][91][ED][C][F3]Y[4]x[8E][C2]H[E7][14]#9[EE]5[B3]=[FA][80][DD][93][EF]3[0]q~22[6]I<[EB][F9]V[D1][9D][A8][A6]:[CE]u[AE]-l[D3]"[D7][FE]iB[84][E0]]B[E][C8]U[E][FD][D2]=[F2][97][88][D3][DA]j[B4][FA][16][D1]^CE2?[9F][89]^A[E9][AF][1A]5[99][CE][7][AF]M[1A][A][CB]^[E1][BA]f[7]-n<[F8]8![A4][81][DA]0[81][D7][A0][3][2][1][12][A2][81][CF][4][81][CC]hgdf j[CF][AE][7F]:![1C]D[F8]3^w[B7];"[3][D8]3"[8]i[9]J[D3]R[F]A[E7]![BE]0<[8][D3]'j`[B7]J[16][A9][F3][E6]=[E5]J[FE].-[A1]t[[2]W[8D]7[F3][8][EC][92][BB][A3]o5h[C1]A[CC][A2][F1][99][AA][93]2{[BA]Mx0[9D][9][CC]![A]Y[12][D8][2][95][17]ml[B4][1A][94]y[1A][BC][D2]I[8F]7Vg2[8E]6[13]:Lx[E6][1][D3][3][7]r?[12][84]3[B1][B5][AA]E)[EA][87][A][9F]Nk[D1]I[FD]{[B8]9#-[D][8]2[CC]C1[A8]Lfl[B0][E8][82][13][F9]t[1A][F6]^[8D] O13[12]L[E7][C0]k[99][E1]J[1F][FE]#[14]u[B][B2][8F][DB][E6]73*[FA][ED][11][F7][9E][B0][DC][D9][19][AB][97][D7][8B][BB]
260 r = sasl_server_start(conn, chosenmech, buf, len,
(gdb) print len
$2 = 1
(gdb) n
257 len = recv_string(in, buf, sizeof(buf));
(gdb)
260 r = sasl_server_start(conn, chosenmech, buf, len,
(gdb)
267 if (r != SASL_OK && r != SASL_CONTINUE) {
Missing separate debuginfos, use: debuginfo-install gssproxy-0.4.1-8.el7_2.x86_64
(gdb) print r
$3 = -1
A -1 response code usually is an error. Looking in /usr/include/sasl/sasl.h:
#define SASL_FAIL -1 /* generic failure */
I wonder if we can figure out why. Let’ see, first, if we can figure out what the client is sending in the authentication request. If it is a bad principal, then we have a pretty good reason to expect the server to reject it.
Let’s let the server continue running, and try debugging the client.
Client code can be found here
$ rpmquery --list cyrus-sasl-debuginfo | grep client.c /usr/src/debug/cyrus-sasl-2.1.26/lib/client.c /usr/src/debug/cyrus-sasl-2.1.26/sample/client.c
At line 258 I see the call to sasl_client_start which includes what appears to be the initialization of the data variable. Set a breakpoint there
Running the code in the debugger like this:
$ gdb sasl2-sample-client
...
(gdb) break 258
Breakpoint 1 at 0x201b: file client.c, line 258.
(gdb) run -s hello -p 1789 -m GSSAPI undercloud.ayoung-dell-t1700.test
Starting program: /bin/sasl2-sample-client -s hello -p 1789 -m GSSAPI undercloud.ayoung-dell-t1700.test
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
receiving capability list... recv: {6}
GSSAPI
GSSAPI
Breakpoint 1, mysasl_negotiate (in=0x55555575cab0, out=0x55555575ccf0, conn=0x55555575b520)
at client.c:258
258 r = sasl_client_start(conn, mech, NULL, &data, &len, &chosenmech);
(gdb) print data
$1 = 0x0
(gdb) print mech
$2 = 0x7fffffffe714 "GSSAPI"
(gdb) print conn
$3 = (sasl_conn_t *) 0x55555575b520
(gdb) print len
$4 = 6
(gdb) n
please enter an authorization id:
So it is the SASL library itself requesting an authorization ID. Let me try putting in the full Principal associated with the service ticket.
please enter an authorization id: ayoung@AYOUNG-DELL-T1700.TEST
259 if (r != SASL_OK && r != SASL_CONTINUE) {
Missing separate debuginfos, use: debuginfo-install gssproxy-0.4.1-8.el7_2.x86_64
(gdb) print r
$5 = 1
(gdb)
And from sasl.h we know that is good.
#define SASL_CONTINUE 1 /* another step is needed in authentication */
Let’s let it continue.
authentication failed
Nope. Continuing through the debugger, I see another generic failure here:
1531 } else {
1532 /* Mech wants client-first, so let them have it */
1533 result = sasl_server_step(conn,
1534 clientin,
1535 clientinlen,
1536 serverout,
1537 serveroutlen);
(gdb) n
1557 if ( result != SASL_OK
(gdb) print result
$15 = -1
Still…why is the Client side SASL call kicking into an interactive prompt? There should be enough information via the GSSAPI SASL library interaction to authenticate. The Man page for sasl_client_start even indicates that there might be prompts returned.
Looking deeper at the client code, I do see that the prompt is from line 122. The function simple at line 107 must be set as a callback. Perhaps the client code is not smart enough to work with the GSSAPI? At line 190 and 192 I see that the simple code is provided as a callback for the responses SASL_CB_USER or SASL_CB_AUTHNAME. Setting a break point and rerunning shows the id value to be 16385 or x4001.
#define SASL_CB_USER 0x4001 /* client user identity to login as */
Humility and Success
If you have followed through this far, you know I am in the weeds. I asked for help. Help, in this case,was Robbie Harwood, how showed me that the sample server/client worked OK if I ran the server as root, and userd the service host instead of hello. That gave me a succesfful comparison other to work with. I ran using strace and noticed that the failing version was not trying to read the keytab file from /var/kerberos/krb5/user/1000/client.keytab. The successful one running as root read the keytab from /etc/krb5.keytab THe failing one was trying to read from there and getting a permissions failure. The final blow that took down the wall was to realize that the krb5.conf file defined different values for default_client_keytab_name and default_keytab_name, with the latter being set to FILE:/etc/krb5.keytab. To work around this, I needed the environment variable KRB5_KTNAME to be set to the keytab. This was the winning entry:
KRB5_KTNAME=/var/kerberos/krb5/user/1000/client.keytab sasl2-sample-server -h $HOSTNAME -p 9999 -s hello -m GSSAPI
And then ran
sasl2-sample-client -s hello -p 9999 -m GSSAPI undercloud.ayoung-dell-t1700.test
Oh, one other tyhing Robbie told me was that the string I type when prompted with
please enter an authorization id:
Should be the Kerberos principal, minus the Realm, so for me it was
please enter an authorization id: ayoung