python tools/install_venv.py

Code coverage can be generated using:

./run_tests.sh -c

Which will generate a report in keystone/covhtml/. An example one is posted here:

Why Kerberize Postgresql

Direct access to the RDBMS might be required for many reasons.

• A shared instance between servers
• The database might be in a large replicated cluster managed as a service for the enterprise
• The database instance  might provide a read only snapshot of live data for reporting
• Some applications might use the Database as a persistant RPC mechanism

In the case of OpenStack, we want to make Keystone highly available.  As such, each Keystone instance will not get its own database instance, but instead will share a back end.

Puppetized Install and Configuraton

From a shell prompt:

Yum install puppet puppet-server tar postgresql
puppet module install puppetlabs/postgresql

Create a site.pp file for applying gss api to the pg_hba.conf file:

class { 'postgresql::server':
  config_hash => {
    'ip_mask_deny_postgres_user' => '0.0.0.0/32',
    #do not explicitly set 'ip_mask_allow_all_users' 
    #and it will default to localhost only
    'listen_addresses'           => '*',
    'manage_redhat_firewall'     => true,
  },
}
postgresql::pg_hba_rule { 'allow application network to access app database':
  description => "Open up postgresql for access from 192.168.0/24",
  type => 'host',
  database => 'all',
  user => 'all',
  address => '192.168.0.0/24',
  auth_method => 'gss'
}

Apply it with

 puppet apply --verbose /root/site.pp

Check the postgres access controls in /var/lib/pgsql/data/pg_hba.conf
You need a line like this.

host    all     all     192.168.0.0/24  gss

Make sure you don’t have some other rule that will conflict with it. For example, In an earlier pass I had to comment out:

#host   all     all     0.0.0.0/0       md5
#host   all     all     ::1/128 md5

Which preceded it and were triggering a password request from the psql command.

Kerberos for Postgres: Create new service in IPA.

ipa service-add postgres/pg.openstack.freeipa.org
ipa-getkeytab -s ipa.openstack.freeipa.org -p postgres/pg.openstack.freeipa.org@OPENSTACK.FREEIPA.ORG  -k /var/lib/pgsql/data/pg.keytab
chown postgres:postgres /var/lib/pgsql/data/pg.keytab

Postgres Config

Edit Postgresql.conf

The information to do this is out of the Postgres manual

# Kerberos and GSSAPI
krb_server_keyfile = '/var/lib/pgsql/data/pg.keytab'
krb_srvname = 'postgres'
host    all     all     192.168.0.0/24  krb5

Firewall:

Either iptables open port 5432:

lokkit -p 5432:tcp

Or open it with firewall-cmd:

firewall-cmd --add-port=5432/tcp

To Test:

psql -h pg.openstack.freeipa.org -d keystone -U keystone

Run klist afterwards to see the Postgres service ticket:

Ticket cache: FILE:/tmp/krb5cc_1615800001
Default principal: keystone@OPENSTACK.FREEIPA.ORG

Valid starting     Expires            Service principal
05/02/13 03:31:28  05/03/13 03:31:28  krbtgt/OPENSTACK.FREEIPA.ORG@OPENSTACK.FREEIPA.ORG
05/02/13 03:31:33  05/03/13 03:31:28  postgres/pg.openstack.freeipa.org@OPENSTACK.FREEIPA.ORG

On the Keystone side install Postgres client libraries for Keystone

yum install python-psycopg2 postgresql

In /etc/keystone/keystone.conf

connection = postgresql://pg.openstack.freeipa.org/keystone?krbsrvname=postgres

Assuming you are going to run this for a non-interactive service, you will need a cron job to fetch the tgt on a regular basis.

crontab /etc/keystone/keystone.crontab
1 0,6,12,18 * * *   su - keystone -c "KRB5CCNAME=FILE:/tmp/krb5cc_1615800001 kinit keystone -k -t /var/kerberos/krb5/user/1615800001/client.keytab"

For Postgresql:

sudo yum install postgresql-devel
. .venv/bin/activate
pip install psycopg2

For MySQL

sudo yum install mysql-devel
. .venv/bin/activate
pip install MySQL-python

Now to run the unit tests, uncomment the driver line in tests/backend_sql.conf and run

./run_tests.sh -x test_sql_upgrade

If tokens must be validated by the owner, we effectively break the ability of Open Stack to hand around bearer tokens to get work done. We are going to have to get a lot of stuff right in order to keep from breaking things. Fortunately, we now have the tools to work around this, and to better secure an OpenStack system: Trusts and Role Based Access Control.

The use case I came up with was “create VM from CLI in a very simplified state.

The actors here are

1. Me, the end user (ayoung with multiple roles in the project younglogic)
2. Keystone
3. Nova API
4. Quantum
5. Glance
6. Swift
7. Nova Compute

We’ll lump all of Quantum’s, Swift’s, and Glance’s separate components together for now.

So I run the CLI command saying “create me a new virtual machine.” At this point, what do I have? I have an unscoped token from Keystone that allows horizon to list my projects, and I probably have a scoped token from keystone with my roles in the younglogic project. Lets assume that there are not transferable: any action I take will get signed by me, and if the service that I pass my token on to needs to call another it won’t be able to use my token. It will have to use its own.

So the token that I’ve gotten out of keystone goes to Nova. The first thing that Nova needs to do is move the image from Glance to the Nova Compute. Nova needs a token the will give it permission to do that. Say that API is protected by the role “Consumes Images” I need to create a trust for the Nova server with:

{trustor=ayoung, trustee=nova-api,project=younglogic,roles=["Consumes Images"]} I probably should give it a relatively short time out: say an hour, as we might be queued up. Impersonation is not required, as this is done at the project level.

Glance already has a trust set up for me to talk to swift. That trust allows the glance service user to access all of my images. Since the images are store by users and not by projects the glance trust is an impersonation trust. How does glance know to use this trust for images in my project when the request came in from the Nova user? Good question, and it is an issue yet to be resolved. But a mapping is possible.

Now nova needs to push the image to nova-compute. Again, this is done via the trust token, and it probably makes sense that the same role, “Consumes Images” be used to push the image to the compute node. While Nova Compute may trust Nova API, by using the trust, we assure that the resources are assigned to the project.

Now we have to go to Quantum and add the new IP address to the network for this Virtual machine. Again, we can use the “Consumes Images” role, but that seems a little specific. Probably more correct to give a role along the lines of “Modifies Network Addresses” as that role could be used for many things. However, both of these roles could be on the initial trust token. So {trustor=ayoung, trustee=nova-api,project=younglogic,roles=["Consumes Images","Modifies Network Addresses"]}

Now we need to actually create a virtual machine from the image. Lets assume that there are people that would be able to create a virtual machine only if the image was in place already, but could not deploy a new image. Thus, we will grant yet a third role “Creates VM”. And a fourth “Power Cycles VM”. It depends on the granularity of trust we want to place around the operations. So our completed token looks roughly like this:

{trustor=ayoung, trustee=nova-api,project=younglogic,roles=["Consumes Images","Modifies Network Addresses", "Creates VM", "Power Cycles VM"], expires=now + 1 hour}

This is pseudo-code: these user IDs would be uuids and the time would be an absolute time, not relative.

Note now that it is the trust that lasts for an hour. However, we can make the tokens very short lived. The rule of thumb should be “long enough to push a request and get a response via HTTP.” In the case of Nova API making calls to Glance, Quantum, and Nova Compute, it can continue to reuse a token until it is about to expire, an the get a new one, so that if the quantum call returns immediately, it doesn’t need to get a new token before going to Nova Compute. If, on the other hand, the glance call takes 15 minutes, it only needs the token to be valid at the start of the call, and it can fetch a new token once it needs it.

This example is obviosuly the abbreviated version of what happens. There are lots of details elided for the sake of clarity. What is important is how Trusts can be used in conjunction with RBAC to provide a more secure Open Stack deployment.

openstack-db fails with a permission on the root user.  However, the following works:

1. As the keystone user (I suspect the openstack-db call made the keystone user, or maybe that is done by the RPM install?)
2. run mysql,  (no params, using the default identification, which I assume is PAM based?)
3. create a user named keystone.
4. and grant that user perms to create a db.

su - keystone
mysql

create user 'keystone'@'localhost' identified by 'keystone';
grant all  PRIVILEGES on *.* to 'keystone'@'localhost';

exit mysql and log in as that user:

mysql --user=keystone --password=keystone

Create the keystone database:

create database keystone;

Log out and run the dbsync

keystone-manage db_sync

Obviously, this leaves the DB User with too many permissions, but it is a start.

If I now try to run the command

openstack-db --service glance --init
Please enter the password for the 'root' MySQL user:

Even setting the password in MySQL doesn’t work

UPDATE mysql.user SET Password=PASSWORD('keystone') WHERE User='root' AND Host='localhost';

[root@f18-keystone mysql]# openstack-db --service glance --init
Please enter the password for the 'root' MySQL user:
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
Failed to connect to the MySQL server.  Please check your root user credentials.

I tried it with the unix password as well.

Note that I can connect using the following SQL Alchemy URL:

connection = mysql://keystone:keystone@localhost/keystone?unix_socket=/var/lib/mysql/mysql.sock

I think this is preferable to exposing TCP sockets around in the case that the Keystone server and MySQL server are co-located.

When and why did you join Red Hat ?

I Started in July of 2009. Why? There is a Long answer to that.

What part of the world are you based in ?
Massachusetts. I work out of the Westford office, which is our Engineering Headquarters.

What were you doing before joining the OpenStack Team?
I worked on FreeIPA, which Red Hat packages as Red Hat Identity Management.

What were you doing before joining Red Hat ?
Immediately prior to joining Red Hat I was at VMware. I was nominally on the High Availability team, but all my efforts were around either running Open Source tools on the VMware platform or getting VMware tools to run on Linux. Much more fun was what I was doing prior to VMware.

In the middle of the Decade, I worked at Penguin Computing on Scyld Beowulf, now Scyld ClusterWare. This was a lot of fun. We were building a single system image where the process space was distributed across multiple physical compute nodes. Although we targeted it at the High Performance Computing market, it was solution that had something to offer to many aspects of large scale computing.

What OpenStack projects are you involved in?
I work almost exclusively on Keystone. I seem to have become the clearing house for LDAP support, but others are picking that up. I am more focuses on HATEOAS, PKI, and the trusts work, as well as general concepts around Identity Management.

What have you contributed to OpenStack in the last 3-6 months ?

For the Essex release, my biggest contribution was reintroducing LDAP support to Keystone. There has been a fair amount of follow on work from that.

For the Folsom release, my biggest contribution was PKI tokens.

As a member of Keystone Core, my biggest contribution is doing code reviews.

Thus far in Grizzly I’ve done a fair number of bug fixes and refactoring. I’ve spent a good amount of time building up the SQL upgrade tests and dealing with issues specific to our threading model.  Probably the most non-interesting but essential thing has been to change how users are assigned to projects.  In order to tie in with Role Based Access Control, that assignment is now done via a Role as well.  It seems small, but it is an essential change, and other things will work better because of it.

What are you working on currently / in the next few months for OpenStack ?
My current tasks involve mechanisms for delegating authority, whether between users or services. More information here:

What do you enjoy most about working on OpenStack ?
The people that are involved with Keystone are top notch, and yet there is a surprising lack of Ego. Joe Heck is one Heck of a tech lead, and Dolph Matthews is just fantastic. We recently picked up a couple more core developers who have really helped move things along: Guang Yee and Henry Nash. The member of non-core contributors is growing, and several of them have picked up pieces of work that were originally on my plate, and proceeded to implement them better than I would have. We all have out interests, and they seem to be complementary. So, one of my roles has been to help get people up to speed in the development process.

The automation support for the workflow is one of the smoothest I’ve had the pleasure to work with: Open Blueprint, record Bug, write code, submit to Gerrit. Making changes due to feedback from code reviews has become painless as I’ve better learned git. Keeping track of what I need to do, including new change requests that are posted for review has never been easier for me. Which is good, because there is a lot to do.

But most of all, I love the fact that Open Stack is generating so much excitement. I feel lucky to be paid to work on it full time. Which is also a part of my answer to the question of Why I work at Red Hat.

What advice do you have for people interested in OpenStack ?
Get involved. Code reviews are always welcome. Nothing will get you in better with the existing team members than providing input. Documentation fixes, bug reports, comments on blueprints, and testing. Get involved. Open Stack is a collaborative effort. Collaborate.

What other open source projects have you been involved in ?
While here at Red Hat, I first worked on RHQ, and later FreeIPA, with a touch of Pulp and Candlepin in the middle. At one point I was focused on the Certificate Server that is used by FreeIPA.
I am still working to get BProc into more of an Open Source development model. Even though it is GPL code, the development has been done in house for too many years.
My first Open Source contribution was to a PalmOS application for studying Chinese called Dragon Character Training where I built a “game” to test you in the pronunciation of the Chinese words.

When did you first start using Linux and what distro was it ?
1998. Red Hat 6. A good friend of mine, Mike DeBeer, helped get me over the initial learning curve. I’ve bounced back and forth between Red Hat and Debian over the years.

What desktop environment are you running ?
I run Gnome 3 and Fedora 18 right now, basically because I try to keep current and support what the Fedora and the Gnome teams are actively developing. I switch to KDE from time to time, as I actually am a C++ coder at heart.

What are your preferred programming language and/or dev tools ?
All programming languages have their sweet spot, and all have their warts. I’m learning to think in Python more and more as I work on Open Stack. I’ve done a fair amount of C++, Java and Straight C in the past, and am pretty handy with Bash as well.

For coding, I am an IDE person. I got very comfortable using Eclipse during my Java coding days. Now that I am focused on Python, I use Eclipse PyDev as my primary IDE, and often fall back to EMACS. I am also conversant in vi and tend to use that for very quick tasks.

Where is your blog / twitter / Google+ / YouTube profile found ?
Blog is http://adam.younglogic.com. I don’t tend to use much of the other technologies. I have a Google+ account, but my launchpad ID is far more used.

What else would you like to say to the OpenStack community?
We need more code reviews. The more eyeballs on the code, the better. Use code reviews as a way to get to learn the code.

sudo yum install puppet
sudo puppet module install puppetlabs/postgresql

Then create a file /etc/puppet/site.pp

Put this line in it:

class { 'postgresql::server': }

postgresql::db{ 'keystone':
  user          => 'keystone',
  password      => 'keystone',
  grant         => 'all',
}

sudo  puppet apply --verbose /etc/puppet/site.pp

Confirm that postgresql is running:

systemctl status postgresql.service

Should get you

postgresql.service - PostgreSQL database server
	  Loaded: loaded (/usr/lib/systemd/system/postgresql.service; disabled)
	  Active: active (running) since Thu, 03 Jan 2013 13:26:46 -0500; 43min ago
	 Process: 17529 ExecStop=/usr/bin/pg_ctl stop -D ${PGDATA} -s -m fast (code=exited, status=0/SUCCESS)
	 Process: 17553 ExecStart=/usr/bin/pg_ctl start -D ${PGDATA} -s -o -p ${PGPORT} -w -t 300 (code=exited, status=0/SUCCESS)
	 Process: 17545 ExecStartPre=/usr/bin/postgresql-check-db-dir ${PGDATA} (code=exited, status=0/SUCCESS)
	Main PID: 17556 (postgres)
	  CGroup: name=systemd:/system/postgresql.service
		  ├ 17556 /usr/bin/postgres -D /var/lib/pgsql/data -p 5432
		  ├ 17557 postgres: logger process   
		  ├ 17559 postgres: writer process   
		  ├ 17560 postgres: wal writer process   
		  ├ 17561 postgres: autovacuum launcher process   
		  └ 17562 postgres: stats collector process

Test we can connect to the PostgreSQL command line tool:

psql -h localhost -U keystone keystone 
Password for user keystone: 
psql (9.1.7)
Type "help" for help.

keystone=> \d

To run the Keystone unit test test against the database, alter the file /opt/stack/keystone/tests/backend_sql.conf. Comment out the sqlite connection line, and uncomment the postgresql line.

[sql]
#connection = sqlite://
#To Test MySQL:
#connection = mysql://root:keystone@localhost/keystone?charset=utf8
#To Test PostgreSQL:
connection = postgresql://keystone:keystone@localhost/keystone?client_encoding=utf8
idle_timeout = 200

Then you can run the unit tests with

./run_tests.sh -N test_sql_upgrade

If the tests fail (and they will), they will leave the database in an unusable state. You can drop the database and recreate with puppet:

sudo su postgres -c "dropdb keystone"
sudo  puppet apply --verbose /etc/puppet/site.pp

SQL upgrade scripts are in keystone/common/sql/migrate_repo/versions and are number in the order in which they are executed. The initial tables for Keystone are created in the script 001_add_initial_tables.p

I am trying to Replace Tenant-User Membership with a default role. The first step is to create the default role.

To start with, I am going to create a new role called “Member” in the Roles table. If a role with this name already exists, I will leave it in place.

This makes downgrade a little tricky. I won’t know if the “member” role existed on upgrade. Ordinarily, I would remove this role, but I can’t without breaking some users deployments. Thus, the downgrade for this is going to be a no-op.

As of the time of this writing, the highest number script is 013. So to start, I create a new script with a sequence number of 014: 014_membership_role.py. I make it as simple as possible.

def upgrade(migrate_engine):
    pass

def downgrade(migrate_engine):
    pass

To run the upgrade tests from the command line.

./run_tests.sh test_sql_upgrade

That seems to imply that all is well. But we don’t really know if anything was done there. I, of course, ran in the debugger, so I could set a break point, but if you are not, all you know is it seems to work. Since we are espousing test driven design, lets make sure we run the upgrade test…at least for very limited values of “sure”.

Open up the file tests/test_sql_upgrade.py and add a new test: I will call it: upgrade_default_roles.

    def test_upgrade_default_roles(self):
        self.upgrade(13)
        self.upgrade(14)

And re-run the test. Since we are going to be focusing on this test, we can speed things up to run just this test:

./run_tests.sh test_sql_upgrade:SqlUpgradeTests.test_upgrade_default_roles
test_upgrade_default_roles (test_sql_upgrade.SqlUpgradeTests) ... ok

If you are paranoid, you will once again admit that you don’t know that this did anything. I like it when new test fail. So, to start with, I am going to explicitly throw an exception in my upgrade script.

from keystone import exception

def upgrade(migrate_engine):
    raise exception.NotImplemented()

def downgrade(migrate_engine):
    pass

Now my output starts with:

./run_tests.sh test_sql_upgrade:SqlUpgradeTests.test_upgrade_default_roles
test_upgrade_default_roles (test_sql_upgrade.SqlUpgradeTests) ... ERROR

And continues on with information to help in debugging.

OK, I run with my tests directory in a ram disk. This speeds up running the tests, but means that if I don’t commit my changes, I’ll lose them on a reboot. I also might get distracted and have to work on something else. So, I commit, to git.

young@ayoung530 keystone (project_member)]$ git status
# On branch project_member
# Changes not staged for commit:
#   (use "git add ..." to update what will be committed)
#   (use "git checkout -- ..." to discard changes in working directory)
#
#	modified:   tests/test_sql_upgrade.py
#
# Untracked files:
#   (use "git add ..." to include in what will be committed)
#
#	keystone/common/sql/migrate_repo/versions/014_membership_role.py
no changes added to commit (use "git add" and/or "git commit -a")
[ayoung@ayoung530 keystone (project_member)]$ git add tests/test_sql_upgrade.py  keystone/common/sql/migrate_repo/versions/014_membership_role.py
[ayoung@ayoung530 keystone (project_member)]$ git commit -m "start of default role work."
[project_member 2115c76] start of default role work.
 2 files changed, 13 insertions(+)
 create mode 100644 keystone/common/sql/migrate_repo/versions/014_membership_role.py

Now, if I come back to this branch later, I run my tests, they fail, and I am immediately back in the coding zone.

I can modify my upgrade script to always insert the new role. I’ll put in a corresponding downgrade script that deletes the row. Since my database is empty to start, I know this will succeed.

import sqlalchemy
import uuid
from keystone import exception

def upgrade(migrate_engine):
    meta = sqlalchemy.MetaData()
    meta.bind = migrate_engine
    role_table = sqlalchemy.Table('role', meta, autoload=True)
    conn = migrate_engine.connect()
    conn.execute(role_table.insert(),
                 id=uuid.uuid4().hex, name="Member", extra="{}")


def downgrade(migrate_engine):
    meta = sqlalchemy.MetaData()
    meta.bind = migrate_engine
    role_table = sqlalchemy.Table('role', meta, autoload=True)
    conn = migrate_engine.connect()
    conn.execute(role_table.delete().where(role_table.c.name=='Member'))

While running the test shows that this succeeds, it doesn’t do much. What I will now do is add a test that queries the table to make sure the new value is in there. The following test runs both the upgrade and the downgrade. This is an interim step. If we were going to keep the downgrade around, we would make a separate unit test for upgrade and for downgrade.

    def test_upgrade_default_roles(self):
        self.upgrade(13)
        session = self.Session()
        count = session.execute("select count(*) as c from role where name='Member'").fetchone()['c']
        self.assertEquals(0, count)
        self.upgrade(14)
        count = session.execute("select count(*) as c from role where name='Member'").fetchone()['c']
        self.assertEquals(1, count)
        self.downgrade(13)
        count = session.execute("select count(*) as c from role where name='Member'").fetchone()['c']
        self.assertEquals(0, count)

Of course, now other unit tests fail, but that is a tale for another day.

Right now, there are only three active core contributors focused on Keystone.  There are a couple people that are core on multiple projects the pay attention to Keystone from time to time, but mostly it is just three of us.

How the review scoring works

On Keystone, we need a total of +4 before merging a commit. This means:

Two core at +2 each.
or
Two community at +1 each and One core at +2.

I could say four community at +1 each, but a core member still needs to click “approve” so their name goes on it anyway.

If the patch is created by one of the core team members, they can’t review or approve it.  If we rely only on the core developers, and one of use writes the patch, all three of us have to be involved. That is fine for really important things, and sometimes we want to hold a patch until the other core see it.  But it doesn’t scale.

How to get involved

Find a review request and review it.  Even if it has been reviewed by someone else. If anything at all seems out of place, -1 it and place a reason on it. If the original author thinks your reason is not sufficient, they will let you know. You can decide remove it later.  If you review something and you don’t understand it, put questions into the review. Feel free to -1 it until you understand the rationale. We are very unlikely to merge something that has a -1 next to it, but we have enough confidence in our convictions that we will do so if we feel warranted. Most of all, if a code review has a -1, the person who posted the patch will want to address it quickly, as most people won’t bother reviewing a patch that already has a -1.  If you criticism is at all warranted, the author will address it and submit another patch.

I assure you that we will take a review more seriously if it has negative comments and constructive criticism than if someone just says “Looks Good.”  If the patch is non-trivial, it will have something that can be improved.

Ask question, get involved, and keep things moving. I assure you, it means that once you post code for review, your review requests will get more attention as they come in.

What to review

You may be wondering how to track what code reviews are out there.

For the Keystone Server, the review requests are here.

Here are the review requests for the client.

You will need a launchpad account. More information here.

How to get comfortable better at code reviews

So, how do you review code? Good question. It can be daunting looking at a brand new code base, and a change you don’t understand. Make the review process be your way of understanding the code better.  I’d recommend actually running  the code.   My previous post talked through how to get to that start state.  Set a break point and step through the code.  Look at the parameters that are sent in to the functions.

Here is a list of things you can think about while looking at the code.

There are some things that are specific to Open Stack worth mentioning.

• Eventlet is a greenthreaded Web server.  Blocking calls in native libraries can make it hang.
• We want to be able to support Apache HTTPD as well as Eventlet, so Eventlet specific code needs to be isolated.
• You can see the code coverage by running ./run_tests.sh –coverage, which will generate an HTML based report in keystone/covhtml. Look at the code that was submitted and make sure it is 100% covered
• No code will get merged if it can’t pass the unit tests and functional tests.  Don’t be worried that you will approve something that is broken.
• If a patch is submitted without a blueprint or a bug, and it is not clear what problem the patch is solving, that is sufficient grounds for a -1.   The code may be perfect, but it needs to be documented.
• We are very concerned about backward compatibility.  Keep an eye on any patches that may change previous behavior.

Feel free to ask me questions. As you can tell, I am not shy, and I try to be n00b13 friendly, especially if you are showing an interest and trying to make Keystone and Open Stack better.

If you are wondering how to get involved in Open Stack, I suggest you do some code reviews.