Running the Keystone Unit tests takes a long time.
To start with a blank slate, you want to make sure you have the latest from master and a clean git repository.
cd /opt/stack/keystone git checkout master git rebase origin/master clean -xdf keystone/ time tox -r ... py27: commands succeeded ERROR: py34: commands failed pep8: commands succeeded docs: commands succeeded genconfig: commands succeeded real 8m17.530s user 33m1.851s sys 0m56.828s
The -r option to tox recreates the tox virtual environments. Additional runs should go faster
py27: commands succeeded ERROR: py34: commands failed pep8: commands succeeded docs: commands succeeded genconfig: commands succeeded real 5m52.367s user 30m57.366s sys 0m35.403s
To run just the py27 tests:
time tox -e py27 ... Ran: 5695 tests in 243.0000 sec. .... py27: commands succeeded congratulations 🙂 real 4m18.144s user 28m51.506s sys 0m31.286s
Not much faster, so we know where most of the time goes. It also reported the slowest tests:
So we have 5000+ test that take 4 minutes to run.
Running just a single test:
time tox -e py27 -- keystone.tests.unit.token.test_fernet_provider.TestFernetKeyRotation.test_rotation .... ====== Totals ====== Ran: 1 tests in 4.0000 sec. .... py27: commands succeeded congratulations 🙂 real 0m17.200s user 0m15.802s sys 0m1.681s
17 Seconds is a little long, considering the test itself only ran for four seconds of it. Once in a while is not a problem, but if this breaks the flow of thought during coding, it is problematic.
What can we shave off? Lets see if we can avoid the discovery step, run inside the venv, and specify exactly the test we want to run;
. .tox/py27/bin/activate time python -m testtools.run keystone.tests.unit.token.test_fernet_provider.TestFernetKeyRotation.test_rotation Tests running... Ran 1 test in 2.770s OK real 0m3.137s user 0m2.708s sys 0m0.428s
That seems to have had only an overhead of a second.
OK, what about some of the end-to-end test that set up an HTTP listener and talk to the database, such as those in: keystone.tests.unit.test_v3_auth?
time python -m testtools.run keystone.tests.unit.test_v3_auth Tests running... Ran 329 tests in 91.925s OK real 1m32.459s user 1m28.260s sys 0m4.669s
Fast enough for a pre-commit check, but not for “run after each change.” How about a single test?
time python -m testtools.run keystone.tests.unit.test_v3_auth.TestAuth.test_disabled_default_project_domain_result_in_unscoped_token Tests running... Ran 1 test in 0.965s OK real 0m1.382s user 0m1.308s sys 0m0.076s
I think it is important to run the tests before you write a line of code, and to run the tests continuously. But if you don’t run the entire body of unit tests, how can you make sure you are exercising the code you wrote? One technique is to put in a break-point.
I want to work on the roles infrastructure. Specifically, I want to make the assignment of one (prior) role imply the assignment of another (inferred) role. I won’t go in to the whole design, but I will start with the database structure. Role inference is a a many-to-many relationship. As such, I need to implement a table which has two IDs: prior_role_id and inferred_role_id. Lets start with the database migrations for that.
time python -m testtools.run keystone.tests.unit.test_sql_upgrade Tests running... Ran 30 tests in 3.528s OK real 0m3.948s user 0m3.874s sys 0m0.075s
OK…full disclosure, I’m writing this because I did too much before writing tests, my tests were hanging, and I want to redo things slower and more controlled to find out what went wrong. I have some placeholders for migrations: a way to keep from changing the migration number for my review as other reviews get merged/ They just execute:
def upgrade(migrate_engine): pass
So..I’m going to cherry-pick this commit and run the migration test.
migrate.exceptions.ScriptError: You can only have one Python script per version, but you have: /opt/stack/keystone/keystone/common/sql/migrate_repo/versions/075_placeholder.py and /opt/stack/keystone/keystone/common/sql/migrate_repo/versions/075_confirm_config_registration.py
Already caught up with me…
$ git mv keystone/common/sql/migrate_repo/versions/075_placeholder.py keystone/common/sql/migrate_repo/versions/078_placeholder.py (py27)[ayoung@ayoung541 keystone]$ time python -m testtools.run keystone.tests.unit.test_sql_upgrade Tests running... Ran 30 tests in 3.576s OK real 0m4.028s user 0m3.951s sys 0m0.081s
OK…lets see what happens if I put a breakpoint in one of these tests.
def upgrade(migrate_engine): import pdb; pdb.set_trace()
(py27)[ayoung@ayoung541 keystone]$ time python -m testtools.run keystone.tests.unit.test_sql_upgrade Tests running... --Return-- /opt/stack/keystone/keystone/common/sql/migrate_repo/versions/078_placeholder.py(18)upgrade()-None -> import pdb; pdb.set_trace()
Ctrl C to kill the test (0r cont to keep running). This may not always work; some of the more complex tests will do manipulations of the thread libraries, and will keep the breakpoints from interrupting the debugging thread. For these cases, use rpdb and telnet.
More info about running the tests in OpenStack can be dfound here: https://wiki.openstack.org/wiki/Testr
I wrote about using rpdb to debug here: http://adam.younglogic.com/2015/02/debugging-openstack-with-rpdb/