Setting up an RDO deployment to be Identity V3 Only

Posted on by

The OpenStack Identity API Version 3 provides support for many features that are not available in version 2. Much of the installer code from Devstack, Puppet Modules, and Packstack, all assumes that Keystone is operating with the V2 API. In the interest of hastening the conversion, I set up a deployment that is V3 only. Here is how I did it.

The order I performed these operations was:

  1. Convert Horizon
  2. Convert the Servcfice Catalog
  3. Disable the V2 API in Keystone
  4. Convert the authtoken stanze and the Endpoint config files to use discovery

Horizon

Horizon was the simplest. To change Horizon to use the V3 API, edit the local_settings. For RDO, this file is in:
/etc/openstack-dashboard/local_settings

At the end, I added:

 OPENSTACK_API_VERSIONS = {
     "identity": 3
 }
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT = True
OPENSTACK_KEYSTONE_DEFAULT_DOMAIN = 'Default'

You might want to make the default domain value something different, especially if you are using a domain specific backend for LDAP.

Service Catalog

Next up is migrating the Keystone service catalog. You can query the current values by using direct SQL.

mysql  --user keystone_admin --password=SECRETE   keystone -e "select interface, url from endpoint where service_id =  (select id from service where service.type = 'identity');"

By Default, the Responses will have V2.0 at the end of them:

+-----------+-------------------------------+
| interface | url                           |
+-----------+-------------------------------+
| admin     | http://10.10.10.40:35357/v2.0 |
| public    | http://10.10.10.40:5000/v2.0  |
| internal  | http://10.10.10.40:5000/v2.0  |
+-----------+-------------------------------+

I used SQL to modify them. For example:

mysql  --user keystone_admin --password=SECRETE   keystone -e "update endpoint set   url  = 'http://10.10.10.40:5000/v3' where  interface ='internal' and  service_id =  (select id from service where service.type = 'identity');" 
mysql  --user keystone_admin --password=SECRETE   keystone -e "update endpoint set   url  = 'http://10.10.10.40:5000/v3' where  interface ='public' and  service_id =  (select id from service where service.type = 'identity');" 
mysql  --user keystone_admin --password=SECRETE   keystone -e "update endpoint set   url  = 'http://10.10.10.40:35357/v3' where  interface ='admin' and  service_id =  (select id from service where service.type = 'identity');"

You cannot use the openstack cli to perform this; attempting to change an URL:

$ openstack  endpoint set --interface public  --service keystone http://10.10.10.40:5000/v2.0
ERROR: openstack More than one endpoint exists with the name 'http://10.10.10.40:5000/v2.0'.

I’ll Open a ticket for that.

To Use the V3 API for Operations, you are going to want a V3 Keystone RC. Here is mine:

export OS_USERNAME=admin
export OS_PROJECT_NAME=admin
export OS_PROJECT_DOMAIN_NAME=Default
export OS_USER_DOMAIN_NAME=Default
export OS_PASSWORD=SECRETE
export OS_AUTH_URL=http://$HOSTNAME:5000/v3
export OS_REGION_NAME=RegionOne
export PS1='[\u@\h \W(keystone_admin)]\$ '
export OS_IDENTITY_API_VERSION=3

Disabling V2.0

In order to Ensure you are using V3, it is worth while to disable V2.0. The simplest way to do that is to modify the paste file that controls the pipelines. On and RDO system this is /etc/keystone/keystone-paste.ini. I did it By commenting out the following lines:

#[pipeline:public_api]
# The last item in this pipeline must be public_service or an equivalent
# application. It cannot be a filter.
#pipeline = sizelimit url_normalize request_id build_auth_context token_auth admin_token_auth json_body ec2_extension user_crud_extension public_service

#[pipeline:admin_api]
# The last item in this pipeline must be admin_service or an equivalent
# application. It cannot be a filter.
#pipeline = sizelimit url_normalize request_id build_auth_context token_auth admin_token_auth json_body ec2_extension s3_extension crud_extension admin_service

and I removed them from the composites:

[composite:main]
use = egg:Paste#urlmap
#/v2.0 = public_api
/v3 = api_v3
/ = public_version_api

[composite:admin]
use = egg:Paste#urlmap
#/v2.0 = admin_api
/v3 = api_v3
/ = admin_version_api

Configuring Other services

THis setup was not using Neutron, so I only had to handle Nova, GLance, and Cinder. The process should be comparable for Neutron.

RDO adds configuration values under /use/share//-dist.conf That over ride the defaults from the python code. For example, the Nova packages has:
/usr/share/nova/nova-dist.conf. I commented out the following values, as they are based on old guidance for setting up authtoken, and are not how the Auth plugins for Keystone Client should be configured:

[keystone_authtoken]
#admin_tenant_name = %SERVICE_TENANT_NAME%
#admin_user = %SERVICE_USER%
#admin_password = %SERVICE_PASSWORD%
#auth_host = 127.0.0.1
#auth_port = 35357
#auth_protocol = http
# Workaround for https://bugs.launchpad.net/nova/+bug/1154809
#auth_version = v2.0

To set the proper values, I put the following in /etc/nova/nova.conf


[keystone_authtoken]
auth_plugin = password
auth_url = http://10.10.10.40:35357
username = nova
password = SECRETE
project_name = services
user_domain_name = Default
project_domain_name = Default
#this values is not needed unless you do not modify /usr/share/nova/nova-dist.conf
#auth_version=v3

A Big thanks to Jamie Lennox for helping me get this straight.

I made a comparable change for glance. For Cinder, the change needs to be made in /etc/cinder/api-paste.ini, but the values are comparable:

[filter:authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory
auth_plugin = password
auth_url = http://10.10.10.40:35357
username = cinder
password=SECRETE
project_name = services
user_domain_name = Default
project_domain_name = Default

You can restart services using the command openstack-service. To Restart Nova, run:

sudo openstack-service restart nova

And comparable commands for Cinder and Glance. I tested the endpoint using the Horizon API. for Glance, use the images page, and for cinder, the volume page. All other pages were Nova controlled. Neutron would obviously be the Network administration. If you get errors on the page saying “cannot access” it is a sign that they are wstill attempting to do V2 API token verification. Looking in the Keystone access log verified that for me. If you see lines like:

10.10.10.40 - - [09/May/2015:03:20:23 +0000] "GET /v2.0 HTTP/1.1" 404 93 "-" "python-keystoneclient"

You know something is trying to use the V2 API.

5 thoughts on “Setting up an RDO deployment to be Identity V3 Only”

  1. Hi Adam,

    I am wondering how did you manage to run Nova using V3 Keystone API.
    I am using Kilo, I followed your article and applied the setting in [keystone_authtoken] sections in both /etc/nova/nova.conf and /etc/neutron/neutron.conf, restarted Nova and Neutron services and I still get this in the nova.log:

    2015-12-03 17:45:05.815 17457 ERROR nova.compute.manager [req-bc22b21c-cf6a-4dbd-97f2-772301d65a80 – – – – -] [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] An error occurred while refreshing the network cache.
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] Traceback (most recent call last):
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] File “/usr/lib/python2.7/site-packages/nova/compute/manager.py”, line 5596, in _heal_instance_info_cache
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] self._get_instance_nw_info(context, instance)
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] File “/usr/lib/python2.7/site-packages/nova/compute/manager.py”, line 1360, in _get_instance_nw_info
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] return self.network_api.get_instance_nw_info(context, instance)
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] File “/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py”, line 747, in get_instance_nw_info
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] preexisting_port_ids)
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] File “/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py”, line 763, in _get_instance_nw_info
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] preexisting_port_ids)
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] File “/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py”, line 1515, in _build_network_info_model
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] client = get_client(context, admin=True)
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] File “/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py”, line 205, in get_client
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] auth_token = _ADMIN_AUTH.get_token(_SESSION)
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] File “/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py”, line 104, in get_token
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] return self.get_access(session).auth_token
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] File “/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py”, line 144, in get_access
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] self.auth_ref = self.get_auth_ref(session)
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] File “/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py”, line 78, in get_auth_ref
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] authenticated=False, log=False)
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] File “/usr/lib/python2.7/site-packages/keystoneclient/session.py”, line 497, in post
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] return self.request(url, ‘POST’, **kwargs)
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] File “/usr/lib/python2.7/site-packages/keystoneclient/utils.py”, line 318, in inner
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] return func(*args, **kwargs)
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] File “/usr/lib/python2.7/site-packages/keystoneclient/session.py”, line 397, in request
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] raise exceptions.from_response(resp, method, url)
    2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] NotFound: The resource could not be found. (HTTP 404) (Request-ID: req-9f014d4a-1fcb-4c5b-98bd-1ed6b16f2099)

    This happens when I browse the /project/ url of Horizon…

    One difference that I find in my setup is that keystone endpoint is SSL enabled so I configured all keystone URL/endpoints with “https” but looking at the error above I do not think this is the reason for novaclient not using v3 Keystone API.

    I have reviewed the novaclient code and it seems to have v1 and v2 hardcoded as options but no v3 in there. How is your novaclient working with V3 at your side?
    Isn’t it that the novaclient code is not able to work with V3?

    Thanks in advance!

  2. NotFound: The resource could not be found. (HTTP 404) (Request-ID: req-9f014d4a-1fcb-4c5b-98bd-1ed6b16f2099) Not sure which request that is, to get the token or to list the projects, but seems like a misconfiguration of the Horizon setup. Doesn’t look like HTTPS, but you sure you have the same protocol for the url in the Horizon config as the Keystone server uses?

  3. Hi Adam,

    Sorry I had to paste more of my log. Here is what I have in the log and why I concluded that it is Nova not recognizing the API version:

    DEBUG keystoneclient.session [req-7e1d69c5-d80b-4417-b9e8-7da00bdc600f – – – – -] Request returned failure status: 404 for url https://:5000/v3/tokens request /usr/lib/python2.7/site-packages/keystoneclient/session.py:396

    It is looking for /v3/tokens but it should be looking for /v3/auth/tokens shouldn’t it?

    My Horizon setting for Keystone is:
    OPENSTACK_KEYSTONE_URL = “https://%s:5000/v3” % OPENSTACK_HOST

    What else could be the problem?

  5. V1 and V2 Are Nova APIs, not Keystone, I think. Looks like your horizon is not properly configured. Please ask on opentack-dev and other people can see the issue as well.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.