While trying to launch kubevirt containers in OpenShift, I continually tripped over problems regarding authorization.
Last time I walked through the process of merging two sets of Kubernetest client configurations into one. For more ephemeral data, you might not want to munge it all into your main configuration. The KUBECONFIG environment variables lets you specify muiltiple configuration files and merge them into a single set of configuration data.
I have two distinct Kubernetes clusters I work with on a daily basis. One is a local vagrant bases set of VM built by the Kubevirt code base. The other is a “baremetal” install of OpenShift Origin on a pair of Fedora workstation in my office. I want to be able to switch back and forth between them.
OK, since I did it wrong last time, I’m going to try creating an user in OpenShift, and grant that user permissions to do various things.
The word Admin is used all over the place. To administer was originally something servants did to their masters. In one of the greater inversions of linguistic history, we now use Admin as a way to indicate authority. In OpenStack, the admin role is used for almost all operations that are reserved for someone with a higher level of authority. These actions are not expected to be performed by people with the plebean Member role.
While many people referred me to run one of the virtualized setups of OpenShift, I wanted something on baremetal in order to eventually test out KubeVirt. Just running
oc cluster up
As some people suggested did not work, as it assumes prerequisites are properly set up; the docker registry was one that I tripped over. So, I decided to give openshift-ansible a test run. Here are my notes.
A quota is a numerical limit on a group of resources. Quotas have to be both recorded and enforced.
We had a session at the summit this past week about hierarchical quotas and, if I took anything away from it, it is that quotas are hard.
Keystone supports a project hierarchy. Here’s a sample one for you:
Hierarchical quotas are assigned to a parent project and applied to a child project. This hierarchy is only 3 levels deep and only has 9 projects. A real deployment will be much larger than this. Often, a large organization has one project per user, in addition to departmental projects like the ones shown above.
Lets assume that our local sys-admin has granted our Internal domain a quota of 100 virtual machines. How would we enforce this. If the user attempts to create a VM in the root project of the hierarchy (a domain IS-A project) then Nova should see that the quota for that domain is 100, and that there are currently 0 VMs, so it should create the VM. The second time this happens, there is a remaining quota of 99, and so on.
Now, lets assume that the quota is stored in Keystone, as in the current proposal we were discussing. When Nova asks Keystone what is the quoat for “Internal” Keystone can return 100. Nova can then query all VMs to find out which have a project ID that matches that of “Internal” and verify that there are 2. Since 100 – 2 > 0, Nova should create the VM.
What if the user wants to create a VM in the “Sales” project? That is where things get hierarchical. We discussed schemes where the quota would be explicitly assigned to Sales and where the quota was assumed to come from “Internal.” Both are tricky.
Lets say we allow the explicit allocation of quota from higher to lower. Does this mean that the parent project is reducing its own quota while creating an explicit quota for the lower project? Or does it mean that both quotas need to be enforced? If the quota for sales is set to 10, and the quota for the three node projects are all set to 10, is this legal or an error?
Lets assume, for a moment, that it is legal. Under this scheme, a user with a token scoped to TestingA create 10 projects. As each project is created, Nova needs to check the number of machines already created in project TestingA. It also needs to check the number of machines in project StagingA, ProductionA, and Sales to ensure that the quota for “Sales” has not been exceeded. If the is an explicit quota on “Internal”, Nova needs to check the number of VMs created in that project and any project under it. Our entire tree must be searched and counted and that count compared with the parent project.
Ideally, we would only ever have to check the quota for a single project. That only works if:
- Every project in the whole tree has an explicit quota
- Quotas can be “split” amongst child projects but never reclaimed.
If that second statement seems strong, assume the “Marketing” project with a quota of 10 chips off 9 for TestingB, creates 5 VMs, drops the quota for TestingB to 0, Sets the quota for StagingB to 9, and creates 9 VMs in that project. This leaves it with 18 VMs running but only an explicit quota of 10.
The word “never” really is too strong, but it would require some form of reconcilliation process, by which Nova confirmed that both projects were within the end-state limits.
Automated Reconciliation is hard. Keystone needs to know how to query random quanties on remote objects, and it probably should not even have acceess to those objects. Or, Nova (and every other service using Quotas) needs to provide an API for keystone to query to confirm resources have been freed.
Manual reconcilliation is probably possible, but will be labor intensive.
One possibility is that Keystone actually record the usage of quotas, as well as the freeing of actual resources. This is also painful, as now every single call that either creates or deletes a resource requires an additional call to Keystone. Or, If quotas are “Batch” fetched by Nova, Nova needs to remember them, and store them locally. If quotas then change in Keystone, the cache is invalid.
This is only a fragment of the whole discussion.
Quotas are hard.