Customizing the KubeVirt Manifests

My cloud may not look like your cloud. The contract between the application deployment and the Kubernetes installation is a set of manifest files that guide Kubernetes in selecting, naming, and exposing resources. In order to make the generation of the Manifests sane in KubeVirt, we’ve provided a little bit of build system support.

The manifest files are templatized in a jinja style. I say style, because the actual template string replacement is done using simmple bash scripting. Regardless of the mechanism, it should not be hard for a developer to understand what happens. I’ll assume that you have your source code checked out in $GOPATH/src/kubevirt.io/kubevirt/

The template files exist in the manifests subdirectory. Mine looks like this:

haproxy.yaml.in             squid.yaml.in            virt-manifest.yaml.in
iscsi-demo-target.yaml.in   virt-api.yaml.in         vm-resource.yaml.in
libvirt.yaml.in             virt-controller.yaml.in
migration-resource.yaml.in  virt-handler.yaml.in

The simplest way to generate a set of actual manifest files is to run make manifests

make manifests
./hack/build-manifests.sh
$ ls -l manifests/*yaml
-rw-rw-r--. 1 ayoung ayoung  672 Aug 21 10:17 manifests/haproxy.yaml
-rw-rw-r--. 1 ayoung ayoung 2384 Aug 21 10:17 manifests/iscsi-demo-target.yaml
-rw-rw-r--. 1 ayoung ayoung 1707 Aug 21 10:17 manifests/libvirt.yaml
-rw-rw-r--. 1 ayoung ayoung  256 Aug 21 10:17 manifests/migration-resource.yaml
-rw-rw-r--. 1 ayoung ayoung  709 Aug 21 10:17 manifests/squid.yaml
-rw-rw-r--. 1 ayoung ayoung  832 Aug 21 10:17 manifests/virt-api.yaml
-rw-rw-r--. 1 ayoung ayoung  987 Aug 21 10:17 manifests/virt-controller.yaml
-rw-rw-r--. 1 ayoung ayoung  954 Aug 21 10:17 manifests/virt-handler.yaml
-rw-rw-r--. 1 ayoung ayoung 1650 Aug 21 10:17 manifests/virt-manifest.yaml
-rw-rw-r--. 1 ayoung ayoung  228 Aug 21 10:17 manifests/vm-resource.yaml

Looking at the difference between, say the virt-api template and final yaml file:

$ diff -u manifests/virt-api.yaml.in manifests/virt-api.yaml
--- manifests/virt-api.yaml.in	2017-07-20 13:29:00.532916101 -0400
+++ manifests/virt-api.yaml	2017-08-21 10:17:10.533038861 -0400
@@ -7,7 +7,7 @@
     - port: 8183
       targetPort: virt-api
   externalIPs :
-    - "{{ master_ip }}"
+    - "192.168.200.2"
   selector:
     app: virt-api
 ---
@@ -23,14 +23,14 @@
     spec:
       containers:
       - name: virt-api
-        image: {{ docker_prefix }}/virt-api:{{ docker_tag }}
+        image: kubevirt/virt-api:latest
         imagePullPolicy: IfNotPresent
         command:
             - "/virt-api"
             - "--port"
             - "8183"
             - "--spice-proxy"
-            - "{{ master_ip }}:3128"
+            - "192.168.200.2:3128"
         ports:
           - containerPort: 8183
             name: "virt-api"
@@ -38,4 +38,4 @@
       securityContext:
         runAsNonRoot: true
       nodeSelector:
-        kubernetes.io/hostname: {{ primary_node_name }}
+        kubernetes.io/hostname: master

make manifests, it turns out, just calls a bash script ./hack/build-manifests.sh. This script uses two files to determine the values to use for template string substitution. First, the defaults: hack/config-default.sh. This is where master_ip get the value of 192.168.200.2. This file also gives priority to the $DOCKER_TAG environment variable. However, if you need to customize values further, you can create and manage them in the file hack/config-local.sh. The goal is that any of the keys from the -default file that are specified in the hack/config-local.sh will use the value from the latter file. The set of keys with their defaults (as of this writing) that you can customize are:

binaries="cmd/virt-controller cmd/virt-launcher cmd/virt-handler cmd/virt-api cmd/virtctl cmd/virt-manifest"
docker_images="cmd/virt-controller cmd/virt-launcher cmd/virt-handler cmd/virt-api cmd/virt-manifest images/haproxy images/iscsi-demo-target-tgtd images/vm-killer images/libvirt-kubevirt images/spice-proxy cmd/virt-migrator cmd/registry-disk-v1alpha images/cirros-registry-disk-demo"
optional_docker_images="cmd/registry-disk-v1alpha images/fedora-atomic-registry-disk-demo"
docker_prefix=kubevirt
docker_tag=${DOCKER_TAG:-latest}
manifest_templates="`ls manifests/*.in`"
master_ip=192.168.200.2
master_port=8184
network_provider=weave
primary_nic=${primary_nic:-eth1}
primary_node_name=${primary_node_name:-master}

Not all of these are for Manifest files. The docker_images key is used in selecting the set of generating Docker images to generate in a command called from a different section of the Makefile. The network_provider is used in the Vagrant setup, and so on.However, most of the values are used in the manifest files. So, If I want to set a master IP Address of 10.10.10.10, I would have a hack/config-local.sh file that looks like this:

master_ip=10.10.10.10
$  diff -u manifests/virt-api.yaml.in manifests/virt-api.yaml
--- manifests/virt-api.yaml.in	2017-07-20 13:29:00.532916101 -0400
+++ manifests/virt-api.yaml	2017-08-21 10:42:28.434742371 -0400
@@ -7,7 +7,7 @@
     - port: 8183
       targetPort: virt-api
   externalIPs :
-    - "{{ master_ip }}"
+    - "10.10.10.10"
   selector:
     app: virt-api
 ---
@@ -23,14 +23,14 @@
     spec:
       containers:
       - name: virt-api
-        image: {{ docker_prefix }}/virt-api:{{ docker_tag }}
+        image: kubevirt/virt-api:latest
         imagePullPolicy: IfNotPresent
         command:
             - "/virt-api"
             - "--port"
             - "8183"
             - "--spice-proxy"
-            - "{{ master_ip }}:3128"
+            - "10.10.10.10:3128"
         ports:
           - containerPort: 8183
             name: "virt-api"
@@ -38,4 +38,4 @@
       securityContext:
         runAsNonRoot: true
       nodeSelector:
-        kubernetes.io/hostname: {{ primary_node_name }}
+        kubernetes.io/hostname: master

Leave a Reply

Your email address will not be published. Required fields are marked *