Running OpenShift Origin built from source

Kubernetes is moving from Third Party Resources to the Aggregated API Server.  In order to work with this and continue to deploy on OpenShift Origin, we need to move from working with the shipped and stable version that is in Fedora 26 to the development version in git.  Here are my notes to get it up and running.


It took a couple tries to realize that the go build process needs a fairly bulky virtual machine. I ended up using one that has 8 GB Ram and 50 GB disk. In order to minimize churn, I also went with a Centos7 deployment.

Once I had a running VM here are the configuration changes I had to make.

Use nmcli c eth0 on and set ONBOOT. This can also be done via editing the network config files or using older tools.

yum update -y
yum groupinstall "Development Tools"
yum instyall -y origin-clients

Ensure that the Docker daemon is running with the following argument:
–insecure-registry  By editing the file /etc/sysconfig/docker.  My Options line looks like this:

OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false --insecure-registry'

Followed dirs from here In order to set up the development environment.

cd $GOPATH/src/
hack/env hack/
OS_BUILD_ENV_PRESERVE=_output/local/releases hack/env make release

Note that the KubeVirt code I want to run on top of this requires a later version of go, and thus I upgrade to go version go1.8.3 linux/amd64  via the tarball install method.

The Hash that gets generated by the build depends on when you run.   To see the images run

docker images

I expand the terminal to full screen lots of columns of data.  Here is a subset:

openshift/hello-openshift 413eb73 911092241b5a 35 hours ago 5.84 MB
openshift/hello-openshift latest 911092241b5a 35 hours ago 5.84 MB
openshift/openvswitch 413eb73 c53aae019d81 35 hours ago 1.241 GB
openshift/openvswitch latest c53aae019d81 35 hours ago 1.241 GB
openshift/node 413eb73 af6135fc50c9 35 hours ago 1.239 GB
openshift/node latest af6135fc50c9 35 hours ago 1.239 GB

The tag is the second column.  This is what I use in order to install. I don’t use “Latest” as that changes over time, and it might accidentally succeed using a remote package when the local build failed.

I want to be able to edit configuration values.  A also want the etcd store to persist across reboots.  Thus,

sudo mkdir /var/lib/origin/etcd
sudo chown ayoung:ayoung /var/lib/origin/etcd

And then my final command line to bring up the cluster is:

oc cluster up --use-existing-config --loglevel=5 --version=413eb73 --host-data-dir=/var/lib/origin/etcd/ | tee /tmp/oc.log 2>&1


Below are some of my Troubleshooting notes. I am going to leave them in here so they show up on future searches for people that have the same problems.  They are rough, and you don’t need to read them.

hack/env make release errored

[WARNING] Copying _output/local/releases from the container failed!
[WARNING] Error response from daemon: lstat /var/lib/docker/devicemapper/mnt/fb199307b2f95649066c42f55e5487c66eb3421e5407c8bd6d2f0a7058bc8cd5/rootfs/go/src/ no such file or directory

Tried with OS_BUILD_ENV_PRESERVE=_output/local but no difference.

Should have been
OS_BUILD_ENV_PRESERVE=_output/local/releases hack/env make release

This did not work (basename error)
export PATH=”${PATH}:$( source hack/lib/; echo “${OS_OUTPUT_BINPATH}/$( os::util::host_platform )/” )”

But I was able to do
export PATH=$PATH:$PWD/_output/local/bin/linux/amd64/

and then

oc cluster up –version=latest

failed due to docker error
— Checking Docker daemon configuration … FAIL
Error: did not detect an –insecure-registry argument on the Docker daemon


To fix

oc seems to be running OK now. But not using my git commit

Told to run:


oc cluster up –version=8d96d48

Leave a Reply

Your email address will not be published. Required fields are marked *