Trying to troubleshoot the issues from Yesterday’s SELinux errors.
Immediately after a new deploy of the manifests, I want to look at the context on the qemu file:
$ kubectl get pods libvirt-81sdh NAME READY STATUS RESTARTS AGE libvirt-81sdh 2/2 Running 0 28s |
Now to look at the file:
$ kubectl exec libvirt-81sdh -c libvirtd -- ls -lZ /usr/local/bin/qemu-system-x86_64 -rwxrwxr-x. 1 root root system_u:object_r:unlabeled_t:s0 2814 Aug 10 00:48 /usr/local/bin/qemu-system-x86_64 |
Running restorecon on it, as the audit2allow output suggests:
[ayoung@drifloon kubevirt]$ kubectl exec libvirt-81sdh -c libvirtd -- restorecon /usr/local/bin/qemu-system-x86_64 [ayoung@drifloon kubevirt]$ kubectl exec libvirt-81sdh -c libvirtd -- ls -lZ /usr/local/bin/qemu-system-x86_64 -rwxrwxr-x. 1 root root system_u:object_r:bin_t:s0 2814 Aug 10 00:48 /usr/local/bin/qemu-system-x86_64 |
unlabeled_t became bin_t.
Once again, attempt to deploy a vm, and see what AVCs we get:
$ kubectl apply -f cluster/vm-pxe.yaml vm "testvm" created [ayoung@drifloon kubevirt]$ kubectl delete -f cluster/vm-pxe.yaml vm "testvm" deleted |
This is what the audit log showed:
type=AVC msg=audit(1504291091.397:2933): avc: denied { transition } for pid=32273 comm="libvirtd" path="/usr/local/bin/qemu-system-x86_64" dev="dm-18" ino=31526884 scontext=system_u:system_r:spc_t:s0 tcontext=system_u:system_r:svirt_tcg_t:s0:c322,c373 tclass=process |
There were several lines like that, but they were identical except for the pid. What does audit2allow show?
#============= spc_t ============== #!!!! The file '/usr/local/bin/qemu-system-x86_64' is mislabeled on your system. #!!!! Fix with $ restorecon -R -v /usr/local/bin/qemu-system-x86_64 allow spc_t svirt_tcg_t:process transition; |
Lets see if the additional parameters make a difference:
$ kubectl exec libvirt-81sdh -c libvirtd -- restorecon -R -v /usr/local/bin/qemu-system-x86_64 $ kubectl exec libvirt-81sdh -c libvirtd -- ls -lZ /usr/local/bin/qemu-system-x86_64 -rwxrwxr-x. 1 root root system_u:object_r:bin_t:s0 2814 Aug 10 00:48 /usr/local/bin/qemu-system-x86_64
The original lableeing of system_u:object_r:unlabeled_t:s0 is now system_u:object_r:bin_t:s0, which is the same as it was after maiong the restorecon call without the additional parameters.
How about the additional line, the allow? I can apply it outside of the container by using audit2allow:
cat /tmp/audit.txt | audit2allow -a -M virt-policy sudo semodule -i virt-policy.pp
Upon deploy, a similar error, with a different context:
type=AVC msg=audit(1504294173.446:3734): avc: denied { entrypoint } for pid=6565 comm="libvirtd" path="/usr/local/bin/qemu-system-x86_64" dev="dm-18" ino=31526884 scontext=system_u:system_r:svirt_tcg_t:s0:c577,c707 tcontext=system_u:object_r:bin_t:s0 tclass=file |
Running this through audit2allow generates
#============= svirt_tcg_t ============== #!!!! WARNING: 'bin_t' is a base type. allow svirt_tcg_t bin_t:file entrypoint; |
While this is a pretty powerful rule, it might be appropriate for what we are doing with virt. Again, lets apply the policy and see what happens.
$ cat virt-policy-2.txt | audit2allow -a -M virt-policy-2 $ sudo semodule -i virt-policy-2.pp |
Now a slew of errors, but different ones:
type=AVC msg=audit(1504294406.893:3797): avc: denied { write } for pid=7236 comm="qemu-system-x86" path="pipe:[423417]" dev="pipefs" ino=423417 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:system_r:spc_t:s0 tclass=fifo_file type=AVC msg=audit(1504294406.893:3797): avc: denied { write } for pid=7236 comm="qemu-system-x86" path="pipe:[423417]" dev="pipefs" ino=423417 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:system_r:spc_t:s0 tclass=fifo_file type=AVC msg=audit(1504294406.894:3798): avc: denied { read } for pid=7236 comm="qemu-system-x86" name="ld.so.cache" dev="dm-18" ino=8388771 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file type=AVC msg=audit(1504294406.894:3799): avc: denied { read } for pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file type=AVC msg=audit(1504294406.894:3800): avc: denied { read } for pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file type=AVC msg=audit(1504294406.894:3801): avc: denied { read } for pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file type=AVC msg=audit(1504294406.894:3802): avc: denied { read } for pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file type=AVC msg=audit(1504294406.894:3803): avc: denied { read } for pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file type=AVC msg=audit(1504294406.894:3804): avc: denied { read } for pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file type=AVC msg=audit(1504294406.894:3805): avc: denied { read } for pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file type=AVC msg=audit(1504294406.894:3806): avc: denied { read } for pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file type=AVC msg=audit(1504294406.895:3807): avc: denied { read } for pid=7236 comm="qemu-system-x86" name="libtinfo.so.6" dev="dm-18" ino=29360804 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file type=AVC msg=audit(1504294406.900:3808): avc: denied { sigchld } for pid=21975 comm="docker-containe" scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:system_r:container_runtime_t:s0 tclass=process |
This process is iterative, and I had to go through it 10 times until I came up with a complete set of audit2allow generated files. Here is the sum total of what was generated.
module virt-policy-2 1.0; require { type svirt_tcg_t; type bin_t; class file entrypoint; } #============= svirt_tcg_t ============== #!!!! WARNING: 'bin_t' is a base type. allow svirt_tcg_t bin_t:file entrypoint; module virt-policy-3 1.0; require { type unlabeled_t; type svirt_tcg_t; type spc_t; type container_runtime_t; class process sigchld; class lnk_file read; class fifo_file write; class file read; } #============= svirt_tcg_t ============== allow svirt_tcg_t container_runtime_t:process sigchld; allow svirt_tcg_t spc_t:fifo_file write; #!!!! WARNING: 'unlabeled_t' is a base type. allow svirt_tcg_t unlabeled_t:file read; allow svirt_tcg_t unlabeled_t:lnk_file read; module virt-policy-4 1.0; require { type unlabeled_t; type svirt_tcg_t; class file open; } #============= svirt_tcg_t ============== #!!!! WARNING: 'unlabeled_t' is a base type. #!!!! The file '/etc/ld.so.cache' is mislabeled on your system. #!!!! Fix with $ restorecon -R -v /etc/ld.so.cache allow svirt_tcg_t unlabeled_t:file open; module virt-policy-5 1.0; require { type unlabeled_t; type svirt_tcg_t; class file getattr; } #============= svirt_tcg_t ============== #!!!! WARNING: 'unlabeled_t' is a base type. #!!!! The file '/etc/ld.so.cache' is mislabeled on your system. #!!!! Fix with $ restorecon -R -v /etc/ld.so.cache allow svirt_tcg_t unlabeled_t:file getattr; module virt-policy-6 1.0; require { type unlabeled_t; type svirt_tcg_t; class file execute; } #============= svirt_tcg_t ============== #!!!! WARNING: 'unlabeled_t' is a base type. #!!!! The file '/usr/lib64/libtinfo.so.6.0' is mislabeled on your system. #!!!! Fix with $ restorecon -R -v /usr/lib64/libtinfo.so.6.0 allow svirt_tcg_t unlabeled_t:file execute; module virt-policy-7 1.0; require { type unlabeled_t; type svirt_tcg_t; type spc_t; class fifo_file { getattr ioctl }; class file { execute_no_trans write }; } #============= svirt_tcg_t ============== allow svirt_tcg_t spc_t:fifo_file { getattr ioctl }; #!!!! WARNING: 'unlabeled_t' is a base type. allow svirt_tcg_t unlabeled_t:file { execute_no_trans write }; module virt-policy-8 1.0; require { type unlabeled_t; type svirt_tcg_t; type sysfs_t; class capability { setgid setuid }; class file append; class filesystem getattr; } #============= svirt_tcg_t ============== allow svirt_tcg_t self:capability { setgid setuid }; allow svirt_tcg_t sysfs_t:filesystem getattr; #!!!! WARNING: 'unlabeled_t' is a base type. allow svirt_tcg_t unlabeled_t:file append; module virt-policy-9 1.0; require { type unlabeled_t; type svirt_tcg_t; class file ioctl; class dir read; } #============= svirt_tcg_t ============== allow svirt_tcg_t unlabeled_t:dir read; #!!!! WARNING: 'unlabeled_t' is a base type. #!!!! The file '/etc/sudoers' is mislabeled on your system. #!!!! Fix with $ restorecon -R -v /etc/sudoers allow svirt_tcg_t unlabeled_t:file ioctl; module virt-policy 1.0; require { type svirt_tcg_t; type spc_t; class process transition; } #============= spc_t ============== #!!!! The file '/usr/local/bin/qemu-system-x86_64' is mislabeled on your system. #!!!! Fix with $ restorecon -R -v /usr/local/bin/qemu-system-x86_64 allow spc_t svirt_tcg_t:process transition; module virt-policy-10 1.0; require { type unlabeled_t; type svirt_tcg_t; class capability { net_admin sys_resource }; class file lock; class netlink_audit_socket create; } #============= svirt_tcg_t ============== allow svirt_tcg_t self:capability { net_admin sys_resource }; allow svirt_tcg_t self:netlink_audit_socket create; #!!!! WARNING: 'unlabeled_t' is a base type. #!!!! The file '/run/utmp' is mislabeled on your system. #!!!! Fix with $ restorecon -R -v /run/utmp allow svirt_tcg_t unlabeled_t:file lock; |
Obviously, using permissive would have been a shorter process. Let me restart the VM and try that. Here’s what I generate after one iteration:
module kubevirt-policy 1.0; require { type unlabeled_t; type svirt_tcg_t; type container_runtime_t; class capability audit_write; class unix_stream_socket connectto; class file entrypoint; class netlink_audit_socket nlmsg_relay; } #============= svirt_tcg_t ============== #!!!! The file '/run/docker.sock' is mislabeled on your system. #!!!! Fix with $ restorecon -R -v /run/docker.sock allow svirt_tcg_t container_runtime_t:unix_stream_socket connectto; allow svirt_tcg_t self:capability audit_write; allow svirt_tcg_t self:netlink_audit_socket nlmsg_relay; #!!!! WARNING: 'unlabeled_t' is a base type. #!!!! The file '/usr/local/bin/qemu-system-x86_64' is mislabeled on your system. #!!!! Fix with $ restorecon -R -v /usr/local/bin/qemu-system-x86_64 allow svirt_tcg_t unlabeled_t:file entrypoint; |
And…. we start into the same pattern. It takes several runs to get to a set of policies that run cleanly. It seems some of the earlier AVCs mask later ones, and running in permissive mode only reports the first of several. Needless to say, the policy for running a VM Via Kubevirt is going to require some scrutiny.
And even then, the VMs still fail to deploy. Disable SELinux and they run. This mystery continues.