SE Linux for CentOS Continued

Trying to troubleshoot the issues from Yesterday’s SELinux errors.

Immediately after a new deploy of the manifests, I want to look at the context on the qemu file:

$ kubectl get pods libvirt-81sdh 
NAME            READY     STATUS    RESTARTS   AGE
libvirt-81sdh   2/2       Running   0          28s

Now to look at the file:

$ kubectl  exec libvirt-81sdh -c libvirtd -- ls  -lZ  /usr/local/bin/qemu-system-x86_64
-rwxrwxr-x. 1 root root system_u:object_r:unlabeled_t:s0 2814 Aug 10 00:48 /usr/local/bin/qemu-system-x86_64

Running restorecon on it, as the audit2allow output suggests:

[ayoung@drifloon kubevirt]$ kubectl  exec libvirt-81sdh -c libvirtd -- restorecon  /usr/local/bin/qemu-system-x86_64
[ayoung@drifloon kubevirt]$ kubectl  exec libvirt-81sdh -c libvirtd -- ls  -lZ  /usr/local/bin/qemu-system-x86_64
-rwxrwxr-x. 1 root root system_u:object_r:bin_t:s0 2814 Aug 10 00:48 /usr/local/bin/qemu-system-x86_64

unlabeled_t became bin_t.

Once again, attempt to deploy a vm, and see what AVCs we get:

$ kubectl apply -f cluster/vm-pxe.yaml 
vm "testvm" created
[ayoung@drifloon kubevirt]$ kubectl delete  -f cluster/vm-pxe.yaml 
vm "testvm" deleted

This is what the audit log showed:

type=AVC msg=audit(1504291091.397:2933): avc:  denied  { transition } for  pid=32273 comm="libvirtd" path="/usr/local/bin/qemu-system-x86_64" dev="dm-18" ino=31526884 scontext=system_u:system_r:spc_t:s0 tcontext=system_u:system_r:svirt_tcg_t:s0:c322,c373 tclass=process

There were several lines like that, but they were identical except for the pid. What does audit2allow show?

#============= spc_t ==============
 
#!!!! The file '/usr/local/bin/qemu-system-x86_64' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /usr/local/bin/qemu-system-x86_64
allow spc_t svirt_tcg_t:process transition;

Lets see if the additional parameters make a difference:

$ kubectl  exec libvirt-81sdh -c libvirtd -- restorecon  -R -v  /usr/local/bin/qemu-system-x86_64
$ kubectl  exec libvirt-81sdh -c libvirtd -- ls  -lZ  /usr/local/bin/qemu-system-x86_64
-rwxrwxr-x. 1 root root system_u:object_r:bin_t:s0 2814 Aug 10 00:48 /usr/local/bin/qemu-system-x86_64

The original lableeing of system_u:object_r:unlabeled_t:s0 is now system_u:object_r:bin_t:s0, which is the same as it was after maiong the restorecon call without the additional parameters.

How about the additional line, the allow? I can apply it outside of the container by using audit2allow:

cat /tmp/audit.txt | audit2allow -a -M virt-policy
sudo semodule -i virt-policy.pp

Upon deploy, a similar error, with a different context:

type=AVC msg=audit(1504294173.446:3734): avc:  denied  { entrypoint } for  pid=6565 comm="libvirtd" path="/usr/local/bin/qemu-system-x86_64" dev="dm-18" ino=31526884 scontext=system_u:system_r:svirt_tcg_t:s0:c577,c707 tcontext=system_u:object_r:bin_t:s0 tclass=file

Running this through audit2allow generates

#============= svirt_tcg_t ==============
 
#!!!! WARNING: 'bin_t' is a base type.
allow svirt_tcg_t bin_t:file entrypoint;

While this is a pretty powerful rule, it might be appropriate for what we are doing with virt. Again, lets apply the policy and see what happens.

$ cat virt-policy-2.txt | audit2allow -a -M virt-policy-2
$ sudo semodule -i virt-policy-2.pp

Now a slew of errors, but different ones:

type=AVC msg=audit(1504294406.893:3797): avc:  denied  { write } for  pid=7236 comm="qemu-system-x86" path="pipe:[423417]" dev="pipefs" ino=423417 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:system_r:spc_t:s0 tclass=fifo_file
type=AVC msg=audit(1504294406.893:3797): avc:  denied  { write } for  pid=7236 comm="qemu-system-x86" path="pipe:[423417]" dev="pipefs" ino=423417 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:system_r:spc_t:s0 tclass=fifo_file
type=AVC msg=audit(1504294406.894:3798): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="ld.so.cache" dev="dm-18" ino=8388771 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
type=AVC msg=audit(1504294406.894:3799): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3800): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3801): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3802): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3803): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3804): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3805): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.894:3806): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="lib64" dev="dm-18" ino=143 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.895:3807): avc:  denied  { read } for  pid=7236 comm="qemu-system-x86" name="libtinfo.so.6" dev="dm-18" ino=29360804 scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:object_r:unlabeled_t:s0 tclass=lnk_file
type=AVC msg=audit(1504294406.900:3808): avc:  denied  { sigchld } for  pid=21975 comm="docker-containe" scontext=system_u:system_r:svirt_tcg_t:s0:c550,c926 tcontext=system_u:system_r:container_runtime_t:s0 tclass=process

This process is iterative, and I had to go through it 10 times until I came up with a complete set of audit2allow generated files. Here is the sum total of what was generated.

 
module virt-policy-2 1.0;
 
require {
	type svirt_tcg_t;
	type bin_t;
	class file entrypoint;
}
 
#============= svirt_tcg_t ==============
 
#!!!! WARNING: 'bin_t' is a base type.
allow svirt_tcg_t bin_t:file entrypoint;
 
module virt-policy-3 1.0;
 
require {
	type unlabeled_t;
	type svirt_tcg_t;
	type spc_t;
	type container_runtime_t;
	class process sigchld;
	class lnk_file read;
	class fifo_file write;
	class file read;
}
 
#============= svirt_tcg_t ==============
allow svirt_tcg_t container_runtime_t:process sigchld;
allow svirt_tcg_t spc_t:fifo_file write;
 
#!!!! WARNING: 'unlabeled_t' is a base type.
allow svirt_tcg_t unlabeled_t:file read;
allow svirt_tcg_t unlabeled_t:lnk_file read;
 
module virt-policy-4 1.0;
 
require {
	type unlabeled_t;
	type svirt_tcg_t;
	class file open;
}
 
#============= svirt_tcg_t ==============
 
#!!!! WARNING: 'unlabeled_t' is a base type.
#!!!! The file '/etc/ld.so.cache' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /etc/ld.so.cache
allow svirt_tcg_t unlabeled_t:file open;
 
module virt-policy-5 1.0;
 
require {
	type unlabeled_t;
	type svirt_tcg_t;
	class file getattr;
}
 
#============= svirt_tcg_t ==============
 
#!!!! WARNING: 'unlabeled_t' is a base type.
#!!!! The file '/etc/ld.so.cache' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /etc/ld.so.cache
allow svirt_tcg_t unlabeled_t:file getattr;
 
module virt-policy-6 1.0;
 
require {
	type unlabeled_t;
	type svirt_tcg_t;
	class file execute;
}
 
#============= svirt_tcg_t ==============
 
#!!!! WARNING: 'unlabeled_t' is a base type.
#!!!! The file '/usr/lib64/libtinfo.so.6.0' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /usr/lib64/libtinfo.so.6.0
allow svirt_tcg_t unlabeled_t:file execute;
 
module virt-policy-7 1.0;
 
require {
	type unlabeled_t;
	type svirt_tcg_t;
	type spc_t;
	class fifo_file { getattr ioctl };
	class file { execute_no_trans write };
}
 
#============= svirt_tcg_t ==============
allow svirt_tcg_t spc_t:fifo_file { getattr ioctl };
 
#!!!! WARNING: 'unlabeled_t' is a base type.
allow svirt_tcg_t unlabeled_t:file { execute_no_trans write };
 
module virt-policy-8 1.0;
 
require {
	type unlabeled_t;
	type svirt_tcg_t;
	type sysfs_t;
	class capability { setgid setuid };
	class file append;
	class filesystem getattr;
}
 
#============= svirt_tcg_t ==============
allow svirt_tcg_t self:capability { setgid setuid };
allow svirt_tcg_t sysfs_t:filesystem getattr;
 
#!!!! WARNING: 'unlabeled_t' is a base type.
allow svirt_tcg_t unlabeled_t:file append;
 
module virt-policy-9 1.0;
 
require {
	type unlabeled_t;
	type svirt_tcg_t;
	class file ioctl;
	class dir read;
}
 
#============= svirt_tcg_t ==============
allow svirt_tcg_t unlabeled_t:dir read;
 
#!!!! WARNING: 'unlabeled_t' is a base type.
#!!!! The file '/etc/sudoers' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /etc/sudoers
allow svirt_tcg_t unlabeled_t:file ioctl;
 
module virt-policy 1.0;
 
require {
	type svirt_tcg_t;
	type spc_t;
	class process transition;
}
 
#============= spc_t ==============
 
#!!!! The file '/usr/local/bin/qemu-system-x86_64' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /usr/local/bin/qemu-system-x86_64
allow spc_t svirt_tcg_t:process transition;
 
 
module virt-policy-10 1.0;
 
require {
	type unlabeled_t;
	type svirt_tcg_t;
	class capability { net_admin sys_resource };
	class file lock;
	class netlink_audit_socket create;
}
 
#============= svirt_tcg_t ==============
allow svirt_tcg_t self:capability { net_admin sys_resource };
allow svirt_tcg_t self:netlink_audit_socket create;
 
#!!!! WARNING: 'unlabeled_t' is a base type.
#!!!! The file '/run/utmp' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /run/utmp
allow svirt_tcg_t unlabeled_t:file lock;

Obviously, using permissive would have been a shorter process. Let me restart the VM and try that. Here’s what I generate after one iteration:

module kubevirt-policy 1.0;
 
require {
	type unlabeled_t;
	type svirt_tcg_t;
	type container_runtime_t;
	class capability audit_write;
	class unix_stream_socket connectto;
	class file entrypoint;
	class netlink_audit_socket nlmsg_relay;
}
 
#============= svirt_tcg_t ==============
 
#!!!! The file '/run/docker.sock' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /run/docker.sock
allow svirt_tcg_t container_runtime_t:unix_stream_socket connectto;
allow svirt_tcg_t self:capability audit_write;
allow svirt_tcg_t self:netlink_audit_socket nlmsg_relay;
 
#!!!! WARNING: 'unlabeled_t' is a base type.
#!!!! The file '/usr/local/bin/qemu-system-x86_64' is mislabeled on your system.  
#!!!! Fix with $ restorecon -R -v /usr/local/bin/qemu-system-x86_64
allow svirt_tcg_t unlabeled_t:file entrypoint;

And…. we start into the same pattern. It takes several runs to get to a set of policies that run cleanly. It seems some of the earlier AVCs mask later ones, and running in permissive mode only reports the first of several. Needless to say, the policy for running a VM Via Kubevirt is going to require some scrutiny.

And even then, the VMs still fail to deploy. Disable SELinux and they run. This mystery continues.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.