Firecracker Limitation: volume support

[mcastelino]

Description of problem

Firecracker today does not have support for host to guest file sharing. Kata achieves minimal file sharing by copying files from host to guest.

Critical kubernetes features like tokens, secrets, configmaps are typically volumes mapping to host side directories with files (and symlinks).

We need to find a method to handle common kubernetes volumes mounts using copy.
This may help us handle short lived PODs that may not need to handle updates (like FaaS usages).

[mcastelino]

/cc @egernst @raravena80 @sboeuf

[mcastelino]

For example the service account token shows up inside the pod as below

config.json snippet

               {
                        "destination": "/var/run/secrets/kubernetes.io/serviceaccount",
                        "type": "bind",
                        "source": "/var/lib/kubelet/pods/6eacc701-054b-11e9-944a-5254003d4a1b/volumes/kubernetes.io~secret/default-token-2ncpz",
                        "options": [
                                "ro",
                                "rbind",
                                "rprivate",
                                "bind"
                        ]
                }

Within the POD it is typically visible as

root@ubuntu-pod-runc:/run/secrets/kubernetes.io/serviceaccount# ls -alp
total 4
drwxrwxrwt 3 root root  140 Dec 21 18:08 ./
drwxr-xr-x 3 root root 4096 Dec 21 18:08 ../
drwxr-xr-x 2 root root  100 Dec 21 18:08 ..2018_12_21_18_08_35.690127622/
lrwxrwxrwx 1 root root   31 Dec 21 18:08 ..data -> ..2018_12_21_18_08_35.690127622/
lrwxrwxrwx 1 root root   13 Dec 21 18:08 ca.crt -> ..data/ca.crt
lrwxrwxrwx 1 root root   16 Dec 21 18:08 namespace -> ..data/namespace
lrwxrwxrwx 1 root root   12 Dec 21 18:08 token -> ..data/token

And the pod spec is

   "spec": {
        "containers": [
            {
                "command": [
                    "top"
                ],
                "image": "ubuntu:14.04",
                "imagePullPolicy": "IfNotPresent",
                "name": "ubuntu",
                "resources": {},
                "stdin": true,
                "terminationMessagePath": "/dev/termination-log",
                "terminationMessagePolicy": "File",
                "tty": true,
                "volumeMounts": [
                    {
                        "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
                        "name": "default-token-2ncpz",
                        "readOnly": true
                    }
                ]
            }
        ],
        "serviceAccount": "default",
        "serviceAccountName": "default",
        "volumes": [
            {
                "name": "default-token-2ncpz",
                "secret": {
                    "defaultMode": 420,
                    "secretName": "default-token-2ncpz"
                }
            }
        ]
    },

The host side directory corresponding to this is

tree -a ./var/lib/kubelet/pods/6eacc701-054b-11e9-944a-5254003d4a1b/volumes/
./var/lib/kubelet/pods/6eacc701-054b-11e9-944a-5254003d4a1b/volumes/
└── kubernetes.io~secret
    └── default-token-2ncpz
        ├── ..2018_12_21_18_08_35.690127622
        │   ├── ca.crt
        │   ├── namespace
        │   └── token
        ├── ca.crt -> ..data/ca.crt
        ├── ..data -> ..2018_12_21_18_08_35.690127622
        ├── namespace -> ..data/namespace
        └── token -> ..data/token

4 directories, 6 files

[mcastelino]

We will not be able to support termination pod binding. An example from a container config.json

               {
                        "destination": "/dev/termination-log",
                        "type": "bind",
                        "source": "/var/lib/kubelet/pods/6eacc701-054b-11e9-944a-5254003d4a1b/containers/ubuntu/b0d9ecbb",
                        "options": [
                                "rw",
                                "rbind",
                                "rprivate",
                                "bind"
                        ]
                },

[raravena80]

Another point of concern I believe, is how to handle secrets/certs securely. host -> VM

[egernst]

/cc @nmeyerhans

[famz]

Have you considered NFS for host to guest file sharing? It's possible to create a host-guest network with virtio-net, or use vsock. Firecracker supports vsock and work has been done on nfs-vsock by @stefanha.

[stefanha]

Have you considered NFS for host to guest file sharing? It's possible to create a host-guest network with virtio-net, or use vsock. Firecracker supports vsock and work has been done on nfs-vsock by @stefanha.

NFS over AF_VSOCK is currently not actively developed because I'm focussing on virtio-fs (https://virtio-fs.gitlab.io/) instead.

If the main requirement for host<->guest file sharing is for small files dynamically generated by the host, then it might be easiest to use kata-agent to send them into the sandbox VM. Perhaps kata-runtime would use inotify on the host to monitor changes (e.g. /etc/hosts).

[jcrowthe]

Is there an update on this ticket, or can it be re-prioritized?

Since the majority of potential container-based workloads require dynamic configuration, secrets, and/or serviceaccount tokens, I believe this limitation is a rather large blocker for anyone to use kata+firecracker outside of sandbox environments. Am I off on this?

[grahamwhaley]

@jcrowthe - I don't believe you are off on your assessment, but, I also believe this is generically a firecracker 'issue', not a Kata issue. It is a feature that is not provided by firecracker, with no easy workaround for Kata to use in its place. Something to also note/discuss to the firecracker folks then I guess?
/cc @egernst @mcastelino

[jpmartin2]

It looks like support for something along these lines is on the firecracker roadmap at least firecracker-microvm/firecracker#1180