Fuse filesystem mount in Kubernetes

The Use Case

I have been thinking about this problem for a while now. I have an NGinx container/pod, that use LetsEncrypt for automatic SSL certificate generation (See my container over on Github). In short, when the container starts - it checks the validity of the SSL certificates in /etc/letsencrypt, and if required, runs simp_le to generate/update them.

This works fine, when you have a single pod, but as soon as you scale up to multiple pods you end up with a situation where each pod is looking at its local storage, so in turn will each generate its own certificate (for the same domain). This brings in the need for shared storage, and more specifically ReadWriteMany shared storage, as each container needs to be able to write to the mount, as any of them could end up doing the update.

ReadWriteMany

Interestingly in Kubernetes the following ReadWriteMany volume mounts are available: AzureFile, CepthFS, Glusterfs, Quobyte, NFS.

Barring AzureFile, all of the others would have required me to run another container exposing that service, introducing more complexity and another single point of failure. And I'm not going to use AzureFile, as I'm hosting on Google Cloud Platform, using Google Container Engine.

We iterated over several ideas such as having a preboot script which pushed and pulled tar.gz files from various storage locations (like zookeeper, or kubernetes secrets), but it all just felt a bit hacky. And then I thought about fuse mounts.

Fuse

I would love for Kubernetes to add support for PersistentVolume fuse mounts, and there is a GitHub issue for it here - which hasn't seen much action at all, so herein lies a nice little workaround where we use the kubernetes lifecycle events postStart and preStop to do the mount and unmount for us.

Now I'm using GCP, but this method should work for mounting any type of fuse storage. Google provide a fuse adapter for Google Cloud Storage, called gcsfuse, which allows you to mount a GCS bucket as if it was a system drive.

Google Cloud Storage

Go and create a bucket in GCS, take note of the name.

Dockerfile

So lets start with making sure we have the gcsfuse binary installed in our container.

First create a gcsfuse.repo file:

[gcsfuse]
name=gcsfuse (packages.cloud.google.com)
baseurl=https://packages.cloud.google.com/yum/repos/gcsfuse-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0

And then in your Dockerfile:

COPY gcsfuse.repo /etc/yum.repos.d/
RUN dnf -y install gcsfuse
RUN mkdir -p /etc/letsencrypt

Kubernetes pod.yml

In order to perform the mount command on kubernetes, we need to run the pod as --privileged, and add the capability SYS_ADMIN.

spec:
  ...
  template:
  ...
    spec:
    ... 
      containers:
      - name: my-container
        securityContext:
          privileged: true
          capabilities:
            add:
              - SYS_ADMIN
        lifecycle:
          postStart:
            exec:
              command: ["gcsfuse", "-o", "nonempty", "your-bucket-name", "/etc/letsencrypt"]
          preStop:
            exec:
              command: ["fusermount", "-u", "/etc/letsencrypt"]

Authentication

If you're running on GCP like me, you just need to ensure your GKE cluster is created with the OAuth scope https://www.googleapis.com/auth/devstorage.read_write, and everything else will be handled automatically.

If you're not using GCP/GKE then you will need to install the gcloud cli and authenticate as part of your container preboot.

Voila

And that's it, your GCS storage will be mounted in all instances of your pod, as ReadWriteMany, shared storage via fuse!

Caveats: writing is slow, this is not a solution for any sort of high write situation. Also, multiple pods mounting the same storage are eventually consistent.