The Use Case
I have been thinking about this problem for a while now. I have an NGinx container/pod, that use LetsEncrypt for automatic SSL certificate generation (See my container over on Github). In short, when the container starts - it checks the validity of the SSL certificates in
/etc/letsencrypt, and if required, runs simp_le to generate/update them.
This works fine, when you have a single pod, but as soon as you scale up to multiple pods you end up with a situation where each pod is looking at its local storage, so in turn will each generate its own certificate (for the same domain). This brings in the need for shared storage, and more specifically
ReadWriteMany shared storage, as each container needs to be able to write to the mount, as any of them could end up doing the update.
Interestingly in Kubernetes the following
ReadWriteMany volume mounts are available: AzureFile, CepthFS, Glusterfs, Quobyte, NFS.
Barring AzureFile, all of the others would have required me to run another container exposing that service, introducing more complexity and another single point of failure. And I'm not going to use AzureFile, as I'm hosting on Google Cloud Platform, using Google Container Engine.
We iterated over several ideas such as having a preboot script which pushed and pulled
tar.gz files from various storage locations (like zookeeper, or kubernetes secrets), but it all just felt a bit hacky. And then I thought about fuse mounts.
I would love for Kubernetes to add support for PersistentVolume fuse mounts, and there is a GitHub issue for it here - which hasn't seen much action at all, so herein lies a nice little workaround where we use the kubernetes lifecycle events
preStop to do the mount and unmount for us.
Now I'm using GCP, but this method should work for mounting any type of fuse storage. Google provide a fuse adapter for Google Cloud Storage, called gcsfuse, which allows you to mount a GCS bucket as if it was a system drive.
Google Cloud Storage
Go and create a bucket in GCS, take note of the name.
So lets start with making sure we have the
gcsfuse binary installed in our container.
First create a
[gcsfuse] name=gcsfuse (packages.cloud.google.com) baseurl=https://packages.cloud.google.com/yum/repos/gcsfuse-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0
And then in your Dockerfile:
COPY gcsfuse.repo /etc/yum.repos.d/ RUN dnf -y install gcsfuse RUN mkdir -p /etc/letsencrypt
In order to perform the mount command on kubernetes, we need to run the pod as
--privileged, and add the capability
spec: ... template: ... spec: ... containers: - name: my-container securityContext: privileged: true capabilities: add: - SYS_ADMIN lifecycle: postStart: exec: command: ["gcsfuse", "-o", "nonempty", "your-bucket-name", "/etc/letsencrypt"] preStop: exec: command: ["fusermount", "-u", "/etc/letsencrypt"]
If you're running on GCP like me, you just need to ensure your GKE cluster is created with the OAuth scope
https://www.googleapis.com/auth/devstorage.read_write, and everything else will be handled automatically.
If you're not using GCP/GKE then you will need to install the
gcloud cli and authenticate as part of your container preboot.
And that's it, your GCS storage will be mounted in all instances of your pod, as
ReadWriteMany, shared storage via fuse!
Caveats: writing is slow, this is not a solution for any sort of high write situation. Also, multiple pods mounting the same storage are eventually consistent.