Persistent Letsencrypt Certificates for Traefik Ingress Controller

2024-06-13

Traefik as a Kubernetes Ingress Controller makes it easy to use Letsencrypt Certificates for TLS termination as e.g. outlined in this blog post. By default, certicate data is stored on a temporary volume, so certificates have to be recreated on every Traefik restart. This brings with it a significant risk to hit Letsencrypt's rate limits, resulting in a temporary ban that ultimately leaves you without a valid certificate for an unacceptable amount of time.

The obvious solution is to store the certificate data in a persistent volume claim. This is supported by the Traefik Helmchart with the initContainers facility and several proposals on how to do it are listed here.

However if you already have a bunch of certificates on a temporary volume you might just as well keep them, which is what this blog post is about.

Move existing certificates to a persistent volume claim

There is a couple of things to keep in mind in order to get it right:

• Certificate data is stored in a file named acme.json that resides in a directory named /data by default.
• Traefik expects only the user to have read access to acme.json i.e. 0600 (go-rw) file permissions.
• Traefik is by default using userid 65532.

So in order to be able to use a given persistent volume claim for certificate data, you will need to setup file permissions before starting Traefik. In order to minimize downtime we create and populate the PVC before reconfiguring Traefik:

• Download acme.json from the cluster using kubectl cp

kubectl cp kube-system/traefik-84c7f6c855-b9rqk:/data/acme.json acme.json

• Create a new PVC in kube-system namespace to hold the certificates

E.g. using longhorn:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: traefik
  namespace: kube-system
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 128Mi
  storageClassName: longhorn
  volumeMode: Filesystem
  volumeName: le-certificates

• Create a busybox deployment and bind the PVC

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dummy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dummy
  template:
    metadata:
      labels:
        app: dummy
    spec:
      containers:
      - image: busybox:1.28
        name: dummy
        command: [ "sh", "-c", "sleep 1h" ]
        volumeMounts:
        - mountPath: /data
          name: data
          readOnly: false
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: traefik

• Copy acme.json to the PVC using the busybox pod

kubectl cp acme.json kube-system/dummy-9f4d6b865-k6cnv:/data/acme.json

• Use the busybox pod to set userid and file permissions on the PVC

kubectl -n kube-system exec -it pod/dummy-577f48b686-6rcxr -- /bin/sh

Then on the pod prompt:

# cd /data
# chown 65532:65532 acme.json
# chmod go-rwx acme.json

• Reconfigure Traefik to use the PVC as the new certificate store

apiVersion: helm.cattle.io/v1
  kind: HelmChartConfig
  metadata:
    name: traefik
    namespace: kube-system
  spec:
    valuesContent: |-
      additionalArguments:
        - "--log.level=DEBUG"
        - "--certificatesresolvers.le.acme.email=t_schorr@gmx.de"
        - "--certificatesresolvers.le.acme.storage=/data/acme.json"
        - "--certificatesresolvers.le.acme.tlschallenge=true"
        - "--certificatesresolvers.le.acme.caServer=https://acme-v02.api.letsencrypt.org/directory"
        - "--entryPoints.web.proxyProtocol.trustedIPs=0.0.0.0/0"
      persistence:
        enabled: true
        existingClaim: "traefik"
        name: data
        path: /data
        annotations: {}