Identity, state and secrets: StatefulSets, volumes and ConfigMaps

Post 53 climbed the first half of the same Techmunch talk’s object ladder (Pod, Deployment, Service, Ingress) and stopped at a chaos Job that kills pods for a Deployment to replace. This is the second half, and the half that matters once the tutorial is over: a Deployment’s pods are interchangeable and forgettable, and most of what you actually build doesn’t want that. A database, a queue, anything that owns its own disk, wants pods with names, stable addresses and storage that survives a restart. That’s a StatefulSet, and the same little Flask app from post 53 (the one that picks a random human name at boot) makes the difference between the two objects something you watch happen in a single refresh, rather than something you take on faith from a diagram.

Why a Deployment isn’t enough

A Deployment’s contract is a headcount: “three of these, please,” reconciled by whatever means necessary. Kill a pod and a Deployment doesn’t try to recreate that pod: it just notices the count dropped and starts a new one, with a new hash-suffixed name and, in this demo, a new random human name too. That’s fine for a stateless web server. It’s not fine for anything that needs to be found again by the same address, or that wrote something to local disk it can’t afford to lose.

A StatefulSet’s contract is different: replicas: 3 still means three pods, but each one gets an ordinal identity (techmunch-0, techmunch-1, techmunch-2), created and torn down in that order, and if techmunch-1 dies, its replacement comes back as techmunch-1, not as a fourth pod with a random suffix. Paired with a PersistentVolumeClaim, that stable identity is what lets Kubernetes reattach the same disk to the same ordinal after a restart. Identity first, storage follows identity: that’s the whole idea, and 05-statefulset.yaml is where it starts.

05: ordinal names and a headless Service

deployments/05-statefulset.yaml, header paragraphs trimmed as noted above, the two kubectl lines and both manifests kept whole:

# kubectl apply -f deployments/05-statefulset.yaml
# kubectl delete -f deployments/05-statefulset.yaml

apiVersion: v1
kind: Service
metadata:
  name: techmunch-headless
  labels:
    app: techmunch-headless
spec:
  # ports:
  # - port: 8888
  #   name: http
  clusterIP: None
  selector:
    app: techmunch
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: techmunch
spec:
  selector:
    matchLabels:
      app: techmunch
  serviceName: techmunch-headless
  replicas: 3 
  template:
    metadata:
      labels:
        app: techmunch 
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: techmunch
          image: sjnarmstrong/techmunch-sholto:20210717
          imagePullPolicy: Always
          ports:
            - containerPort: 8888
              protocol: TCP
              name: http
          resources:
            requests:
              memory: "500Mi"
              cpu: "1000m"
            limits:
              memory: "500Mi"
              cpu: "1000m"

Two things make this a StatefulSet rather than a relabelled Deployment. First, clusterIP: None on the Service, a headless Service, one with no single virtual IP of its own. Instead of load-balancing, it gives every pod behind it its own DNS record: techmunch-0.techmunch-headless…, techmunch-1.techmunch-headless…, and so on. Second, spec.serviceName on the StatefulSet points at that headless Service by name, which is what tells the StatefulSet controller to publish those per-pod records in the first place.

That per-pod DNS is exactly what /neighbours walks (app.py, lines 74–91):

@app.route("/neighbours")
def neighbours():
    neighbors = []
    while True:
        try:
            # resp = requests.get(f"http://{my_app_name}-{len(neighbors)}:{app_port}/whoami")
            resp = requests.get(f"http://techmunch-{len(neighbors)}.techmunch-headless.techmunch-sholto.svc.cluster.local:{app_port}/whoami")
        except requests.exceptions.ConnectionError:
            break
        if resp.status_code != 200:
            break
        neighbors.append(resp.text)
    if len(neighbors) == 0:
        return "<h1>I have no neighbors</h1>"
    resp = ["<h1>My neighbors are:</h1>", "<ul>"]
    resp.extend((f"<li>{i}</li>" for i in neighbors))
    resp.append('</ul>')
    return ''.join(resp)

Ten lines, and it’s a real pattern: resolve ordinal 0, then 1, then 2, stopping the moment one doesn’t answer. That’s how a lot of clustered databases (etcd, Cassandra, Kafka’s own broker discovery in some setups) find their peers without a separate service registry: the ordinals are the registry. The gotcha, which the widget below reproduces on purpose rather than smoothing over, is that the walk stops at the first gap even if a higher ordinal is healthy. If techmunch-1 is mid-restart when you hit /neighbours, you get told about techmunch-0 and nothing else, even though techmunch-2 might be sitting there answering requests perfectly well.

Play with it: kill a pod on both sides

Both mini-clusters start with three pods each, no real cluster or backend behind either, the same state-machine engine from post 53, src/widgets/_shared/cluster-sim/, just with a StatefulSetController running next to the DeploymentController instead of alone. Kill a random pod (both) hits one pod on each side at once: the Deployment’s replacement always gets a fresh name, the StatefulSet’s replacement always comes back at the same ordinal, and, as long as Volumes attached stays checked, with the same self-reported name too. Uncheck it and kill another StatefulSet pod: the ordinal stays put, but the name changes, because there’s nothing left claiming to remember what it used to be. That one toggle is the whole argument for why the PersistentVolumeClaim in the next section exists: the ordinal identity was never the thing making the name persist, the volume was, and unplugging it proves it.

07: the volume that makes the identity trick real

deployments/07-persistant-volumes.yaml (the filename’s own typo, kept as the repo has it), quoted whole:

# kubectl apply -f deployments/07-persistant-volumes.yaml
# kubectl delete -f deployments/07-persistant-volumes.yaml

apiVersion: v1
kind: Service
metadata:
  name: techmunch-headless
  labels:
    app: techmunch-headless
spec:
  clusterIP: None
  selector:
    app: techmunch
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: techmunch
spec:
  selector:
    matchLabels:
      app: techmunch
  serviceName: techmunch-headless
  replicas: 3 
  template:
    metadata:
      labels:
        app: techmunch 
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: techmunch
          image: sjnarmstrong/techmunch-sholto:20210717
          imagePullPolicy: Always
          ports:
            - containerPort: 8888
              protocol: TCP
              name: http
          resources:
            requests:
              memory: "500Mi"
              cpu: "1000m"
            limits:
              memory: "500Mi"
              cpu: "1000m"
          volumeMounts:
            - name: data
              mountPath: "/etc/namesvol"

  volumeClaimTemplates: # Create PVC
  - metadata:
      name: data
    spec:
      resources:
        requests:
          storage: 1Gi
      accessModes: 
        - ReadWriteOnce
      # storageClassName: gp2

volumeClaimTemplates is the one field that makes a StatefulSet more than a Deployment with ordinals: it asks Kubernetes to create one PersistentVolumeClaim per pod ordinal (data-techmunch-0, data-techmunch-1, data-techmunch-2) and to reattach data-techmunch-N to whichever pod is currently running as ordinal N. The container never knows or cares which physical disk it got; it just always finds the same claim mounted at /etc/namesvol for its ordinal.

That mount path is exactly where the app’s whole trick lives (app.py, lines 18–26):

if os.path.isfile('/etc/namesvol/file'):
    my_name = open('/etc/namesvol/file', 'r').read()
else:
    my_name = NAMES[random.randint(0, len(NAMES)-1)]
    try:
        with open('/etc/namesvol/file', 'w') as fp:
            fp.write(my_name)
    except Exception as e:
        print(f"Could not save my name: {e}")

Before 07, /etc/namesvol was never mounted anywhere, so this branch always fell through to a fresh random name. From 07 on, it’s a real file on a real (claimed) volume: the first boot picks a name and writes it down, every boot after that reads it back. Kill a pod in the widget above with volumes attached and the replacement finds its own name already sitting on disk. That single observation (refresh, same name) is the entire practical difference between a Deployment and a StatefulSet, made visible without reading a line of controller source.

08: configuration the crude way

deployments/08-env-variables.yaml swaps the volume for a literal environment variable, and drops the PVC entirely along with it:

# kubectl apply -f deployments/08-env-variables.yaml
# kubectl delete -f deployments/08-env-variables.yaml

apiVersion: v1
kind: Service
metadata:
  name: techmunch-headless
  labels:
    app: techmunch-headless
spec:
  clusterIP: None
  selector:
    app: techmunch
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: techmunch
spec:
  selector:
    matchLabels:
      app: techmunch
  serviceName: techmunch-headless
  replicas: 3 
  template:
    metadata:
      labels:
        app: techmunch 
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: techmunch
          image: sjnarmstrong/techmunch-sholto:20210717
          imagePullPolicy: Always
          ports:
            - containerPort: 8888
              protocol: TCP
              name: http
          resources:
            requests:
              memory: "500Mi"
              cpu: "1000m"
            limits:
              memory: "500Mi"
              cpu: "1000m"
          env:
            - name: APP_CONFIG
              value: "This value is now different"

env.value is the crudest way to get a string into a container: it’s baked straight into the manifest, which means it’s baked into whatever system renders or templates that manifest, a Helm values file, a CI variable, someone’s shell history. It works for exactly one deployment of exactly one value. Change it and you’re editing YAML, not configuration. Worth noticing on the way past: because this manifest also has no volumeMounts and no volumeClaimTemplates, /etc/namesvol doesn’t exist here either: the StatefulSet’s ordinal identity is still stable (that’s a property of the StatefulSet controller itself, independent of storage), but the app’s self-reported name goes back to being fresh on every restart, exactly like a Deployment pod, until 07’s volume comes back.

09: the same ConfigMap, two mechanisms

deployments/09-configmap.yaml puts the value in a ConfigMap and consumes it two ways in the same container: as an injected environment variable, and as a mounted file:

# kubectl apply -f deployments/09-configmap.yaml
# kubectl delete -f deployments/09-configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: techmunch-config
data:
  # property-like keys; each key maps to a simple value
  config_property_1: "I am Property 1"
  config_property_2: "I am Property 2"
  config_property_name: "Elon"
---
apiVersion: v1
kind: Service
metadata:
  name: techmunch-headless
  labels:
    app: techmunch-headless
spec:
  clusterIP: None
  selector:
    app: techmunch
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: techmunch
spec:
  selector:
    matchLabels:
      app: techmunch
  serviceName: techmunch-headless
  replicas: 3 
  template:
    metadata:
      labels:
        app: techmunch 
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: techmunch
          image: sjnarmstrong/techmunch-sholto:20210717
          imagePullPolicy: Always
          ports:
            - containerPort: 8888
              protocol: TCP
              name: http
          resources:
            requests:
              memory: "500Mi"
              cpu: "1000m"
            limits:
              memory: "500Mi"
              cpu: "1000m"
          env:
            - name: APP_CONFIG
              valueFrom:
                configMapKeyRef:
                  name: techmunch-config
                  key: config_property_1
          volumeMounts:
            - name: config
              mountPath: "/etc/namesvol"
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: techmunch-config
            items:
            - key: "config_property_name"
              path: "file"

Two ConfigMap keys, two mechanisms, in the same pod at once: configMapKeyRef pulls config_property_1 (“I am Property 1”) into the APP_CONFIG environment variable the same way 08 did with a literal string, and a configMap volume with an items list projects config_property_name (“Elon”) as a single file named file. I’d planned to call these “the same value, two ways”, that’s the clean pedagogical framing, but reading the manifest closely, they’re not: config_property_1 and config_property_name are two different keys with two different values. What’s actually being shown side by side is two mechanisms pulling from the same ConfigMap object, not one value going two routes. Worth correcting rather than repeating the tidier version I’d have preferred to be true.

09: and the Secret that isn’t quite a secret

deployments/09-secrets.yaml is the same shape again, Secret in place of ConfigMap:

# kubectl apply -f deployments/09-secrets.yaml
# kubectl delete -f deployments/09-secrets.yaml
# https://kubernetes.io/docs/concepts/configuration/secret/
apiVersion: v1
kind: Secret
metadata:
  name: techmunch-secret
data:
  # property-like keys; each key maps to a simple value
  config_property_1: "VGhpcyBpcyBub3QgYSBzZWNyZXQgdmFsdWUuLi4uIE9yIGlzIGl0Cg=="
  config_property_2: "VGhpcyBpcyBhIHNlY3JldCB2YWx1ZQo="
  file: "U3RpbGwgRWxvbgo="
---
apiVersion: v1
kind: Service
metadata:
  name: techmunch-headless
  labels:
    app: techmunch-headless
spec:
  clusterIP: None
  selector:
    app: techmunch
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: techmunch
spec:
  selector:
    matchLabels:
      app: techmunch
  serviceName: techmunch-headless
  replicas: 3 
  template:
    metadata:
      labels:
        app: techmunch 
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: techmunch
          image: sjnarmstrong/techmunch-sholto:20210717
          imagePullPolicy: Always
          ports:
            - containerPort: 8888
              protocol: TCP
              name: http
          resources:
            requests:
              memory: "500Mi"
              cpu: "1000m"
            limits:
              memory: "500Mi"
              cpu: "1000m"
          env:
            - name: APP_CONFIG
              valueFrom:
                secretKeyRef:
                  name: techmunch-secret
                  key: config_property_1
            - name: APP_SECRET_CONFIG
              valueFrom:
                secretKeyRef:
                  name: techmunch-secret
                  key: config_property_2
          volumeMounts:
            - name: config
              mountPath: "/etc/namesvol"
              readOnly: true
      volumes:
        - name: config
          secret:
            secretName: techmunch-secret
            # items:
            # - key: "config_property_name"
            #   path: "file"

kubectl never sends or stores those data: values as plain text: a Secret’s values are base64, and base64 is an encoding, not encryption. Anyone with get permission on the Secret, or anyone who can read the pod’s environment or its mounted file, has the plaintext in one command. The values here are the joke made explicit: run echo VGhpcyBpcyBub3QgYSBzZWNyZXQgdmFsdWUuLi4uIE9yIGlzIGl0Cg== | base64 -d on stage and out comes “This is not a secret value… Or is it”, decode the widget below to see all three, or type your own text in and watch it “become a Secret” the same way kubectl would store it.

With JavaScript enabled, this becomes a live decoder over the real data: block in 09-secrets.yaml, plus a two-way box where you can watch your own text turn into (and back out of) base64.

The volumes block here has its items list commented out, unlike 09-configmap.yaml’s. Leaving it out doesn’t mean nothing gets mounted: it means Kubernetes projects every key in the Secret as its own file, named after the key: /etc/namesvol/config_property_1, /etc/namesvol/config_property_2, and /etc/namesvol/file. That last one exists again, same path as before, decoding to “Still Elon.” The joke from 09-configmap.yaml survives the swap to Secrets completely intact, every pod at this stage of the demo introduces itself as “Still Elon” regardless of what names.py would have picked, for exactly the same accidental-looking-deliberate reason.

2021 vs. now

Four years is a long time in this ecosystem. 04-ingress.yaml’s extensions/v1beta1 API, covered in post 53, was already removed by the time this talk happened. app.run() at the bottom of app.py runs at import time rather than behind an if __name__ == "__main__": guard, and /kill’s call to request.environ.get('werkzeug.server.shutdown') was removed from Werkzeug in 2022, both fine for a five-minute conference demo pinned to Flask==2.0.1, neither something I’d ship. The container still runs as root by default.

What’s changed in Kubernetes itself since 2021, for exactly the objects in this post: StatefulSet’s volumeClaimTemplates gained the ability to be resized in place on supporting storage classes without recreating the pod (allowVolumeExpansion, generally available by 1.24); minReadySeconds and a configurable podManagementPolicy on StatefulSets are both more commonly reached-for now for exactly the “don’t stampede every ordinal at once” problem this demo doesn’t hit at three replicas. On the Secrets side, KMS-backed encryption providers matured, and the ecosystem’s default advice moved further toward “a Kubernetes Secret is a delivery mechanism into a pod, not your source of truth”, which is the same conclusion the base64 punchline above was already pointing at in 2021, just with better tooling to act on it now.