Kubernetes Secrets Management & Encryption at Rest for the CKS Exam: etcd Encryption, KMS Providers, RBAC & External Secrets

Introduction

Almost every candidate who fails a Secrets question on the Certified Kubernetes Security Specialist (CKS) exam does so for the same reason: they believe a Kubernetes Secret is encrypted. It isn’t. By default a Secret is stored in etcd as base64-encoded plaintext — encoding, not encryption. Anyone who can read the etcd datastore, or who has broad get secret permission through RBAC, can recover the value in one command.

Protecting Secrets is a recurring theme across the CKS curriculum. It touches Cluster Setup (encrypting the datastore), Cluster Hardening (restricting who can read Secrets), Minimize Microservice Vulnerabilities (how workloads consume secrets safely), and Supply Chain Security (keeping credentials out of images). The exam tests it as a practical task: enable encryption at rest, re-encrypt what’s already there, verify it worked, and lock down access — all against a live cluster and a clock.

This guide walks the full picture the way a practitioner builds it: what Kubernetes actually does with a Secret, how to turn on encryption at rest with an EncryptionConfiguration, how envelope encryption with a KMS provider raises the bar, how to consume secrets without leaking them, and where external secret stores fit. Every step includes the YAML and kubectl/etcdctl you’d type on exam day.

What Kubernetes Actually Does with a Secret

Create a Secret and inspect it, and the “encryption” illusion falls apart immediately:

kubectl create secret generic db-cred \
  --from-literal=username=admin \
  --from-literal=password=S3cr3t!

kubectl get secret db-cred -o jsonpath='{.data.password}' | base64 -d
# S3cr3t!

The value in etcd is data.password: UzNjcjN0IQ== — a base64 string that anyone can decode. Base64 is a transport encoding, not a security control. That single fact drives the whole topic:

On the wire, API traffic is protected by TLS, so a Secret in transit is safe.
At rest, the Secret sits in etcd. Unless you have configured encryption at rest, it sits there as recoverable plaintext.
In access terms, anyone whose RBAC allows reading the Secret — directly, or by creating a Pod that mounts it — can read the value regardless of at-rest encryption.

So there are three defenses to configure, and the exam can test any of them: encrypt the datastore, restrict access with RBAC, and consume secrets safely in workloads. Encryption at rest is the headline task, so start there.

Encryption at Rest: The EncryptionConfiguration

Encryption at rest is controlled by an EncryptionConfiguration object that you hand to the kube-apiserver. The API server encrypts resources on write and decrypts them on read; etcd never sees the plaintext once this is on.

A minimal configuration that encrypts Secrets with AES-CBC looks like this. Place it on the control-plane node, for example at /etc/kubernetes/enc/enc.yaml:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <BASE64_ENCODED_32_BYTE_KEY>
      - identity: {}

Generate the key with head -c 32 /dev/urandom | base64. Two details in that file decide everything, and both are classic exam traps:

Provider order matters. The first provider in the list is used to encrypt new writes. Every provider listed can be used to decrypt reads, tried in order. Here aescbc encrypts and identity (plaintext) is the fallback for decrypting anything not yet encrypted.
identity means no encryption. If identity were listed first, new Secrets would be written in plaintext. A common trap answer puts identity at the top — that silently disables encryption while looking configured.

Provider Options

Provider	Encryption	Notes
`identity`	None (plaintext)	The default. First-in-list = encryption effectively off.
`aescbc`	AES-CBC with PKCS#7	Widely used in exam tasks; strong and simple.
`aesgcm`	AES-GCM	Faster, but you must rotate keys frequently; use with automation.
`secretbox`	XSalsa20 + Poly1305	Alternative symmetric option.
`kms` (v1/v2)	Envelope encryption via external KMS	Best practice for production — keys never live in the config file.

For the exam, know how to configure aescbc (or aesgcm) confidently, and understand what kms buys you conceptually.

Wiring It into the API Server

The API server is a static Pod on the control-plane node, so you edit its manifest directly at /etc/kubernetes/manifests/kube-apiserver.yaml. Add the flag and mount the file:

spec:
  containers:
    - command:
        - kube-apiserver
        - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
        # ...existing flags...
      volumeMounts:
        - name: enc
          mountPath: /etc/kubernetes/enc
          readOnly: true
  volumes:
    - name: enc
      hostPath:
        path: /etc/kubernetes/enc
        type: DirectoryOrCreate

The moment you save that file, the kubelet restarts the API server Pod. Watch it come back with kubectl get pods -n kube-system (or crictl ps if the API is briefly unavailable). Editing a static Pod manifest without bricking the API server is a hands-on skill the CKS rewards — the same muscle you build wiring up admission plugins in the admission control guide and hardening the API server in the cluster setup and hardening guide.

Re-Encrypting Existing Secrets

Turning on encryption only affects future writes. Every Secret that already existed is still plaintext in etcd. To fix that, force a rewrite of all Secrets so they pass back through the API server and get encrypted:

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

This reads every Secret and writes it straight back. Because encryption is now enabled, each rewritten Secret is stored encrypted. This “read-all, replace-all” one-liner is one of the most frequently demonstrated CKS commands — commit it to memory.

Verifying It Actually Worked

Never trust that encryption is on because the config “looks right.” Verify by reading the raw bytes straight out of etcd:

ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/db-cred | hexdump -C | head

An encrypted Secret begins with a provider prefix such as k8s:enc:aescbc:v1:key1: followed by ciphertext. If you instead see readable text (admin, your password), the Secret is still plaintext — either encryption isn’t active or you forgot the re-encrypt step. This etcd check is the definitive proof, and examiners love asking you to demonstrate it.

Envelope Encryption with a KMS Provider

Storing the AES key inside enc.yaml on the control-plane node is better than nothing, but the key now lives on disk next to the data it protects. The production-grade answer is a KMS provider, which implements envelope encryption:

A Key Encryption Key (KEK) lives in an external KMS (a cloud KMS or an in-cluster provider like a Vault-backed plugin) and never leaves it.
The API server generates a Data Encryption Key (DEK) to encrypt each resource, then asks the KMS to encrypt the DEK with the KEK. Only the encrypted DEK is stored alongside the data.

The benefit: the sensitive root key is managed, audited, and rotated in a dedicated system, and you can revoke access centrally. KMS v2 (stable since Kubernetes 1.29) improves performance by caching DEKs and adds key-ID tracking so you can tell when a rotation is needed. Configuration points the provider at a local unix socket exposed by a KMS plugin:

providers:
  - kms:
      apiVersion: v2
      name: myKmsPlugin
      endpoint: unix:///var/run/kmsplugin/socket.sock
  - identity: {}

For the CKS you won’t stand up a full cloud KMS, but you should be able to explain why KMS is preferred over static keys (the KEK never touches the node, central rotation and revocation) and recognize the DEK/KEK split when a question describes it.

Locking Down Access with RBAC

Encryption at rest defends against someone reading etcd or a disk snapshot. It does nothing against someone with API permission to read the Secret. If a ServiceAccount can get or list Secrets, it can read their decrypted values through the API. So least-privilege RBAC is the second, equally important defense.

Audit who can read Secrets:

kubectl auth can-i list secrets --as=system:serviceaccount:default:my-app
# ideally: no

Grant Secret access narrowly — a Role scoped to a single named Secret in one namespace, never a cluster-wide get secrets:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: payments
  name: read-db-cred
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["db-cred"]     # this Secret only
    verbs: ["get"]

Two related hardening steps the exam values:

Disable ServiceAccount token auto-mount where a Pod doesn’t need the API. Set automountServiceAccountToken: false on the ServiceAccount or Pod so a compromised container can’t use a mounted token to pull Secrets.
Beware indirect access. Anyone who can create Pods in a namespace can mount that namespace’s Secrets into a container and read them. Restricting Pod creation is therefore part of protecting Secrets — a point that connects to Pod Security Standards in the minimize microservice vulnerabilities guide.

Consuming Secrets Safely in Workloads

How a Pod reads a Secret changes its exposure. There are two mechanisms, and they are not equal:

Method	How	Exposure trade-off
Environment variable	`env.valueFrom.secretKeyRef`	Convenient, but the value lands in the process environment. Child processes inherit it, crash dumps and some logs can capture it, and it doesn’t update when the Secret changes.
Projected volume	`volumes.secret` mounted into the container	Preferred. The value is a file, not an env var; it is not shown in `kubectl describe pod`, and the kubelet updates the file when the Secret changes.

A volume mount is the safer default:

spec:
  containers:
    - name: app
      image: myapp:1.4
      volumeMounts:
        - name: cred
          mountPath: /etc/db
          readOnly: true
  volumes:
    - name: cred
      secret:
        secretName: db-cred
        defaultMode: 0400

Set a tight defaultMode (owner read-only) so other users in the container can’t read the file. And the golden rule that spans the whole supply chain: never bake secrets into a container image or commit them to Git — an image layer is world-readable to anyone who can pull it, and a committed secret lives forever in history.

Where External Secret Stores Fit

Native Secrets, even encrypted, still live in your cluster. Many teams keep the source of truth in a dedicated secrets manager and sync into Kubernetes on demand. For the CKS you should recognize the patterns rather than master every tool:

External Secrets Operator (ESO): watches an ExternalSecret custom resource and pulls values from an external store (HashiCorp Vault, cloud secret managers), creating a normal Kubernetes Secret from them. The source of truth stays outside the cluster.
HashiCorp Vault (with the Agent Injector or CSI): injects secrets into Pods at runtime, often as files, so they never persist as Kubernetes Secrets at all.
Sealed Secrets: lets you commit an encrypted SealedSecret to Git safely; an in-cluster controller decrypts it into a real Secret. This solves the “secrets in GitOps” problem.

These are open-source, vendor-neutral building blocks. The exam won’t ask you to configure Vault end-to-end, but it may describe a scenario (“keep the real credential out of the cluster and out of Git”) and expect you to name the right approach.

Common CKS Secrets Traps to Watch For

Base64 is not encryption. A default Secret is recoverable plaintext in etcd.
identity first = encryption off. The first provider encrypts; put a real cipher first, identity last.
Enabling encryption doesn’t touch existing Secrets. Run kubectl get secrets -A -o json | kubectl replace -f - to re-encrypt.
Verify with etcdctl, not with the config file. Look for the k8s:enc: prefix in the raw etcd bytes.
RBAC beats encryption for API-level access. Encryption at rest doesn’t stop a Subject who can get the Secret.
Pod creators can read namespace Secrets. Restrict who can create Pods, not just who can read Secrets.
Prefer volume mounts over env vars, and never put secrets in images or Git.

Conclusion

Secrets management is a topic that reads simple and breaks in practice — exactly the kind of thing the CKS is built to test. The concepts fit on a page: base64 isn’t encryption, the API server encrypts at rest via an EncryptionConfiguration, KMS envelope encryption keeps the root key out of the cluster, and RBAC decides who can read the decrypted value. The exam tests whether you can do it — edit the API server static Pod without bricking it, re-encrypt existing Secrets, and prove the result by reading etcd directly.

Build the sequence as muscle memory: write the config, wire the flag and volume, restart and verify the API server, re-encrypt, then confirm with etcdctl. Then layer the access controls — least-privilege Roles on named Secrets, no unnecessary token auto-mounts, volume mounts over env vars.

The fastest way to make exam day uneventful is to break and repair a real cluster under time pressure. Sailor.sh’s Certified Kubernetes Security Specialist (CKS) Mock Exam Bundle gives you full-length, browser-based performance exams against live clusters, with the same style of tasks — enabling encryption at rest, re-encrypting Secrets, and tightening RBAC — so the workflow is automatic by the time it counts.

If you’d rather start free and local, the CKS practice environment guide walks through building a lab, and the CKS study plan sequences the domains. For the wider picture, pair this with the Kubernetes security best practices guide and the full CKS exam guide for 2026. The official Kubernetes encryption-at-rest documentation is allowed during the exam — practice navigating it quickly.

Frequently Asked Questions

Are Kubernetes Secrets encrypted by default?

No. By default a Secret is stored in etcd as base64-encoded data, which is trivially decodable — it is encoding, not encryption. You must configure encryption at rest with an EncryptionConfiguration (using aescbc, aesgcm, secretbox, or a kms provider) to store Secrets encrypted in etcd.

How do I encrypt existing Secrets after enabling encryption at rest?

Enabling encryption only affects new writes. Re-encrypt everything already stored by forcing a rewrite: kubectl get secrets --all-namespaces -o json | kubectl replace -f -. Each Secret is read and written back, and because encryption is now active, it is stored encrypted.

How can I verify that a Secret is actually encrypted in etcd?

Read the raw value straight from etcd with etcdctl get /registry/secrets/<namespace>/<name>. An encrypted Secret starts with a provider prefix like k8s:enc:aescbc:v1:key1: followed by ciphertext. If you can see the plaintext value, encryption is not working or you skipped the re-encrypt step.

What is the difference between an aescbc provider and a KMS provider?

With aescbc the AES key lives inside your EncryptionConfiguration file on the control-plane node. A kms provider uses envelope encryption: a Key Encryption Key stays inside an external KMS and never touches the node, while per-resource Data Encryption Keys are encrypted by it. KMS is preferred because the root key is centrally managed, audited, and revocable.

Does encryption at rest protect a Secret from someone with RBAC access?

No. Encryption at rest protects the datastore (etcd, disk snapshots). It does nothing against a Subject who has API permission to get/list the Secret, or who can create a Pod that mounts it — they receive the decrypted value. You need least-privilege RBAC in addition to encryption.

Should I inject secrets as environment variables or as mounted files?

Prefer mounted volume files. Environment variables are inherited by child processes, can surface in logs or crash dumps, and don’t update when the Secret changes. A projected Secret volume keeps the value out of kubectl describe, updates automatically, and can be locked down with a restrictive defaultMode.