Kubernetes Secrets are base64 by default -- not encrypted. Enabling etcd encryption at rest closes the disk-level exposure. HashiCorp Vault provides enterprise-grade secret lifecycle management, dynamic credentials, and lease-based rotation that Kubernetes native Secrets cannot match.
Configure EncryptionConfiguration on kube-apiserver. Understand AES-GCM vs KMS provider. Rotate the encryption key. Understand what encryption at rest does and does not protect.
Design enterprise secret management with Vault: dynamic database credentials (no static passwords), PKI secrets engine (auto-issue TLS certs), Vault Agent Injector for zero-secret-in-pod patterns, lease renewal for credential rotation.
Kubernetes Secrets are base64 by default -- not encrypted. Enabling etcd encryption at rest closes the disk-level exposure. HashiCorp Vault provides enterprise-grade secret lifecycle management, dynamic credentials, and lease-based rotation that Kubernetes native Secrets cannot match.
etcd snapshots backed up to S3 with public-read ACL; encryption at rest not enabled
Security researcher finds public S3 bucket containing etcd snapshots
All 47 Secrets decoded from base64 in etcd snapshot -- plaintext credentials
Responsible disclosure; incident response begins; all credentials rotated
EncryptionConfiguration with AES-GCM deployed; S3 bucket ACL fixed; Vault integration planned
The question this raises
What does etcd encryption at rest actually protect against -- and what does it not protect against?
You enable etcd encryption at rest with EncryptionConfiguration. An attacker compromises a node running kube-apiserver and has root access to the node. What Kubernetes Secrets can they access?
Lesson outline
The Disk Exposure Threat Model
Kubernetes Secrets are base64-encoded in etcd -- trivially decodable. If an attacker gets the etcd data files or backups, they have all your Secrets. Encryption at rest means those files contain ciphertext, not plaintext. But the kube-apiserver still decrypts Secrets on every read -- so the API server itself remains a trust boundary.
EncryptionConfiguration (AES-GCM)
Use for: Configure via kube-apiserver --encryption-provider-config flag. AES-GCM key stored in config file on API server. Encrypts new Secrets on write; existing Secrets must be re-encrypted with: kubectl get secrets --all-namespaces -o json | kubectl replace -f -
KMS Envelope Encryption
Use for: API server generates per-Secret data encryption key (DEK); DEK encrypted by AWS KMS / GCP KMS / Vault key (KEK). Even if etcd + API server is compromised, master key is in external KMS. Two separate access paths required.
Vault Dynamic Credentials
Use for: Vault database secrets engine creates short-lived DB users on demand (TTL: 1h). Pod requests credential via Vault Agent sidecar; credential injected as file. No static password ever stored in K8s Secret.
Option A: AES-GCM (simpler, weaker) kubectl create secret db-pass --from-literal=password=hunter2 -> kube-apiserver encrypts with AES-GCM key from config file -> etcd stores: k8s:enc:aescbc:v1:key1:BASE64_CIPHERTEXT -> kube-apiserver config file on disk holds the AES key !! Root access to API server node = decrypt all secrets Option B: KMS Envelope (stronger, 2-party) kubectl create secret db-pass --from-literal=password=hunter2 -> kube-apiserver requests DEK from KMS plugin (gRPC) -> KMS plugin calls AWS KMS to encrypt DEK with CMK -> etcd stores: k8s:enc:kms:v2:ENCRYPTED_DEK + ENCRYPTED_DATA -> compromising etcd alone is not enough (encrypted DEK needs CMK) -> compromising API server alone is not enough (cannot get CMK without KMS auth) Option C: Vault Dynamic (no static secret at all) DB: no static users; Vault leases control access Pod startup -> Vault Agent -> Vault DB secrets engine -> Vault generates: username=vault-pod-xyz-1234, password=<random> -> Injected as /vault/secrets/db.env (TTL: 1h) -> Lease expires -> Vault revokes DB user -> No static password ever in K8s Secret
Three tiers of secret protection: each requires compromising an additional system to gain credential access
Secret Security Architecture Evolution
Production cluster with base64 Secrets and etcd backups in S3
“etcd backup leaked to public S3; all 47 credentials decoded in 30 seconds; full breach”
“KMS encryption at rest; etcd backups encrypted; even if S3 bucket is exposed, ciphertext is useless without KMS access”
Database password rotated on compromise
“Static password in K8s Secret; rotation requires: update Secret, rolling restart all pods that use it; downtime window”
“Vault dynamic credentials; revoke all leases immediately (break glass); all DB connections fail within TTL; new credentials issued on next pod startup; no K8s Secret to update”
Enabling etcd encryption at rest (AES-GCM)
01
1. Create EncryptionConfiguration YAML with resources: [secrets] and AES-GCM key (32-byte base64)
02
2. Store config on all control plane nodes at /etc/kubernetes/enc/encryption-config.yaml
03
3. Add --encryption-provider-config flag to kube-apiserver manifest
04
4. Restart kube-apiserver; NEW Secret writes are encrypted; existing Secrets remain plaintext
05
5. Re-encrypt existing Secrets: kubectl get secrets --all-namespaces -o json | kubectl replace -f -
06
6. Verify: etcdctl get /registry/secrets/default/my-secret | hexdump -- should show k8s:enc prefix not readable text
1. Create EncryptionConfiguration YAML with resources: [secrets] and AES-GCM key (32-byte base64)
2. Store config on all control plane nodes at /etc/kubernetes/enc/encryption-config.yaml
3. Add --encryption-provider-config flag to kube-apiserver manifest
4. Restart kube-apiserver; NEW Secret writes are encrypted; existing Secrets remain plaintext
5. Re-encrypt existing Secrets: kubectl get secrets --all-namespaces -o json | kubectl replace -f -
6. Verify: etcdctl get /registry/secrets/default/my-secret | hexdump -- should show k8s:enc prefix not readable text
1apiVersion: apiserver.config.k8s.io/v12kind: EncryptionConfiguration3resources:4- resources:5- secrets6providers:7- aescbc:8keys:9- name: key110secret: <32-byte-base64-encoded-key>identity: {} fallback -- allows reading Secrets written before encryption was enabled11- identity: {} # fallback for reading pre-encryption secretsAfter re-encrypting all Secrets, remove identity: {} to prevent plaintext fallback12# For KMS (stronger):13# - kms:14# name: aws-kms15# endpoint: unix:///var/run/kmsplugin/socket.sock16# cachesize: 100017# timeout: 3s
Encryption misconfiguration failure modes
EncryptionConfiguration without re-encrypting existing Secrets
# Step 1: Add EncryptionConfiguration to kube-apiserver ✓
# Step 2: Restart kube-apiserver ✓
# FORGOT Step 3: re-encrypt existing Secrets
# Result: NEW secrets encrypted, OLD secrets still plaintext
# etcd backup still exposes old secrets
# kubectl get secret old-secret -o yaml shows plaintext data# After enabling encryption, re-encrypt ALL existing Secrets:
kubectl get secrets --all-namespaces -o json | kubectl replace -f -
# Verify a specific secret is now encrypted:
# etcdctl get /registry/secrets/default/old-secret | strings
# Should show: k8s:enc:aescbc:v1:key1:... (not readable text)
# Only after verification, remove identity: {} fallback from configEnabling encryption does not retroactively encrypt existing Secrets. The kubectl replace command reads and re-writes all Secrets, triggering the API server to encrypt them with the new config. Must be done before removing the identity fallback provider.
| Tier | Protects against | Does NOT protect against | Complexity |
|---|---|---|---|
| Base64 (default) | Nothing | Any access to etcd data | None |
| AES-GCM encryption | etcd disk theft, backup exposure | API server compromise, key file theft | Low |
| KMS envelope | etcd + API server compromise (separate keys) | Compromise of both API server + KMS auth | Medium |
| Vault dynamic creds | Credential leak (TTL-limited) | Active session until TTL expiry | High |
| KMS + Vault + RBAC + Audit | Most practical threat vectors | Coordinated multi-system breach | Very high |
What etcd encryption at rest protects
📖 What the exam expects
Encrypts Secret data before writing to etcd disk. Protects against: etcd backup exposure, etcd disk theft, unauthorized etcd server access.
Toggle between what certifications teach and what production actually requires
Security architecture questions about protecting sensitive credentials and compliance questions about data-at-rest encryption.
Common questions:
Strong answer: Mentions KMS envelope encryption for key separation, Vault dynamic credentials to eliminate static passwords, and Vault Agent Injector for zero-secret-in-pod patterns.
Red flags: Thinking encryption at rest prevents all Secret exposure, or not knowing that the kube-apiserver holds the key for AES-GCM encryption.
Related concepts
Explore topics that connect to this one.
Ready to see how this works in the cloud?
Switch to Career Paths for structured paths (e.g. Developer, DevOps) and provider-specific lessons.
View role-based pathsSign in to track your progress and mark lessons complete.
Questions? Discuss in the community or start a thread below.
Join DiscordSign in to start or join a thread.