Interactive Explainer

🎯Key Takeaways

A secret is any value that grants access — API keys, passwords, certificates, tokens, encryption keys.

Never store secrets in source code, .env files, Docker images, or environment variables passed via CLI.

Dynamic secrets (generated on-demand with a TTL) are far safer than static secrets — they expire automatically.

HashiCorp Vault (or AWS Secrets Manager, GCP Secret Manager) should be the single source of truth for all secrets.

Use OIDC federation in CI/CD to eliminate static cloud credentials from your pipeline configuration.

Rotate secrets automatically — dynamic credentials do this by design; static credentials need explicit rotation schedules.

Audit logs are non-negotiable — you need to know who accessed which secret and when for incident response and compliance.

Kubernetes Secrets are not a secrets manager — they are base64-encoded and require etcd encryption at rest.

Secrets Management

How to store, rotate, inject, and audit credentials, API keys, and certificates so they never live in code, env files, or container images.

~12 min read

Be the first to complete!

Why this matters

Credentials hardcoded in source code, .env files committed to Git, or secrets baked into Docker images are the #1 cause of cloud breaches.

Without this knowledge

A single leaked API key can give attackers full access to your cloud account, database, or third-party services — and you may not know for months.

With this knowledge

Secrets live in a dedicated vault with short-lived leases, automatic rotation, fine-grained access policies, and a complete audit trail of who accessed what and when.

What you'll learn

A secret is any value that grants access — API keys, passwords, certificates, tokens, encryption keys.
Never store secrets in source code, .env files, Docker images, or environment variables passed via CLI.
Dynamic secrets (generated on-demand with a TTL) are far safer than static secrets — they expire automatically.
HashiCorp Vault (or AWS Secrets Manager, GCP Secret Manager) should be the single source of truth for all secrets.
Use OIDC federation in CI/CD to eliminate static cloud credentials from your pipeline configuration.
Rotate secrets automatically — dynamic credentials do this by design; static credentials need explicit rotation schedules.
Audit logs are non-negotiable — you need to know who accessed which secret and when for incident response and compliance.
Kubernetes Secrets are not a secrets manager — they are base64-encoded and require etcd encryption at rest.

Lesson outline

What Is a Secret?

A secret is any credential or sensitive value your application uses at runtime that must not be visible to unauthorised parties.

Common secret types

API keys & tokens — Stripe, Twilio, GitHub PATs, cloud provider access keys
Database credentials — Connection strings with usernames and passwords
TLS/SSL certificates & private keys — Used for mTLS, HTTPS termination, signing
SSH keys — Server access, deployment keys
OAuth client secrets — Used in server-side OAuth flows
Encryption keys — Data-at-rest encryption, KMS master keys
Webhook signing secrets — Used to verify payload integrity

If it grants access — it is a secret

Any value that, if leaked, allows an attacker to impersonate your service, access your data, or call a third-party API on your behalf must be treated as a secret. This includes "read-only" keys — they still grant access.

Common Mistake

The Secrets Anti-Patterns That Keep Getting Companies Breached

Most breaches involving credentials follow one of these patterns — and all of them are avoidable.

Anti-Pattern	Why It Happens	Real Risk
Hardcoded in source code	Developer convenience during dev	Exposed in GitHub, code reviews, compiled binaries
.env files committed to Git	"It is in .gitignore" — until it is not	Leaked in CI logs, container images, public forks
Secrets in environment variables passed via CLI	Visible in ps aux, /proc/$pid/environ on Linux	Exploitable by any process on the same host
Secrets in Docker image layers	COPY .env or RUN curl -H "Authorization: $KEY"	Any docker history or image pull leaks the secret
Secrets shared over Slack/email	"Temporary" — then forgotten	Persists in chat history indefinitely
Same secret in dev/staging/prod	Convenience	Dev breach → prod breach
No rotation policy	"It has never been rotated because nothing broke"	A leaked secret stays valid indefinitely

The .gitignore false sense of security

.gitignore prevents future commits of a file — but if the file was ever committed (even once, even "by accident"), it lives in Git history forever. git log --all -p -- .env will find it. Tools like truffleHog and gitleaks scan the full commit history.

Secrets Vault Architecture: HashiCorp Vault Deep Dive

HashiCorp Vault is the industry standard open-source secrets manager. Understanding its architecture clarifies how any secrets manager works.


  ┌─────────────────────────────────────────────────────────────┐
  │                    HashiCorp Vault                          │
  │                                                             │
  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐  │
  │  │  Auth Methods│  │ Secret Engines│  │  Audit Backends  │  │
  │  │              │  │              │  │                  │  │
  │  │ • Kubernetes │  │ • KV v2      │  │ • File           │  │
  │  │ • AWS IAM    │  │ • AWS        │  │ • Syslog         │  │
  │  │ • GitHub     │  │ • Database   │  │ • Socket         │  │
  │  │ • LDAP       │  │ • PKI        │  │                  │  │
  │  │ • AppRole    │  │ • Transit    │  │  Every request   │  │
  │  │ • JWT/OIDC   │  │ • SSH        │  │  logged with     │  │
  │  └──────┬───────┘  └──────┬───────┘  │  accessor+time   │  │
  │         │                 │          └──────────────────┘  │
  │         └────────┬────────┘                                │
  │                  ▼                                          │
  │         ┌────────────────┐                                 │
  │         │  Policy Engine │ (HCL — who can read/write what) │
  │         └────────────────┘                                 │
  │                  │                                          │
  │         ┌────────▼───────┐                                 │
  │         │  Storage Layer │ (Raft/Consul/etcd/DynamoDB)      │
  │         │  Encrypted at  │                                 │
  │         │  rest (AES-256)│                                 │
  │         └────────────────┘                                 │
  └─────────────────────────────────────────────────────────────┘

Vault decouples authentication (who you are) from authorisation (what you can access) from storage (encrypted at rest).

Key Vault concepts

Auth Methods — Verify identity — Kubernetes service account, AWS IAM role, GitHub token, username+password
Secret Engines — Plugins that generate or store secrets — KV (static), Database (dynamic), PKI (certificates), AWS (ephemeral IAM creds)
Policies (HCL) — path "secret/data/myapp/*" { capabilities = ["read"] } — fine-grained per-path ACLs
Leases & TTLs — Dynamic secrets have a time-to-live and are auto-revoked — no long-lived credentials
Audit Backend — Every read, write, token creation is logged — required for SOC 2/PCI compliance
Seal/Unseal — Vault data is encrypted; requires unseal keys (Shamir's Secret Sharing) or auto-unseal via cloud KMS

Aha!

Dynamic Secrets: The Game-Changer

Static secrets (a password you store and retrieve) are dangerous because they last forever unless manually rotated. Dynamic secrets are generated on-demand and expire automatically.

How dynamic database credentials work

Your app requests credentials from Vault → Vault creates a temporary DB user with a 1-hour TTL → App uses those credentials → TTL expires → Vault auto-revokes that DB user. If the credentials leak, they expire in 1 hour — not years.

Dynamic secret lifecycle (Database engine)

→

App authenticates to Vault using its Kubernetes service account JWT

→

Vault validates the service account with the Kubernetes API

→

App requests credentials: vault read database/creds/my-role

→

Vault connects to the DB and executes: CREATE USER "v-app-xyz" WITH PASSWORD "..." VALID UNTIL NOW() + INTERVAL '1 hour'

→

Vault returns username + password + lease_id + TTL to the app

→

App uses credentials for DB connections

At TTL expiry, Vault executes: DROP USER "v-app-xyz" — credential is gone

App authenticates to Vault using its Kubernetes service account JWT

Vault validates the service account with the Kubernetes API

App requests credentials: vault read database/creds/my-role

Vault connects to the DB and executes: CREATE USER "v-app-xyz" WITH PASSWORD "..." VALID UNTIL NOW() + INTERVAL '1 hour'

Vault returns username + password + lease_id + TTL to the app

App uses credentials for DB connections

At TTL expiry, Vault executes: DROP USER "v-app-xyz" — credential is gone

Dynamic secrets eliminate the rotation problem

Static credentials need a rotation schedule, reminders, runbooks, and someone to actually do it. Dynamic credentials rotate themselves — by design. The "rotation" is just the credential expiring and a new one being issued on next request.

vault-dynamic-creds.sh

1# Dynamic database credentials — request and use in one step
2# Vault CLI
vault read = request new dynamic credentials
3vault read database/creds/myapp-role
4# Key                Value
5# ---                -----
6# lease_id           database/creds/myapp-role/AbCdEf123456
7# lease_duration     1h
8# lease_renewable    true
9# password           A1B2-c3d4-E5f6-G7H8   ← auto-generated, expires in 1h
10# username           v-myapp-AbCdEf         ← auto-created DB user
11 
12# Kubernetes — Vault Agent Sidecar injection
13# The sidecar requests secrets from Vault and writes them to a shared volume
14# No Vault SDK needed in your application code
15apiVersion: v1
16kind: Pod
Agent sidecar — no Vault SDK in your app code
17metadata:
18  annotations:
Kubernetes service account auth — no static token needed
19    vault.hashicorp.com/agent-inject: "true"
20    vault.hashicorp.com/role: "myapp-role"
21    vault.hashicorp.com/agent-inject-secret-db: "database/creds/myapp-role"
22    vault.hashicorp.com/agent-inject-template-db: |
23      {{- with secret "database/creds/myapp-role" -}}
24      DB_USER={{ .Data.username }}
25      DB_PASS={{ .Data.password }}
26      {{- end -}}
27spec:
28  containers:
29  - name: myapp
30    image: myapp:latest
31    # App reads /vault/secrets/db — populated by the Vault Agent sidecar
32    env:
33    - name: DB_CREDS_FILE
34      value: /vault/secrets/db

Secrets Injection Patterns in Kubernetes

Getting secrets from a vault into a running container without touching environment variables (which leak into process listings) requires one of these patterns.

Pattern	How It Works	Pros	Cons
Vault Agent Sidecar	Sidecar container fetches secrets, writes to shared volume	No SDK changes, auto-renewal	Extra container per pod
External Secrets Operator (ESO)	K8s operator syncs vault secrets → K8s Secrets	Native K8s objects, GitOps-friendly	Secrets stored in etcd (encrypt at rest!)
Secrets Store CSI Driver	Mounts secrets as a volume from external provider	No K8s Secret object created	Complex setup, driver per provider
Init Container	Init container fetches secrets, writes to shared emptyDir	Simple, no sidecar	No auto-renewal, manual refresh
Direct SDK (Vault API)	App code calls Vault API directly at startup	Full control, dynamic renewal	SDK dependency, code changes required

External Secrets Operator is the GitOps-friendly choice

ESO lets you define an ExternalSecret CR that says "fetch this path from Vault and create a K8s Secret named X". Your manifests are in Git (no real secrets), and ESO syncs them. Combine with sealed-secrets or encrypt etcd at rest to protect the K8s Secret objects.

Never use Kubernetes Secrets as your secrets store

Kubernetes Secrets are base64-encoded (not encrypted) by default and stored in etcd. Any user with "get secrets" RBAC permission can decode them. Always encrypt etcd at rest and use a proper vault for the source of truth.

external-secrets-operator.yaml

1# External Secrets Operator — sync Vault secret to K8s Secret
2apiVersion: external-secrets.io/v1beta1
SecretStore — where to fetch secrets from
3kind: SecretStore
4metadata:
5  name: vault-backend
6  namespace: production
7spec:
8  provider:
9    vault:
10      server: "https://vault.internal:8200"
11      path: "secret"
12      version: "v2"
13      auth:
14        kubernetes:
15          mountPath: "kubernetes"
Kubernetes service account auth — no static Vault token
16          role: "myapp-production"
17---
18apiVersion: external-secrets.io/v1beta1
19kind: ExternalSecret
20metadata:
21  name: myapp-db-secret
22  namespace: production
ExternalSecret — what to fetch and where to put it
23spec:
24  refreshInterval: 15m           # Re-sync every 15 minutes
25  secretStoreRef:
26    name: vault-backend
27    kind: SecretStore
28  target:
refreshInterval — how often to re-sync from Vault
29    name: myapp-db-creds          # Creates this K8s Secret
30    creationPolicy: Owner
31  data:
32  - secretKey: DB_PASSWORD        # Key in K8s Secret
33    remoteRef:
34      key: secret/data/production/myapp  # Path in Vault
35      property: db_password              # Field within that path

Secret Rotation: Manual vs Automated

Rotation is the practice of replacing a secret with a new value on a schedule. Static secrets that never rotate are ticking time bombs.

Automated rotation workflow (AWS Secrets Manager example)

→

Rotation trigger fires (schedule: every 30 days, or immediately on suspected leak)

→

Secrets Manager invokes a Lambda rotation function

→

Lambda creates new DB password and sets it on the RDS instance

→

Lambda updates the secret value in Secrets Manager with the new password

→

Lambda tests the new credentials (optional but recommended)

→

Old version is demoted to AWSPREVIOUS (kept for rollback)

Applications using Secrets Manager SDK automatically get the new value on next fetch

Rotation trigger fires (schedule: every 30 days, or immediately on suspected leak)

Secrets Manager invokes a Lambda rotation function

Lambda creates new DB password and sets it on the RDS instance

Lambda updates the secret value in Secrets Manager with the new password

Lambda tests the new credentials (optional but recommended)

Old version is demoted to AWSPREVIOUS (kept for rollback)

Applications using Secrets Manager SDK automatically get the new value on next fetch

Zero-downtime rotation requires dual-active credentials

During rotation, both old and new credentials are temporarily valid. The DB accepts both while applications cache refresh. Once all apps have the new secret, the old one is revoked. This "dual-write" period prevents connection failures during rotation.

Secret Type	Rotation Strategy	Typical TTL
Database passwords	Vault Database engine / AWS Secrets Manager rotation Lambda	1–24 hours (dynamic) or 30–90 days (static)
API keys (third-party)	Manual rotation + automated alerts for age > 90 days	90 days
TLS certificates	cert-manager (Let's Encrypt), AWS ACM auto-renewal	90 days (Let's Encrypt)
Cloud IAM access keys	Avoid long-lived keys; use IAM roles / instance profiles instead	Never (use roles)
OAuth client secrets	Rotation via IdP API + application config update	1 year
SSH host keys	Vault SSH engine (OTP or signed certificates)	Per-session (OTP)

Secrets in CI/CD Pipelines

CI/CD pipelines need secrets to deploy infrastructure, push images, and call APIs — but pipeline logs are often public and artifacts are archived.

Safe patterns for CI/CD secrets

OIDC federation (preferred) — GitHub Actions / GitLab CI generate a short-lived OIDC token. Cloud providers (AWS, GCP, Azure) accept this token to issue temporary credentials — no static secrets stored in CI at all.
Vault AppRole — CI runner authenticates with a role_id + secret_id (both stored as masked CI variables) → gets a short-lived Vault token → fetches real secrets. secret_id has a use-count limit and TTL.
CI/CD native secret stores — GitHub Encrypted Secrets, GitLab CI Variables (masked), CircleCI Contexts — adequate for small teams, limited audit trail vs Vault.
Never echo secrets — Mask all secret variables. Use set +x in bash. Never run env in pipeline logs. CI platforms auto-mask known secret variables but custom vars need explicit masking.

OIDC eliminates static CI secrets entirely

With GitHub Actions OIDC + AWS: the workflow requests a JWT from GitHub → exchanges it for temporary AWS credentials via sts:AssumeRoleWithWebIdentity → credentials expire when the job finishes. Zero secrets stored in GitHub Settings.

github-actions-oidc.yml

1# GitHub Actions — OIDC federation with AWS (no static AWS keys)
2name: Deploy
3 
4on:
5  push:
6    branches: [main]
7 
8permissions:
id-token: write is required to request OIDC token
9  id-token: write   # Required for OIDC token request
10  contents: read
11 
12jobs:
13  deploy:
14    runs-on: ubuntu-latest
15    steps:
16    - uses: actions/checkout@v4
17 
18    - name: Configure AWS credentials via OIDC
19      uses: aws-actions/configure-aws-credentials@v4
OIDC — no static AWS keys stored in GitHub Secrets
20      with:
21        role-to-assume: arn:aws:iam::123456789012:role/github-actions-deploy
22        aws-region: us-east-1
23        # No AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY needed!
24        # GitHub generates a JWT, AWS exchanges it for temp credentials
25 
26    - name: Fetch app secrets from Vault
27      uses: hashicorp/vault-action@v2
28      with:
Vault also accepts OIDC JWT — single auth mechanism
29        url: https://vault.internal:8200
30        method: jwt          # Use the GitHub OIDC JWT to auth to Vault too
31        role: github-deploy
32        secrets: |
33          secret/data/production/myapp db_password | DB_PASSWORD ;
34          secret/data/production/myapp api_key    | STRIPE_KEY
35 
36    - name: Deploy
37      run: |
38        # DB_PASSWORD and STRIPE_KEY are available as env vars
39        # They are masked in logs automatically
40        ./deploy.sh

How this might come up in interviews

Secrets management appears in system design rounds (security layer of any cloud architecture), DevSecOps rounds (supply chain and pipeline security), and incident post-mortems (root cause: leaked credential).

Common questions:

How do you manage secrets in a Kubernetes environment?
What is the difference between static and dynamic secrets?
How would you prevent a developer from accidentally committing an API key to GitHub?
Walk me through how you would rotate a database password with zero downtime.
What is OIDC federation and why is it better than storing AWS keys in CI?

Strong answer: Mentions OIDC federation for CI, dynamic secrets with TTLs, audit logging requirements, and pre-commit hooks for secret scanning.

Red flags: Suggesting .env files in production, not knowing what secret rotation means, or thinking base64 = encryption.

Quick check · Secrets Management

1 / 4

Which of the following is the safest way to provide AWS credentials to a GitHub Actions workflow?

Key takeaways

A secret is any value that grants access — API keys, passwords, certificates, tokens, encryption keys.
Never store secrets in source code, .env files, Docker images, or environment variables passed via CLI.
Dynamic secrets (generated on-demand with a TTL) are far safer than static secrets — they expire automatically.
HashiCorp Vault (or AWS Secrets Manager, GCP Secret Manager) should be the single source of truth for all secrets.
Use OIDC federation in CI/CD to eliminate static cloud credentials from your pipeline configuration.
Rotate secrets automatically — dynamic credentials do this by design; static credentials need explicit rotation schedules.
Audit logs are non-negotiable — you need to know who accessed which secret and when for incident response and compliance.
Kubernetes Secrets are not a secrets manager — they are base64-encoded and require etcd encryption at rest.

Before you move on: can you answer these?

A developer accidentally pushes an AWS access key to a public GitHub repo. What are the first three things you do?

(1) Immediately revoke/delete the AWS access key in IAM — do not wait, assume it has already been found. (2) Check CloudTrail for API calls made with that key in the last 24–72 hours to assess blast radius. (3) Create a new key (or better, replace with an IAM role) and update all legitimate consumers. Then run truffleHog on the full repo history to check for other leaked secrets.

What is the difference between Vault's KV engine and its Database engine?

KV (Key-Value) stores static secrets you write manually — Vault stores them and you retrieve them. The Database engine is dynamic: Vault connects to your DB and generates a unique, time-limited username/password on each request, then automatically revokes it when the TTL expires. The Database engine eliminates long-lived DB credentials entirely.

Why is "encrypt the secret before storing in Git" not a good secrets management strategy?

The encryption key itself becomes the secret you need to manage — you have just moved the problem. You also lose audit logging (who decrypted it when?), automatic rotation, fine-grained access control, and lease-based revocation. It is acceptable for bootstrapping (sealed-secrets in K8s, SOPS for config files) but not a replacement for a proper secrets manager.

From the books

Hacking the Cloud (practical blog series, hacking.cloud)

Documents real-world AWS credential theft techniques used by attackers — understanding the attacker's playbook is essential for building effective defences.

HashiCorp Vault: Up & Running (O'Reilly)

Chapters 4–6: Secret Engines and Dynamic Secrets

Covers Vault architecture, auth methods, secret engines, and operational patterns. The most comprehensive resource for Vault implementation.

🧠Mental Model

💡 Analogy

A vault for your house keys

⚡ Core Idea

You would not leave your house key under the doormat (hardcoded in code), share it with everyone (broad IAM policies), or never change the lock (no rotation). A secrets manager is a guarded lockbox: you prove who you are, you get the key you need for the time you need it, and every access is logged.

🎯 Why It Matters

Credentials are the skeleton keys to your entire infrastructure. One leaked API key can mean full database access, data exfiltration, ransomware deployment, or fraudulent API charges — without any exploit of application logic.

Related concepts

Explore topics that connect to this one.

Ready to see how this works in the cloud?

Switch to Career Paths for structured paths (e.g. Developer, DevOps) and provider-specific lessons.

View role-based paths

Discussion

Questions? Discuss in the community or start a thread below.

Join Discord

Secrets Management

What Is a Secret?

The Secrets Anti-Patterns That Keep Getting Companies Breached

Secrets Vault Architecture: HashiCorp Vault Deep Dive

Dynamic Secrets: The Game-Changer

Secrets Injection Patterns in Kubernetes

Secret Rotation: Manual vs Automated

Secrets in CI/CD Pipelines

Which of the following is the safest way to provide AWS credentials to a GitHub Actions workflow?

Discussion

In-app Q&A

Secrets Management

What Is a Secret?

The Secrets Anti-Patterns That Keep Getting Companies Breached

Secrets Vault Architecture: HashiCorp Vault Deep Dive

Dynamic Secrets: The Game-Changer

Secrets Injection Patterns in Kubernetes

Secret Rotation: Manual vs Automated

Secrets in CI/CD Pipelines

Which of the following is the safest way to provide AWS credentials to a GitHub Actions workflow?

Discussion

In-app Q&A