Secrets management: Vault, k8s Secrets, sealed-secrets

Why secrets need their own infrastructure

A "secret" is a value that nobody should see except those who must. Database passwords, API keys, TLS private keys, OAuth client secrets.

Ways to do it wrong:

• In code (API_KEY = "sk-..."), it lands in git history, in code review, and leaks forever (still visible in history even after a revert)
• In a .env committed to git, the same problem, plus it accidentally ends up in the Docker image
• In a Dockerfile via ENV, visible in docker history, baked into the image
• In CI as a plain variable, it shows up in the CI log and build artifacts
• In a k8s ConfigMap, base64 is not encryption, kubectl get cm reveals it

What you actually need:

1. The secret is stored encrypted
2. Access is authenticated (you know who asked)
3. Access is authorized (this identity is allowed)
4. Every request is logged (audit trail)
5. Rotation is possible without restarting the application
6. Secrets do not live in git (not even encrypted, when you can avoid it)

Level 0: env vars from the orchestrator

The simplest approach: set an env var through docker-compose, systemd, or k8s, with no dedicated store.

# docker-compose.yml

services:

  app:

    image: myapp

    environment:

      DB_PASSWORD: ${DB_PASSWORD}     # from .env (NOT in git!)

Downsides: rotation means a restart, there is no audit, and the secret is visible in docker inspect. This approach fits local dev and small pet projects. For production it is not enough.

HashiCorp Vault, the general-purpose option

The most mature secrets manager. It runs as a REST service plus an encrypted backend (etcd, raft, S3).

Key concepts:

• Secrets engine, what to store (KV, database, AWS, PKI, transit)
• Auth method, how a client authenticates (token, AppRole, kerberos, k8s SA, AWS IAM, OIDC)
• Policy, who is allowed what (an HCL policy on a path)
• Audit device, where the audit log goes

# Write a secret

vault kv put secret/app/db password="s3cret" username="myapp"

# Read it

vault kv get -field=password secret/app/db

Dynamic secrets, the killer feature

Instead of storing a database password, issue a short-lived one on the fly:

vault write database/config/postgres-prod \

    plugin_name=postgresql-database-plugin \

    connection_url="postgresql://{{username}}:{{password}}@db:5432/" \

    allowed_roles=readonly \

    username=vault-admin password=...

vault write database/roles/readonly \

    db_name=postgres-prod \

    creation_statements="CREATE USER \"{{name}}\" WITH PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \

                        GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \

    default_ttl=1h max_ttl=24h

# client:

vault read database/creds/readonly

There are no long-lived credentials, so there is nothing to rotate. Each pod gets its own unique ones. If one leaks, the ttl expires within an hour.

Transit engine, encryption as a service

Vault encrypts and decrypts while the key never leaves Vault:

vault write transit/encrypt/app plaintext=$(echo "secret" | base64)

vault write transit/decrypt/app ciphertext="vault:v1:..."

This is useful for encryption-at-rest without managing KMS keys in your code.

k8s Secrets, native but handle with care

apiVersion: v1

kind: Secret

metadata: { name: app-db }

type: Opaque

stringData:

  password: s3cret                    # k8s base64-encodes it for you

The pod uses it:

spec:

  containers:

  - name: app

    env:

    - name: DB_PASSWORD

      valueFrom:

        secretKeyRef: { name: app-db, key: password }

Pitfalls:

• base64 is not encryption, kubectl get secret app-db -o yaml | base64 -d exposes it to anyone with the get secrets permission
• By default it sits in etcd in plaintext! Turn on encryption-at-rest through EncryptionConfiguration (a KMS key or an in-file AES key)
• RBAC on secrets is separate. Do not grant cluster-reader automatically (it often includes secrets)
• Mount as a file rather than an env var, a file is preferable (less leakage through /proc/<pid>/environ)

sealed-secrets, secrets in git

If you run GitOps (ArgoCD or Flux), you want everything in git, including Secrets. But a Secret is base64 in the open, which is no good.

The fix is sealed-secrets (Bitnami):

• A controller in the cluster holds a key pair
• The kubeseal CLI encrypts a Secret with the public key into a SealedSecret CRD
• The SealedSecret is committed to git
• The controller decrypts it and creates the real Secret in the cluster

kubectl create secret generic db -n prod --from-literal=pwd=s3cret \

  --dry-run=client -o yaml | \

  kubeseal -o yaml > db-sealedsecret.yaml

git add db-sealedsecret.yaml && git commit

Only this specific cluster can decrypt it (the private key lives only there). If git leaks, the secret stays protected.

The downside: if the cluster's private key is lost, all sealed-secrets become unrecoverable. Backing up the key is critical.

external-secrets, a bridge to a cloud vault

If you use AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, or Vault, the standard CRD operators apply:

apiVersion: external-secrets.io/v1beta1

kind: ExternalSecret

metadata: { name: app-db }

spec:

  refreshInterval: 1h

  secretStoreRef:

    name: aws-secrets

    kind: ClusterSecretStore

  target: { name: app-db }

  data:

  - secretKey: password

    remoteRef:

      key: prod/app/db

      property: password

Every hour the operator goes to AWS Secrets Manager, pulls the value, and updates the k8s Secret. The source of truth is the cloud vault, and the k8s Secret is a cache.

Bonus: rotating the secret in AWS automatically updates it in k8s.

CSI driver for secrets (Secrets Store CSI Driver)

An alternative to external-secrets: mount the secret straight into the pod as a file, with no intermediate k8s Secret:

volumes:

- name: secrets

  csi:

    driver: secrets-store.csi.k8s.io

    readOnly: true

    volumeAttributes:

      secretProviderClass: app-secrets

Upsides: it never sits in etcd, and rotation happens without a restart (with enableSecretRotation).

Rotation, the main headache

Storing a secret is the easy part. The hard part is updating it everywhere it is used without downtime.

Approaches:

• Restart the pod when the Secret changes, highly undesirable for stateful workloads
• The app watches the secret file, the pattern used with the CSI driver
• A sidecar reloads it (Vault Agent injector plus sigtemplate)
• Dynamic secrets, the update is automatic via TTL

Never assume a secret lasts forever. Rotate at least once a year (compliance: SOC 2 and PCI require it).

Audit and compliance

Any production vault must log every read and write with:

• the client identity
• which secret path
• a timestamp
• the source IP

Vault: vault audit enable file file_path=/var/log/vault/audit.log. AWS Secrets Manager: CloudTrail. k8s Secrets: an [[auditd|audit policy]] on the apiserver.

During an incident the audit log is the only thing that shows who read a secret and when.

What not to store in secrets

• User password hashes, those belong in the database, not in a secret
• Sessions and JWT tokens, use a separate store (Redis)
• Public data (URLs, env flags), use a ConfigMap, not a Secret

Putting everything in a Secret means a larger attack surface and a harder audit.

When things go wrong

• Error: secret not found in a pod, the Secret is in another namespace, or the ServiceAccount lacks RBAC to read this Secret.
• Restoring from an etcd backup shows the old secret: encryption-at-rest used a KMS key, but the KMS key is lost. The backup is useless. Back up the KMS key too.
• pod cannot read secret file after rotation, the application cached the old file handle (a cached open). Fix: a SIGHUP handler or a sidecar watch.
• sealed-secret invalid, the controller regenerated its key (a new cluster). The old one cannot be decrypted, so regenerate from scratch.
• Vault is sealed after a restart, Vault starts sealed (an HA feature). You need vault operator unseal with N of M shamir keys (or auto-unseal through a cloud KMS).
• The secret appears in application logs, the app prints the environment at startup (debug). Never do this. Lint for it in CI: grep -i "password\|secret" app-logs.
• .env committed to git, run git filter-repo --invert-paths --path .env, then ROTATE all secrets, because git history remembers forever.