How secrets end up in state
Every apply writes resource attributes to state, including computed values returned by the provider. Common examples:
aws_db_instance.main.password: if you supplied the password, it is in state.aws_iam_access_key.ci.secret: the secret key sits in state in plain text.random_password.db.result: a generated password is stored in state in plain text.aws_secretsmanager_secret_version.x.secret_string: if you read it via a data source, the current value also ends up in state.
The backend owns state, whether that is a local file or S3. Anyone with access to the file can see the secret. "sensitive=true" changes only how Terraform displays the value in CLI output and logs. In state the value is always plain text.
This means protecting secrets is the same problem as protecting state.
Backend encryption
Minimum requirements for production:
- S3 + SSE-KMS. State lives in an S3 bucket; a KMS key encrypts at the
object level. Without
kms:Decrypton that key you cannot read state, even if you haves3:GetObject. - Versioning and MFA delete. This prevents the "I accidentally broke state and have nothing to roll back to" situation.
- Bucket policy: block public access, allow only specific IAM roles or users.
resource "aws_s3_bucket" "tf_state" {bucket = "company-tf-state"
}
resource "aws_kms_key" "tf_state" {description = "tfstate encryption key"
deletion_window_in_days = 30
enable_key_rotation = true
}
resource "aws_s3_bucket_server_side_encryption_configuration" "tf_state" {bucket = aws_s3_bucket.tf_state.id
rule { apply_server_side_encryption_by_default {kms_master_key_id = aws_kms_key.tf_state.arn
sse_algorithm = "aws:kms"
}
}
}
See tf-remote-backend-s3 for a complete backend setup.
Keep real secrets out of HCL
Anti-pattern:
variable "db_password" {default = "supersecret" # password in code = password in git
}
Anti-pattern 2:
TF_VAR_db_password="supersecret" terraform apply
This leaks if shell history records environment variables or if CI logs expose them.
The canonical pattern is to use an external secret manager.
AWS Secrets Manager
data "aws_secretsmanager_secret_version" "db_master" {secret_id = "prod/db/master"
}
resource "aws_db_instance" "main" {# ...
password = data.aws_secretsmanager_secret_version.db_master.secret_string
}
The secret is created outside Terraform, by someone in the AWS Console or through a separate provisioning flow. Terraform reads it; it does not create it. The current secret value will appear in state (again, plain text in the state file), so the backend must be encrypted.
HashiCorp Vault
provider "vault" {address = "https://vault.internal:8200"
}
data "vault_kv_secret_v2" "db" {mount = "kv"
name = "db/master"
}
resource "aws_db_instance" "main" {password = data.vault_kv_secret_v2.db.data["password"]
}
Authentication to Vault is a separate topic: JWT/OIDC, Kubernetes service accounts, AppRole. The Terraform runner needs a way to authenticate to Vault.
Generate and store
An alternative: Terraform creates the secret and stores it in a manager.
resource "random_password" "db" {length = 32
special = true
}
resource "aws_secretsmanager_secret" "db" {name = "prod/db/master"
}
resource "aws_secretsmanager_secret_version" "db" {secret_id = aws_secretsmanager_secret.db.id
secret_string = random_password.db.result
}
resource "aws_db_instance" "main" {password = random_password.db.result
}
The downside is that random_password.db.result is in state. The upside is
that everything is declarative and rotation is a single
apply -replace=random_password.db. The right choice depends on your team's
threat model.
OIDC: no more access keys for the Terraform runner
The most common "secret" is the credentials for Terraform itself:
AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY. Teams put these in CI secrets,
rotate them infrequently, and the keys eventually leak through logs and build
caches.
The modern answer is OIDC: the CI runner gets a temporary AWS token via federated trust, with no long-lived key involved.
Scenario with GitHub Actions:
-
In AWS, create an IAM role that trusts the GitHub OIDC provider.
-
The trust policy restricts access to a specific repo and branch:
json{"Effect": "Allow",
"Principal": { "Federated": "arn:aws:iam::ACCOUNT:oidc-provider/token.actions.githubusercontent.com" },"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {"StringLike": {"token.actions.githubusercontent.com:sub": "repo:linuxlab/terraform-infra:ref:refs/heads/main"
}
}
}
-
In the workflow:
yamlpermissions:
id-token: write
contents: read
steps:
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::ACCOUNT:role/github-tf-runner
aws-region: us-east-1
The runner receives temporary credentials valid for one hour. No access keys in Secrets. See tf-oidc-aws for the full setup.
Other values that land in state as secrets
These are not always obvious:
user_dataonaws_instance. If it contains a hardcoded password, that password is in state. For bootstrapping with secrets, use cloud-init together with a secret store (the instance reads Secrets Manager on startup).environment.variablesonaws_lambda_function. Every environment variable is in state. For secrets, have Lambda read Secrets Manager instead.aws_ssm_parameterof typeSecureString. The value is in state, encrypted only in SSM. Decrypting it in HCL means it is plain text in state.tls_private_key.this.private_key_pem. A private key generated by Terraform is stored in state in plain text. The alternative is to generate the key withssh-keygenbeforehand and upload only the public key.
Secret rotation
- External secrets in a manager. Rotation is handled by the manager. Terraform picks up the new value on the next apply because the data source refreshes.
- Secrets generated in Terraform. Rotation is
terraform apply -replace=random_password.X. This also recreates every resource that depends on the password (the database with a new password, with downtime). - Always take a state backup before rotating. If something goes wrong, you have a rollback point.
Pitfalls
-
"sensitive=true" is not encryption. It is redaction for CLI output. See tf-sensitive.
-
.tfstate.backupalso contains secrets. If you encrypt state, encrypt the backup too. On S3, encryption by default covers all objects, including backups. -
terraform show -json plan.tfplanexposes sensitive values. A plan file passed as an artifact between jobs spreads secrets. Keep retention short, encrypt the transfer, and delete the file after apply. See tf-plan-apply-ci. -
An OIDC role with an overly broad trust policy.
repo:org/*:*lets any repository in the organization assume the role. Bind the trust to a specific repo and ref. For a production role, also tie it to a GitHub Actions environment with required reviewers. -
KMS encryption is not the same as in-state encryption. An S3 backend with KMS encrypts the file at rest. When Terraform reads the file, it decrypts it and holds the contents in memory in plain text. KMS protects against "someone stole the S3 object," not against "someone has an IAM role that can read this state."
-
Secrets Manager costs money. You pay per secret. Do not create thousands of secrets, one per small value. Group related values into a JSON object.
-
Vault requires ongoing maintenance. It is a stateful service, typically a cluster of three to five nodes that you must deploy, upgrade, and operate. For ten secrets, AWS Secrets Manager is simpler. At around a hundred secrets, or when you need cross-account access, Vault starts to pay for itself.
See also in LinuxLab
- file-permissions: a
terraform.tfstatewith permissions 0644 is readable by any user who can run sudo. It should be 0600. The same applies to*.tfvarsfiles and lock files in a remote backend. - secrets-management: how secret issuance and rotation work at the Linux level (systemd-credentials, secret-tool, vault-agent). Terraform is a consumer; the infrastructure described in that article is the source.
- setuid-setgid-sticky: the habit of checking permission bits on directories where Terraform writes state and log files. A setuid bit on a CI agent is a classic attack vector.