TF_LOG: Terraform diagnostic logs

When to turn on TF_LOG

By default Terraform says very little. A plan is a couple dozen lines, an apply lists the resources it created and a summary. When something fails, you get a short error message and that is all.

In three situations that is not enough:

• Apply hangs. It does not fail with an error, nothing just happens for minutes on end. Where is it stuck right now? On which resource?
• A strange error from the provider. "InvalidParameter": which parameter? Which API call? With what data?
• A suspicion that something is off with dependencies. Why does resource A get created before B?

For all of this, reach for TF_LOG.

Levels

There are five levels, from loud to quiet:

Almost every time you want DEBUG. TRACE produces megabytes of output that you cannot read without grep.

Basic use

Linux/macOS:

TF_LOG=DEBUG terraform apply

Windows PowerShell:

$env:TF_LOG="DEBUG"

terraform apply

Windows cmd:

set TF_LOG=DEBUG

terraform apply

The output gets an order of magnitude longer right away, with every internal step printed to stderr.

TF_LOG_PATH: save to a file

DEBUG-level logs run to hundreds of lines per resource. You cannot read that in the terminal. The fix:

TF_LOG=DEBUG TF_LOG_PATH=/tmp/tf.log terraform apply

After the apply, /tmp/tf.log holds the whole debug output. Open it in an editor or with less/grep.

The file is appended to, not overwritten. Several applies in a row and the logs run together. If you want a clean file, run rm /tmp/tf.log before you start.

TF_LOG_PROVIDER and TF_LOG_CORE

Sometimes you want logs from the provider only (the AWS plugin), without Terraform's internals:

TF_LOG_PROVIDER=DEBUG terraform apply   # providers only

TF_LOG_CORE=DEBUG terraform apply       # core only

TF_LOG=DEBUG terraform apply            # everything

This cuts the volume sharply when you are debugging AWS API problems.

What to look for in the logs

HTTP requests to the cloud

---[ REQUEST ]-------------------------------------

PUT /linuxlab-hello-abc/ HTTP/1.1

Host: s3.amazonaws.com

Authorization: AWS4-HMAC-SHA256 ...

Each resource Terraform creates is one or more HTTP requests. When an apply hangs, the last REQUEST without a response shows you where.

Error responses

---[ RESPONSE ]------------------------------------

HTTP/1.1 400 Bad Request

...

<Error>

  <Code>InvalidBucketName</Code>

  <Message>The specified bucket is not valid.</Message>

</Error>

This is the original message from the cloud, before Terraform filters it. Sometimes it is clearer.

Dependency resolution

[DEBUG] ReferenceTransformer: "aws_s3_bucket.demo" references: [random_id.suffix]

[DEBUG] Starting graph walk: walkApply

You can see the dependency graph at work. If you expect A to depend on B and the logs do not show it, the implicit dependency did not fire and you need depends_on.

Acquiring/Releasing lock

[DEBUG] Acquiring state lock. This may take a few moments...

[DEBUG] backend/local: state lock acquired

If an apply hangs on "Acquiring state lock", someone else is holding the lock. See terraform force-unlock, but use it with care.

Useful filtering patterns

HTTP requests only:

TF_LOG=DEBUG terraform apply 2>&1 | grep -E '^(PUT|GET|POST|DELETE)'

Errors only:

TF_LOG=DEBUG terraform apply 2>&1 | grep -iE '(error|fail|denied)'

Requests and responses for a specific resource type:

grep -A 5 's3' /tmp/tf.log

Turn logging off

unset TF_LOG TF_LOG_PATH        # Linux/macOS

Remove-Item Env:\TF_LOG          # PowerShell

Or just open a new terminal session.

Pitfalls

• TRACE produces gigabytes on a medium project. Do not run TRACE without TF_LOG_PATH, or it floods the terminal.

• The logs contain secrets. API keys in HTTP headers, database passwords in request bodies. Do not send raw logs to HashiCorp issue trackers without redacting them first.

• TF_LOG does not help with broken HCL. When the syntax is bad, Terraform fails before it runs. Debugging is for runtime errors.

• DEBUG in CI slows the build. It writes a lot to stderr and clutters the job logs. Turn it on only when that job fails, not across all of CI.

• Provider logs are in English. There is no localization, even if the TF CLI runs with LC_ALL=ru_RU. Every error from the AWS API comes back in English.

• You need the 2>&1 redirect for most filters. TF writes debug output to stderr and the main output to stdout. Without 2>&1, grep finds nothing.

See also in LinuxLab

• file-descriptors: why 2>&1 must go AFTER the command and before the pipe, not just anywhere; what stderr/stdout are as file descriptors.
• cmd-journalctl: if you run terraform under a systemd timer, the TF_LOG_PATH logs go to a file while the rest goes to the journal. A handy baseline for long-running pipelines.
• bash-scripting: wrapping TF_LOG=DEBUG … in a small utility script is cleaner than setting the env variables by hand every time.