how/state

Remote backend and lock: how two engineers work with one state

S3 stores the state, DynamoDB holds the lock. What happens when two people run apply at the same time, and why a "team of one" is the only case where you can live without this.

A local terraform.tfstate next to main.tf works fine while you are alone. The moment a second person joins the team, a remote backend is mandatory. Without one you either commit the state to git and edit it inside merge conflicts (a disaster), or you overwrite each other's changes (a quiet disaster).

The de facto standard on AWS is S3 + DynamoDB:

S3 stores the state file itself, plus encryption and versioning.
DynamoDB holds the lock, a tiny table where Terraform writes "I am working right now".

Press ▶ to walk through the scenario where two people run apply at the same time.

step 1/5·00 · two engineers, one state

§ steps

Alice and Bob work with the same infrastructure. They run terraform apply from their own laptops, but the state is stored not on them, but in the team's S3 bucket.

DynamoDB is empty for now: nobody is working. The state in S3 has serial: 10 (that is, 10 successful applies have already been made on it by someone on the team).

recap

What to remember:

A lock is not a database transaction, it is just a record in DynamoDB: a PutItem with the condition "the key does not exist". Whoever gets there first takes the lock; the second one gets ConditionalCheckFailed and sees "Lock held by …".
There is a TTL on the lock in Terraform Cloud / Atlantis, but in plain DynamoDB there is none. If terraform apply dies on the network, the lock stays held until someone runs terraform force-unlock <LockID>. This is a normal procedure, not a reason to panic.
Every apply increments serial in the state. Before writing, the backend checks that serial has grown. This is protection against concurrent overwrites (even without a lock).
GCS has a built-in lock (no separate table). So does Terraform Cloud. So does Azure Storage. Only S3 needs DynamoDB on the side.
Do not confuse a lock with state encryption. A lock protects against concurrent writes; encryption protects against reads by outsiders. You need both.

Next: [[tf-state-mv-rm-import|how to move resources by hand inside the state file]].

§ dig into the knowledge base

§ try it hands-on

§ steps

Alice and Bob work with the same infrastructure. They run terraform apply from their own laptops, but the state is stored not on them, but in the team's S3 bucket.

DynamoDB is empty for now: nobody is working. The state in S3 has serial: 10 (that is, 10 successful applies have already been made on it by someone on the team).