Skeletons in the State Bucket: Identity is Permanent
Season Greetings
Since the summer, I have been helping a major client in the refactoring of their cloud application state, replacing implementations of resources and adapting to changing deployment contexts. As the leaves started to change colors outside and the days grew shorter, it began to feel like a plodding horror movie: Shadows of former references, lingering state, the previously destroyed resources popping back out of the ground – skeletons and zombies stalking the fleet.
I was puzzled. Why was Infrastructure as Code (IaC) so good at scaling up and down, straight-forwardly managing hundreds of compute resources, yet truly frightful at refactoring and migrating between architectural designs?
This blog is about my take on these reasons; about the limits of our typical tooling; and about some metaphors for thinking of these residuals in deployments.
And if you have not done any IaC in a while, or have no easy way to investigate some of these issues, this blog is still for you. I have included a little lab that demonstrates some of the concerns that you can experiment with. No cloud, no credentials, no services required.
The sharp edge of IaC
The sheer size of modern cloud applications makes failure a certainty rather than a risk. Where fleets of services collaborate for an optimal customer experience, something is bound to go amiss. Disks crash, motherboards fail, connections flake out, software falls behind the recommended patch level. What devops intended to be the case is not the case continuously. We say, the state drifts. And the purpose of IaC like kubernetes or terraform / terragrunt is to rectify that drift; to return the fleet to the desired state. IaC creates the illusion of stasis with continuous reconciliation. Because stasis is the backbone of every 24/7 cloud application.
This desired state is spelled out in code – hence infrastructure as code. Typically this is implemented as a set of files in a source code repository such as hosted git. CI/CD pipelines and other processes ensure that the latest state is reflected in the fleet. Resources that become unavailable are automatically replaced with functioning ones. Resources no longer needed are reclaimed. Resources that are missing are spun up.
The state of the cloud application consists of three types of information: addresses in the desired state that function as keys; internal IDs for the resources backed by the providers (GCP storage buckets, gitlab project IDs, Kafka queues); and application IDs for accessing those resources programmatically (GS URLs, gitlab paths, Kafka topics).
This state information is typically backed by cloud storage or a dedicated database. This is the colloquial “state bucket” from the title of this blog.
Conceptually, the simplicity of a desired state description is surprising. To move from the drifted state to the desired state is a state transition. That is a three-part relation: actual state, change, desired state. The descriptive power of IaC lies in two simplifications.
- IaC treats the actual state as not worthy of mention. The actual state either matches the intention – then it is redundant – or it is a short-lived aberration to be fixed. Hence there is no need to specify the actual state.
- During fixing, IaC infers the change necessary from the desired state. Hence there is no reason to specify the change.
These two simplifications reduce the devops specification problem to spelling out the desired state in the abstract in the source repository.
Reducing this three-way relation to a single component works great until it does not. And the situation where this modeling reduction falls flat is when the desired state itself undergoes transformative change. Not in the load-balancing sense of more VMs here or one database server less over there. Not in the additive sense of a caching layer or a bastion host. But in an architectural way, when one approach replaces another, e.g. Redis taking the place of Kafka for notifications. Because then we need to talk about two desired states at the same time: the previous desired state, with Kafka and no Redis; and the future desired state, with Redis and no Kafka. Which runs counter to the very simplifications that IaC makes.
Tools for Change
How then do the tools of the trade support such restructuring? In short: Not as well as we would like.
Terragrunt – Modeling two States Sequentially
Since each commit can only represent one desired state, classical terragrunt models such a replacement operation using sequential commits. How does that play out in our example?
Initially, only the “before” intention state using the Kafka notification exists. On the main branch, at commit A, the tree contains a terragrunt stack describing the Kafka resources. The state object for the resources' backend key holds the notification topic, say events. This commit A is the sole place where the old dependencies, inputs, and output names are described.
Suppose the team proposes to use Redis instead of Kafka in an architectural decision record (ADR). They fork a feature branch from commit A and rewrite the terragrunt stack to use Redis resources. Until merge, this is just a proposal. The branch’s history can be rebased freely, since nothing about the world has changed yet.
The merge request MR undergoes review. The MR diff is taken against the merge-base M, which need not be the moving tip of main. Commit range [M...H] shows the removal. Notice that a terragrunt plan without --filter-allow-destroy shows only a warning and no destroy plan (cf experiment e01 in our lab below), so the reviewer sees the removal in the diff but not the destruction in the plan. Call this the review gap.
The merge creates the commit B of the “after” intention state using the Redis notification. Main advances from commit A to commit B. On a push pipeline, the CI variables CI_COMMIT_BEFORE_SHA (A) and CI_COMMIT_SHA (B) name the transition. The reason for the change lives outside both trees, in the merge commit message or MR description, possibly with a reference to the architectural decisions record (ADR).
Given the merge to main, the CI/CD pipeline performs the transition via something like:
terragrunt run --all apply \
--filter '[A...B]' --filter-allow-destroyThis means that terragrunt materializes a temporary worktree at commit A. Terragrunt finds units present at A and absent at B, and destroys them from A’s worktree, in reverse dependency order, drawing upon real outputs from A’s backend state. Terragrunt then plans and applies the changed and added units from B’s worktree. One run moves in both temporal directions, the past and the future, and the two halves don’t gate each other – with all the dangers that this implies.
Finally, the state is updated to reflect the physical outcome. The resources backing the units are destroyed, and the state object for the Kafka configuration typically remains without resources in it. The flesh is gone, and only the skeleton survives in the bucket.
The only record that this process succeeded is the green light in the CI/CD pipeline. The forge will hold that outcome, but no one else does. The repository has the commits A and B, and their linkage, but the modifications are in the state. Neither however is connected with the intention as captured by the architectural decision record that caused this state change in the first place. Thus the full state transition is not modeled properly.
Yet, the walk-through shows clearly why that full transition is necessary, why both ends of the transition must be modeled, not just the final desired state – IaC’s default. Terragrunt cannot destroy the old resources without A’s tree, because the destroy needs both the dependency graph and the outputs. The mechanics require them, and the commit sequence is the only place both survive.
Moreover, notice the fragility. The transition is identified by a pair of commit hashes on a first-parent line of main. Any rebase, force-push, or squash that moves A or B potentially changes the transition being executed. The malleability of the repository does not help to preserve an accurate record here. And since the intention record is not part of the merge that established the commit pair, nothing ties the executed transition to its justification, except perhaps convention.
Terraform 1.7 – Modeling removed Resources
Terraform 1.7 introduced the removed block, presumably to pull the description of the removal out of the commit sequence and put it into the terraform description directly. Spoiler: This solves the branch fragility and the review gap, but it does not close the knowledge gap. Let’s walk through the example again with updated tooling to appreciate why this is so.
Again, in the beginning, only the “before” intention state using the Kafka notification exists. On the main branch, at commit A, the tree contains a terragrunt stack describing the Kafka resources. Assuming the popular mongey Kafka provider, there exists a resource under the address kafka_topic.events in the state object of the backend.
Next the team records the intention in an ADR and authors a merge request. A branch forking from A contains the updated terragrunt stack. The MR does contain the Redis resource definitions, but that is not the focus here. More importantly, the MR removes the Kafka event topic resource block and replaces it with a removal block.
removed {
from = kafka_topic.events
lifecycle {
destroy = true # false would only forget it
}
}Because the MR describes the change in a single commit, the terraform plan can show the destruction properly. This is a huge improvement, as it closes the review gap at plan approval time.
The merge then creates the new commit B. The single commit adds the removed block but deletes the resource description block for kafka_topic.events. The two blocks remain linked through the from key, which forces a check against the state.
The CI/CD pipeline can now act upon the merge. Given that all the information is present and visible to the planning, It needs only B’s worktree. Terraform matches the state address against the removed block, destroys the resource, and drops the address from the state object in the backend. Since no temporary worktree at A is needed, no sequence [A...B] is required, just the commit B. Rebasing a single commit does not change the transition intended.
Unsurprisingly, the state records the outcome, removing the address from the state.
What about the removed block? This is where the drama starts. A block in the code has now become the tombstone. The knowledge of that deletion resides not in the state, but in the description. This description is not only default-destroy; it also lives in the rewritable history of a repository. Cleanup that lives in the resource block is tied to it: a bare tombstone takes the provisioner down with it, leaving the flesh behind (e07 s1).
And that tombstone needs to persist until every state that holds that address has applied it. If someone removes it after the main environment applies but before a lagging one does, the laggard environment sees an address with no config and no tombstone, so terraform will plan to destroy – the default – against the declared purpose of the tombstone (e07 s3).
Once all environments have run though, and the tombstone is cleaned up, any old branch can happily recreate the resource. The state only remembers what exists; not what has been destroyed.
In the lab, a surviving environment that never applied the destroy plans a second create beside the resource it still runs (e07 s4) – though most real providers would probably flag the duplicate with “already exists”. Nevertheless, this underscores that the only record of retirement is in the tree, however focused in a single commit, but critically not in the state.
Discussion
Those that pass on leave something behind. Yet our tooling gives those leftovers no home.
As our drawing showed, terraform’s state holds three kinds of information: addresses, which the config uses as keys; internal IDs, which the provider uses; and application IDs, which other systems use to reach the resource. Only the first two are visible to the tooling. Application IDs are the names that other things bind to such as the Kafka topic string producers write to. They are allocated by the same code that declares the flesh, but nothing records that they are identities rather than implementation. When the flesh is removed, a removed block says nothing about them. Some are separate resources and survive, pointing at an endpoint that no longer exists. Others are module inputs and vanish with the module; the first consumer to resolve them fails with no Terraform error. The fate of these identities depends on how the code happened to be structured.
The natural place for maintaining that record is the architectural decision record. An ADR says why Kafka gave way to Redis, which identities are retired, and which are carried over. It is the only artifact that holds the rationale for the change of intention. But ADRs sit outside the toolchain. The IaC cannot use them as a witness to guard the state.
The obvious patch is to make the tree carry more of the record. A removed block per retired address, kept until every environment has applied it, does capture the retirement in the commit that introduces it. But it scatters the record across the code, one tombstone per resource, each with its own lifetime. The result is a forest of blocks, and obfuscates how the system evolved as a whole. Teams will reopen the zombie door just to declutter their view – a scary tradeoff.
So identity has no first-class home. The state knows the machine IDs and addresses but not the names the code depends on. The tree knows the names, but only as a rewritable snapshot. The ADR knows the reason, but the tooling cannot query it.
No wonder homeless identities prowl Halloween night.
Bonus Round 1: The Metaphor Map
When discussing these issues in a team, it is convenient to have evocative metaphors for these complex processes and situations.
| Halloween | Infrastructure | Notes |
|---|---|---|
| Skeleton | Identity: the names and IDs other systems bind to, the state key, grants and roles | Outlives the flesh. Never reuse the bones. |
| Flesh | The resource itself: the Kafka cluster, its brokers and volumes | What destroy removes. |
| Front-yard tombstone | State-based deletion: a removed block | Visible, and it stays until someone takes it down. |
| Spectre | Orphaned state: event-based deletion whose moment has passed (e04 S1) | Nothing in the config points at it, but its resources live on. |
| Ghost | Orphaned config without backing | The config points at it, but the resource is already gone; term taken from Firefly |
| Zombie | Resurrection: a destroyed unit brought back by a stale branch (e04 S3b), or a new product copying old identities | “Zombie resources” is already an industry term. We give it two backstories. |
| Poltergeist | Lost destroy-time side effect: a provisioner or cleanup step that only ran while its config existed (e07 S3) | The object is gone, but what it did is still happening. Nothing in state or the repo records it. |
| Halloween (one night the veil is thin) | The deliberate retirement run | --filter-allow-destroy is the gate, opened on purpose and then closed. |
| Haunted house | (1) A run where the dead and the living move in opposite directions and get stuck (e03). (2) A repo whose dependency graph lives in three places | |
| Spectre hunt | A sweep for state that no config and no tombstone accounts for (e06); won’t find poltergeists as they have no config | A level-triggered safety net. |
Let’s play it through for our guiding example. The Kafka-to-Redis story maps directly onto the palette. The Kafka cluster qua cloud resource is flesh. The topic name events and its consumer-group offsets are skeleton. The removed block is the tombstone. A run that destroys Kafka while creating Redis is a haunted house. If the destroy never happens and no later run’s range mentions it, the Kafka state is a spectre. If the Kafka cluster is removed outside of the CI/CD pipeline, the configuration in the state is a ghost. An old branch that still declares the events topic and applies it is the zombie.
Bonus Round 2: The Lab
The companion lab for this blog is available on Posedio’s GitHub; the details are in the README.md.
The lab exists so that every behavioural claim in this post can be checked. It has two tiers. The core experiments (e01 to e07) need only shell tools. They run with make in 1-2 minutes, with no cloud, no credentials, and no services. Since e03 demonstrates a stuck state via timeout, so it takes about 40s on its own.
The backend experiment (b01) needs a local S3-compatible server that the reader starts. It runs only with an explicit make backend; it refuses anything but localhost; and it forces dummy credentials, so lab state cannot reach a real account.
Each experiment records two kinds of results. A claim is documented behaviour that may still surprise you. An observation is undocumented behaviour that the lab has recorded. The results were observed with Terragrunt v1.1.3 and Terraform v1.14.8 on macOS arm64. The lab is the evidence for the argument. It does not replace the argument.
References
- •
- •
- •
- •
- •
- •
- •
- •
- •
- •
- •
- •
- •
- •
- •
- •
- •
- •
- •
- •
