"It Worked Last Week": Catching API Drift Before It Breaks Production
Introduction
Recently, I sat in a meeting with clients. There were discussions about interfacing with services and APIs from other parties in their large company. One of their developers – let’s call him Frustrated Fred – griped about the shifting sands that made integration work so complicated.
“I know this worked two weeks ago. And now it doesn’t.”
And he followed up with that perennial bogeyman of cloud computing,
“Something must have changed in the permissions.”
Indeed, these are our rapidly morphing working environments. Literally hundreds of teams are scrumming through their Kanban boards every week, releasing – equally literally – hundreds of tweaks and twists to the surfaces of their offerings. All of these surfaces are built on top of libraries and components that themselves grind through version numbers as if that were their principal raison d’etre. In such flux, experimentation means little, and experience perhaps even less.
I have been fascinated by technical risk assessment for a long time, and this is an exciting and challenging environment. This blog post is my analysis of how we got here; what that pressure does to us, emotionally; why our current solutions have failed us; and how we can recombine their strengths into new tooling that can help stabilize these experiences.
If my hunches are proximate, my pal Frustrated Fred would have found out within 24h that his expectations had been thwarted by some well-meaning action – not two weeks later when it mattered. Perhaps that frustrated colleague will even read this blog post?
If so: Bro — this one’s for you!
How we got here
Faced with the number of machines, services, and software components, enterprise cloud computing had to shift away from imperative conceptualizations. Some disk, some fan, some installer was always going to crash. Large assemblages of parts experience drift between how the world should be and how the world is, non-stop.
Infrastructure as code, as realized in Terraform/OpenTofu (TF) or Kubernetes Operators, allowed cloud engineers to talk about what they wanted. The focus shifted to the “should-be” world, rather than how to make that happen. Intentions replaced actions as the unit of description. Resources rather than VMs or permissions or accounts became the units of conceptualization.
Except … eventually, some agent has to do something. We can push the actions down the stack, away from the design documents and the terraform code. But somewhere – as the Americans say in their car-fixated way – the rubber of the tires has to hit the road.
And what we learned, often the hard way, was that the realities of the APIs were neither that stable nor all that straight-forward to use anyway. Our neat stacks of architectural decision records and validated terraform HCL files would not stand straight on the slippery surfaces of these tricky integration points; points which we needed to provision our cloud vision.
The Shame of Misunderstanding
I vividly recall the chagrin of having placed my wager on that API, through two design drafts, only to find out at TF module writing time that the default settings were blowing up in my face.
This is doubly bad: not only embarrassing, but also incentivizing bad impulses. Why indeed write dozens of design document versions, incorporating feedback from parties and review meetings, when none of this captures how the API actually works? Sure; this is the gremlins talking to lead us astray. And it is a criticism we need to hear.
People avoid pain; disappointments push us to go to implementation faster. Sure, we will make it a feedback loop! “We are just discovering how things really work; then we will update the design document with the latest insights! Deal?”
The validity of the stance is that technical risk assessment is necessary. The problem remains that our typical solutions have proven inadequate to the task. My tack is that their capabilities are badly matched to the problem at hand.
Why our typical solutions fail us
That our typical solutions have failed us is evident from my writing this blog post. The more interesting question is how that came about. I will start with my toolbox for investigating technical risk. In my box I find: the prototype, the tracer bullet, the example and of course that old stalwart, experience. Let’s take them in turn.
The Prototype
The prototype implements the whole product for the sake of experience, then writes up the design document and starts over. Officially at least; criticism of the prototype, reaching all the way back to Fred Brooks and his Mythical Man Month, is older than the original Star Wars movie.
Prototyping without design can be an eye-wateringly expensive and time-consuming way to get an inferior product. But that is not our concern here. In fact, I will gladly admit that prototyping explores the risk surface exhaustively; how could it not? Prototyping will also stay abreast of any shifts in the risk surface, as long as it is in development mode. These are real strengths.
The problem for technical risk assessment is the massive overkill. Prototyping does so much more than establish a risk surface. It is a bit like bringing a battle cruiser to cross the Danube near Spitz in the Wachau: simply too much. And because of that “too much”, it is not sustainable for long either.
The Tracer Bullet
I always think of the tracer bullet as an attempt to reap the benefits of the prototype without getting stuck in its tar pit – to wit, the building of an exact replica of the final product.
The focus of the tracer bullet is that exemplary trajectory: one flow of data through a product. The tracer bullet chooses an interesting set of transition points and then implements just enough – often through stubs – to make that work.
Again, this will get a realistic assessment of the risk surface for what is covered in the data flow. And it will do so for a fraction of the price of the replica prototype.
Wherein lies the rub. That fraction is proportional to the coverage of the data flow. If the path is too focused – and thus more affordable – the risk surface is insufficiently sampled. And since it is not even a replica of the product, it won’t stick around for long.
So while cheaper compared to the prototype, it is still too much for too little information.
The Example
The example takes a radical shift by atomizing the problem space and focusing on one surface. The example then shows one use or at most a simple CRUD sequence.
This approach cleanly distinguishes between the risk assessment process and the overall design process. Each example can be tackled on its own terms. Examples can be pursued in parallel – a great use of team resources. It also stops distracting from the design process: once it runs, the job is done and attention loops back to the documentation.
So what are the drawbacks? Most importantly: examples to teach an API have pedagogical intent. They make simplifying assumptions to minimize distractions. They use any halfway sensible default that makes the example succeed. In short: the example distorts what the product will in fact need.
For that same reason, examples use toy data; nothing like the stuff the finished product needs to handle, if the user stories are any indication.
I know this because this is exactly what happened to Frustrated Fred and me. Fred was overbooked and needed some help to get a terraform module checked off. I had a blank spot on my calendar and offered to write him a minimal implementation that would showcase the API call in question.
Suffice it to say, when Fred remained buried and the still-open task circled back to me, I quickly realized how unhelpful my trivial example had been. Other than some tutorial, I had no excuse for assuming I knew what Fred needed. My “minimal solution” when beefed up to production level ended up repeatedly crashing the CI/CD jobs because of a toxic default that my code never tickled in isolation. It took version exploration of two major versions (it crashed in all of them) and Go-level debugging of the API provider to figure out what the realistic case should have looked like!
Experience
We all rely on experience to help us architect solutions and evaluate designs. But in the context that this blog post is focussing on – tracking the technical risk surface of a product – experience comes across as the legend of an example that was run somewhere & somewhen. Experience serves as an existence proof that this worked once. But it fails to address the question of version drift. And it would be pure luck if the data of the current user stories had the same shape and requirements as the data did back then.
Finally, because it is often just a story, not the software itself, one cannot simply rerun it.
Discussion
Clearly there is a place for prototypes, tracer bullets, examples or experience. What I am arguing is that, even when handled well, these tools solve the wrong problem. They are not useful for mitigating the surface risks that products face in the modern drifting software landscape.
The reason these tools have been used to explore the risk surfaces on which software products run is that they all get something right. Seeing that none of them gives the full solution, take their strengths and combine them into something that will allow us to tackle technical risk differently.
My Solution? Technical Risk Probes {#my-solution?-technical-risk-probes}
If you are now wondering what a technical risk probe is, no need to google it. Claude Opus and I coined the term, because we could not think of anything better either. I can tell you that the runner-ups were API conceptualization test case and API call prototype. Anyone have a better name? Drop me a line.
Bracketing terminology, here is what a technical risk probe is trying to accomplish. A technical risk probe is trying to make the surface squeal. Your design says X, Y and Z; your terraform rephrases that claim in HCL; but the probe will have none of it and wants to see it in action. Let’s see if all these assumptions are even half-true.
And how do we build this Doubting Thomas? Recall that we are trying to draw inspiration from the strengths of all the typical solutions we discussed above. We are trying to package these strengths in a new way to avoid (many of) their downsides.
The better Example
During the exposition, the example was a strong favorite. I argued experience as a weaker example, while the prototype and the tracer bullet were too complex and unwieldy for our purpose.
The example earned our praise by being small, focused, and independent. That makes it easy to implement, adapt and maintain – again, in parallel.
One concern was pedagogical simplification, in terms of implementation, default assumptions and data used. A simplified example would not test what a real-world example required. So, the first improvement to the example as such is that we make it more germane to the product’s needs:
- •Identical implementation language as the deployment language
- •Identical library version as planned for the deployment
- •A sufficiently complex usecase, especially option- and datawise, taken from user stories
Contributions from Prototype and Tracer Bullet
But there were strengths to the prototype and the tracer bullet as well. These solutions achieve good coverage because they test multiple different things. We can match this advantage by writing many technical risk probes; in parallel, if necessary.
Furthermore, both prototype and tracer bullet, due to their lengthy development, continuously retest the surfaces already integrated. This feature we lack. So we stipulate that the technical surface probes be rerun at fixed intervals. Perhaps nightly is a good compromise between the velocity of alerting and resource expenditure.
With the desire to rerun probes, we have picked up an additional requirement, however. Like a software test fixture, we now need to be able to handle some setup and tear-down. That makes our probes a bit less focused, but increases the realism of their experimentation.
Constructing Technical Risk Probes
To summarize, a technical risk probe is
- •an Independent Software Component that
- ◦… implements a focused surface interaction (API call, library call, etc)
- ◦… including setup and tear down
- ◦… using the targeted API version range
- ◦… implemented in the targeted language
- ◦… using task-appropriate defaults
- ◦… using real data taken from user stories
- ◦
That’ll do. And how do we go about getting them? Your design documents will tell you.
Identify each necessary interaction with a risk surface, whether API call or service interaction.
For each interaction:
- Find an example from a tutorial, book, youtube video, stackoverflow, …; or let some LLM generate one for you.
- Rewrite the example to match your target language and target library version range.
- Rewrite the example to use data from your user stories.
- Rewrite the example to specify where the defaults don’t work.
- Add setup and teardown to make it usable over and over and over.
- Get it to work.
- In the design document, refer back to the probe (e.g. the URL of the git-repo).
Most likely the insights will begin to roll in immediately. All those little details the designs never mention – ordering constraints, required scoping, a setting that is the difference between runs sometimes and runs reliable – the probe will drum them out of the woodwork. Because they guard your design, they will inform your design. Mark my words.
All that is missing now is a nightly runner – a cronjob, a scheduled pipeline, some cloud runner or function in a hyper scaler’s framework — your choice.
And a little dashboard. For your Observability folks. So they can say, “All green!” They love saying that.
Even more benefits?!?!
When they hear about the observability dashboard that monitors their technical risk surface daily, most people are sold. And while that is the core of the pitch, here are the side benefits.
- Documentation: Nothing puts meat on your design like a bit of runnable code that illustrates just that.
- Communication piece: Your collaborators don’t quite understand what you want to do? A piece of code they can noodle around with will move the conversation forward. Worst case, they fix your probe and send it back! Win-win!
- Flexible to maintain: Suppose they add a library. You write another probe. They switch the target library on you – pandas for numpy, for sake of argument? You rewrite just those probes. They take a library out? Just leave the probes in. It’s pennies on your cloud bill and worth your sleep.
- Onboarding & Training: Writing a probe is a great way to enter the project. Implementing a module for the product based on a probe is so obvious, that might even allow you to take on late help without making the project later.
What Probes are Not
You knew it – some restrictions would apply! Don’t they always?
But seriously, there is a reason we have prototypes and tracer bullets. And probes are not that. You can string together a whole necklace of amazingly informative technical risk surface probes. That won’t tell you how the data flows. Or whether the whole assemblage even makes any sense.
That’s not the point. That’s not their job.
Probes are also no substitute for testing. Component, system, integration, end-to-end, fuzzy, edge cases or whichever testing. Again, not their point. Not their job.
They tell you when an iceberg is heading your way. That’s what they are paid to do.
In A Nutshell
In the rapidly shifting world of building products in cloud environments, technical risk surface probes are friends, because they are
- •Stand-alone software components which implement
- ◦… a single Repeatable relevant system interaction
- ◦… in a way that is Implementation-compatible with Deployment
- ◦
They continuously monitor for drift in the product’s surface risk.
While, en passant, documenting, validating and communicating its design.
Nuff said.
This blog was conceived and written by the author. It may incorporate suggestions from an LLM.
