A Control Is a Position in Time

The release process got a new rule a while back: nothing goes to production unless the exact revision being deployed already has a passing build behind it. Not the branch, not something close to it. That revision.

Then I went back to check what was already live against the new rule, which is how I ended up looking at a timestamp I have not been able to stop thinking about.

Thirty-six seconds

One of the live revisions had a passing build. The right revision, the right branch, every check green. The build finished thirty-six seconds after the deploy went public.

Nothing about that evidence was false. The code did pass. If you had asked me whether that revision was proven good, the honest answer was yes, and the record said so, and the record was right.

It just had nothing to do with the decision to ship. By the time the evidence existed, the thing it was supposed to authorise had already happened. The deploy had not been gated by it. The deploy had been gated by nothing at all, and then a few seconds later some evidence turned up and arranged itself next to the deploy in a way that, from any distance, looked exactly like a control working.

That is the part I had wrong. I had been treating certified as a property of a revision. It is not a property of a revision. It is a claim about the ordering of two events, and a record that stores only the outcome cannot tell you whether the ordering held.

What the audit actually found

Three revisions were worth looking at, and they came out in three different states.

One was deployed with evidence that post-dated it by thirty-six seconds. The case above.

One was deployed with no qualifying evidence at all, and that status is the one I found most interesting, because it cannot be repaired. It had shipped from a side branch, and a push to a side branch does not produce the kind of build the rule requires. So no qualifying evidence for that revision exists, and none can now be brought into existence. It is not proven bad. It is unprovable, permanently. Those are different things, and most dashboards will render them in the same colour.

And one revision had genuinely passed through the intended ordering: build first, publish second. It was the commit that implemented the rule. It was never deployed to anyone.

So exactly one revision in the entire history had done this correctly, and it was not either of the releases that users had actually been served.

The tidying I did not do

The obvious move was available, and I want to be honest about how attractive it was.

I could have run a build on the unprovable revision there and then. It would have passed. I could have attached the result, marked the revision certified, and every artefact in the system would have agreed with every other one. The history would have read as clean.

The statement would even have been true. It would simply have been true about a different question: whether that code passes today, rather than whether anything had checked it before it went in front of users. One of those is a control. The other is a fact with convenient timing.

So the record kept three separate statuses instead of flattening them into one, and the rule applies forward only. The uncomfortable version was the accurate one, and there was no version that was both tidy and true.

A few days later a deploy went through the whole sequence in the intended order. Every condition checked, and only then was anything published. Not a better outcome than the thirty-six second case. The same outcome, reached in the order that makes it mean something.

The generalisation

Every control you have is a claim about ordering, and almost nothing you build records the ordering. It records the verdict.

So ask it of anything you rely on. The review that happened, did it happen before the merge, or after the merge and before anybody looked? The sign-off on the document, was it given on the version that shipped, or on the one before it? The test run attached to the release, when did it start?

If the answer comes back as we have the result, you do not know yet. A result tells you what was true. A control tells you what was known at the moment somebody was permitted to proceed.

Evidence that arrives after the decision it was meant to gate is not a gate. It is a coincidence with good paperwork.