An Answer Is Not a Write

Someone on the team told me they could not trust a report I had built. The report lists, each morning, which work items are actually available for them to pick up. That morning it listed two they had spent the previous evening on.

I did what I think most engineers would do, which is go and check whether the report was wrong.

The report was right

It was not wrong. I pulled both items straight out of the tracker and compared them against what the report had said. Both sat in states that mean, unambiguously, this is available for someone to work on. One had been in that state since it was opened, weeks earlier. The other had not been touched in two days.

The report had read its source correctly and described it accurately. If the question was whether the report faithfully reflected the tracker, the answer was yes, and I could have said so in a sentence.

I am glad I did not, because the other half was also true. They had worked both items. On the first, they had tried to reproduce the problem and could not assemble the conditions it needed. On the second, they had been told to hold off until somebody else weighed in. Two real attempts, two real obstacles, each of which stopped the work for a legitimate reason.

So the report was accurate and the person was accurate, and I spent a while assuming that meant one of us had to be subtly mistaken.

Where the obstacles were

Neither obstacle was a mystery. Both had been stated plainly, in writing, with timestamps, before the report ever ran.

Both had been stated by the automation. The same automation that generated the report.

The night before, it had told this person that the conditions needed to reproduce the first item were not met. It had told them, separately, not to spend time on the second until someone else answered. Both messages were correct, useful and specific. Then the next morning the same system assembled a list of work available to that person and put both items on it.

It had not forgotten anything. There was no race, no stale cache, no bug I could point at. It had simply said those things in a chat channel, and it computes the report from the tracker, and nothing it says in a chat channel has ever become tracker state.

Saying is not writing

That is the whole lesson, and it took me embarrassingly long to see, because I had been looking for a defect in the report.

An actor that answers questions in one place and computes answers from another place will contradict itself. Not occasionally, and not because it is badly built. Routinely, and with total confidence on both occasions, because the two outputs are drawn from two different stores and only one of them is the store it writes to.

The knowledge was never missing. It was not missing from the team, and it was not even missing from the system that produced the contradiction. It was missing from the path between what that system had said and what it later computed.

I think that is why this is so hard to notice from the inside. There is no gap to find. Every individual statement is correct and well sourced. The report is faithful to the tracker, the messages were faithful to the situation, and each one would survive any audit you ran on it alone. The inconsistency exists only across the two, and nothing in the system was looking across the two.

What I changed

The small fix is the obvious one. When the automation tells someone they are blocked, it has to write that to the tracker and not only to the person. An obstacle that lives in a sentence is an obstacle that will be rediscovered by whoever next reads a list.

The larger change is to how I read my own confidence. If I am about to tell someone that their experience disagrees with my data, that is now the moment I go looking for something I have asserted and never stored. Being able to show that a report matches its source is a far weaker claim than it feels like while you are making it. It establishes that the pipeline is intact. It establishes nothing about whether the source knows what I know.

The generalisation

Every assertion you make that is not written where your own code looks is a fact you have spent and not kept.

This goes well past agents and chat channels. The incident you narrated in a thread and never put in the writeup. The constraint you explained in a review comment that never became a type. The decision you announced in a meeting that lives in no document. In each case the information was transmitted perfectly and recorded nowhere, and the system that will later need it is not subscribed to the channel it went out on.

So when a report and a person disagree, and the report provably matches its source, be suspicious of the version of the story where the person is confused. They are usually holding the input your pipeline cannot see, and they are the only part of the system that noticed.