The Biggest Line Was the Only Copy

The machine was nearly full. Not a crisis, but close enough that work had started failing in ways that pointed at storage, and the obvious next step was to reclaim some.

I already had a culprit in mind. Months of doing several things at once had left a large number of separate working copies scattered around the machine, the ordinary residue of that habit. They were the thing I could name, so they were the thing I was going to remove. I wrote the reclamation as the first step: remove the dead ones, tidy up, check the free space again.

Before running it I asked for a census instead. Read-only, with an explicit list of what it was not allowed to do: delete nothing, tidy nothing, move nothing, just report sizes.

The named culprit held almost nothing

Nearly all of those working copies turned out to still be in use, and together they were not where the space had gone. The thing I could name was innocent. Had I run the reclamation as written, I would have freed very little and destroyed the comparison that showed me so.

The space was in two places I had not considered. One was a directory of abandoned copies left behind weeks earlier by a batch of automated sessions — on its own, three times as large as all the free space I had left. The other was a single project that existed twice on the machine, in two unrelated locations, each copy about the size of the other.

That second one was the largest single reclaimable item on the report. It was also the one item that must never be touched.

Why the biggest number was the worst candidate

Inside one of those two copies were large binary files that had never been sent anywhere else. They existed on that machine and nowhere else in the world. Removing the redundant copy was worth a genuinely large amount of space — but only one of the two copies was redundant, and nothing on the report said which.

So the report had, without meaning to, sorted my options by exactly the wrong thing.

A list of cleanup candidates ordered by size is ordered by what you get back. It is silent on what it costs you if you are wrong. For most items on such a list, being wrong is cheap and bounded: you remove something you can fetch again, and you have lost a few minutes. For one item on that list, being wrong meant the thing ceased to exist. Those are not different amounts of the same quantity. They are different quantities, and putting them in a single column ranked them against each other as though they were comparable.

This is why storage cleanups go wrong in a particular, recognisable way. The tool that finds the space hands you a list sorted by benefit. The list looks like a work queue, so you work it from the top. But the top of a benefit-sorted list is where the irreversible cost tends to sit, because big things are often big precisely because nothing else is holding a copy of them.

Two orderings, not one

What I should have asked for, and now do, is a second column. For each candidate: if I remove this and I turn out to be wrong, what gets it back? A copy held elsewhere. A rebuild. A download. Nothing.

Only that last answer changes the kind of decision I am making. Everything above it is a question of convenience; that one is a question of whether the thing survives at all. An item whose recovery answer is nothing does not belong on the same list as the rest, however much space it would free.

Size ranks what you stand to gain. It says nothing whatsoever about what you are risking, and on a list of things to delete, those two orderings can be exact opposites.