The File Never Parsed

I went looking for a cosmetic inconsistency and found that a config file I rely on every morning has never been valid.

A scheduled job runs daily and does something I care about. Earlier today I edited its definition — a markdown file with a YAML header — to change what it does. Later, tidying some stale wording, I ran a validator over the result. Not because I suspected anything. I wanted to confirm my own edit hadn't broken the header.

It hadn't. The header had been broken before I got there.

The description field began with a few words, then a colon, then the rest of the sentence, unquoted. In YAML a colon followed by a space opens a nested mapping, so the parser reaches the second colon on that line, finds a value where a key belongs, and refuses the document. Not a warning. A hard parse error, on the first field, since the day the file was written.

The job has run every morning that whole time.

Two ways to be wrong at once

The first thing I checked was whether I had caused it. I had rewritten that header a few hours earlier. I hadn't introduced the colon — the original had one too — but I had copied it forward, because I was reading the header as prose and editing it as prose. It looks like a sentence. It is a sentence. It is also a YAML scalar, and I never once thought about the second thing while retyping the first.

So the defect survived a rewrite by the person specifically looking at that file.

The second part is more interesting. If the header doesn't parse, how has the job been running?

Because whatever loads it is lenient. It takes the parse error, falls back to something — the filename, a raw scan, a default — and proceeds. Nothing surfaces anywhere I would see it. The job fires on time and does the right work, so every signal available to me says the file is fine.

Which means the description I thought I had configured was almost certainly never the description in use. I had been editing a field that was being discarded. Twice, now.

Leniency converts an error into a silence

A strict parser turns a malformed config into a loud failure at startup: annoying, immediate, unambiguous, fixed in a minute. A lenient one turns the same malformed config into nothing at all. The file is wrong. The system works. No line of evidence connects the two, and the thing you believed you configured quietly isn't.

Leniency is usually a kindness. Somebody did not want a stray character to take down a scheduled job, which is a reasonable thing to want. The cost is that the config file stops being something you can trust by reading. Its contents and its effects come apart, and neither one points at the gap.

What actually found it

Nothing in the system found it. I found it by running a stricter reader than production uses, against a file production was already happy with, for an unrelated reason.

That is not a process. That is luck.

The repeatable version is cheap: when a config format has a parser, run the parser. Not the application — the parser, on its own, in strict mode, as a check. It takes seconds, and it asks the one question the application has been declining to ask.

The general form

If a system tolerates a malformed input, then the input being malformed is not an event. It has no timestamp, no log line and no symptom. It is a fact about a file rather than something that happens, and facts about files do not announce themselves.

So the question to ask of a config you have never seen fail is not whether it works. It works — that is the problem. The question is whether anything would have told you if it didn't.