Every organisation has validation steps. Data validation, supplier validation, pre-flight checks, sign-offs. They appear on process maps and in board papers as controls.
Here is the test that sorts the real ones from the decorative ones. When did that check last fail something?
If the honest answer is rarely, or nobody could say, it is not validating anything. It is confirming, and confirmation is what a process does when it is designed to proceed.
The difference is structural, not cultural
This is not about people being lax. It is usually built into the mechanism.
A check that can only verify presence rather than correctness cannot fail anything meaningful. Are the required fields populated? Yes. Does the certificate exist? Yes. Has someone signed? Yes. Every answer is yes because the questions can only be answered yes once the work has been done at all.
Real validation requires a stated criterion that the thing being checked can fail. Without one, you have a completeness check that reports success by design.
The largest documented example
American healthcare has run this at scale, and the results are public because the final checker is a federal agency that publishes.
US insurers covering Medicare patients are paid more for covering sicker people, with illness established by diagnosis codes drawn from clinical notes. Software reviews those notes and decides which diagnoses to submit. That review step is the validation layer.
It was built to find diagnoses that had been missed, because finding more generated revenue. Its output is a ranked list of candidates with confidence scores, and a ranked list has no entry meaning “this existing diagnosis is unsupported.” You can score something low. You cannot mark it wrong.
So the validation step could only ever add. It ran for years, looked like a control, and functioned as a completeness check.
What the external check found
The US Office of Inspector General published its detection method in December 2023, including the database queries it uses to flag diagnoses at high risk of lacking support. Any organisation could have run the same test on itself.
When the agency ran it, roughly 70 percent of those codes were not supported by the associated medical records, with some categories above 90 percent. One audit sampled 97 records carrying an acute stroke diagnosis. None held up. In 68 of those, the patient genuinely had had a stroke years earlier and the record described it as current.
Thirty months between the method being published and the audit landing. The test was available the whole time.
Three questions for your own controls
- When did this check last reject something, and what happened next? A control with no recent rejections is either unnecessary or not working, and you cannot tell which from inside.
- Is there a written criterion the thing being checked could fail? If the standard lives in a reviewer’s judgment rather than on paper, it cannot be audited and it drifts without anyone noticing.
- Could an outsider run this check and reach the same answer? If not, you have expertise rather than control, and expertise leaves when people do.
The practical lesson
In the healthcare case, organisations are rebuilding the review layer so it can return a negative result, and preparing for the external check on the assumption that it is coming. The formal name for that external check is risk adjustment data validation, and the shift it has forced is from reviewing to prove things were found, to reviewing to prove things hold up.
Most businesses never get handed the regulator’s test in advance. That makes writing down what your own checks are supposed to catch a cheap exercise compared with discovering it from someone else’s sample.
Leave a Comment