A typical at-home COVID antigen test has two places where a line can appear. The test line reports the result. The control line decides whether there is a result to read. If the control line is missing, one test's instructions say the result is invalid, even if a test line appears. An invalid result means the test could not tell whether you have COVID-19.
AI systems are tested too. The test, usually called an evaluation, tends to follow the same four steps: give the system a set of tasks, score its work, combine the scores, report a number. Nothing in those four steps does the control line's job. An evaluation can fail by successfully producing a score.
In "Evaluated for What?" I argued that a test proves only what it measured. A test showing that an AI system answers a question correctly does not show that the answer remains correct when the question is reworded. This piece is about the next problem. Even a test that measures the right thing can report a result the evidence does not support.
The third outcome
A passing score and a failing score are both statements about the system being tested. An evaluation also needs a third outcome, and this outcome is about the test: no valid result. This third outcome reports that the conditions for a meaningful score were not met, and it names the condition that failed.
A failing score and an invalid test call for different decisions. "The system failed the test" is a reason to fix the system or stop relying on it. "The evidence was insufficient to run a valid test" is a reason to fix the test. An evaluation that cannot tell the two apart reports a number in both cases.
To tell the two apart, an evaluation needs the third outcome, and the third outcome needs two rules. The first rule is that the conditions that permit a score are set before the results arrive: enough tasks to support each score, a scoring method checked against known answers, a rule for missing data, and test conditions held fixed. The control line works this way: the instructions say what a missing line means before anyone takes a sample. If an evaluator sets the conditions afterward instead, that evaluator can look at the results first, keep a favorable score, and declare an inconvenient one invalid.
The second rule is that everything left unscored stays in the report: how many tasks or whole categories of tasks were left out, why they were left out, and how many tasks were counted in each reported score. Without both rules, "no valid result" can become a way to hide bad scores.
Why a number can still arrive
The way an evaluation arrives at a number its evidence does not support is usually mundane. Without a rule for missing data, the missing data gets whatever treatment the evaluation applies by default: a zero, a skipped task, or an average of whatever remains. Some categories contain too few tasks to support a separate score, and the arithmetic still produces a percentage for the category. Nothing crashes. The evaluation completes, and completion is easy to mistake for validity.
A score that the evidence did not earn is a small instance of what I have called evidence theater: confidence in a conclusion that reaches further than the test, audit, or review offered to support that conclusion.
Where the evidence gives out
A score depends on three things besides the system being tested: a standard for what counts as correct, a method for comparing the system's work to that standard, and the conditions of the test. The evidence behind a reported score can give out at any of the three: the standard, the comparison, or the conditions.
The answer key. The standard is often an answer key: a set of answers treated as correct. Evaluators often label it "gold," but the answers are not authoritative because someone named them that. Someone wrote those answers down, and that person can be wrong, out of date, or in disagreement with the next person. Suppose an AI system that reviews contracts is tested on how many days of notice each agreement in a set requires before termination. Two lawyers prepared the answer key, and for one of the agreements the first lawyer wrote 30 days and the second wrote 60. Until someone resolves that disagreement, the system can be marked wrong for giving the right answer, and the reported score treats the answer key as settled.
The scorer. A system's work does not become a score by itself. Something has to compare that work to the answer key and decide what counts as correct, what to do when the system leaves something out, and which scores to average into the reported number. That something is the scorer: a person, a script, or another AI system. The comparison is harder than it sounds. If the answer key says "30 days" and the system writes "thirty days," a scorer that matches characters will mark the system wrong, and the score will report a difference in wording as a difference in judgment. Every score therefore reflects the scorer's judgment as well as the system's work, and the scorer can be wrong. Before relying on the scorer, the evaluator has to test the scorer on sample work whose correct score the evaluator already knows. That sample work should include a response that says nothing useful where a real answer exists, to confirm the scorer does not mark that response correct. This check does for the scorer what the control line does for the COVID test.
The conditions. A test result describes one AI system under the conditions of one test: the version of the system, the instructions it was given, the documents it drew on. If the contract-review system in the example above is updated to a new version, the earlier result does not automatically apply to the updated system, and the evaluator has to run the test again. I described how an earlier assessment goes stale in "The Expiring Audit."
In each of the three cases, the evaluation can run to completion and print a score that looks no different from a sound score.
What a lawyer can check
Lawyers see scores in vendor reports, pilot summaries, and diligence files. A lawyer does not need to read the scoring code to check this one control: whether the evaluation was permitted to decline to score a task, and whether the report counts the tasks the evaluation declined.
A table of scores with a number in every row looks more complete than a table with blanks. A full table may mean only that the evaluation had no way to say it did not know something. A blank with a stated reason, left under a rule set before the run, is some evidence that the evaluation knows where its proof ends. That evidence remains valid only if the report also tallies every blank and shows how many tasks support each number.
The ability to report no valid result does not validate the other scores. What that ability shows is that the evaluation can recognize a limit on its own evidence. Without that ability, an evaluation can report a number without revealing whether the evidence was sufficient to support it.
Lana Akopyan is an IP attorney and independent AI-evaluation researcher. She writes about using AI in legal practice without surrendering judgment, privilege, or the duty of competence at The Agentic Lawyer.
Educational only, not legal advice, and no attorney-client relationship is created. Views are my own. Attorney advertising in some jurisdictions.