A special operations analyst asked an AI chatbot to assess the manifest of a Chinese vessel. The output said the ship carried components for a nuclear weapons program. The analyst read it, believed it, and passed it on. By the time anyone caught the error, aircraft were airborne and armed personnel were preparing to board. The chatbot had misidentified the cargo.
Every AI governance policy written in the last three years would have scored this workflow as compliant. A human was in the loop. A human read the output. A human made the call.
That is the problem. The human was there. The control was not.
Oversight is not the same as review
Most boards have accepted "human in the loop" as the default answer to AI risk. It sounds like a control. In practice it often describes a person who reads a conclusion and decides whether it feels right. That is review. Verification means checking the conclusion against its source, and nothing in most policies requires it.
The analyst in this case did not lack judgment. The analyst lacked a step that required going back to the source data before treating the finding as fact. The AI produced a hypothesis. The workflow handled it as a finding. Nothing in the process forced the difference to matter.
Oversight that only rereads the output checks whether the conclusion is plausible. AI output is almost always plausible. That is what these systems are built to produce.
Three gaps worth checking in your own organization
1. Provenance, not review
Real oversight traces a finding back to its inputs. Which document did the model read? Does the source say what the output claims it says? If no one in the workflow is required to answer those questions, the oversight is cosmetic.
2. A named owner for verification
After the fact, no one could say who owned the decision to validate the finding before the operation moved. That is the most common gap we see. Organizations assign an owner to the AI system and an owner to the business decision. The step between them, checking the output against its source, belongs to no one.
3. A stated answer on speed versus verification
Speed was an explicit priority in the strategy behind this workflow. That is a legitimate choice. It becomes a governance failure when it is never written down, because then each analyst makes the tradeoff alone, under pressure, in the moment. Leaders should decide in advance which wins when the two conflict, and for which decisions.
Where this sits in the Four Signals Framework
This is an Authority failure. The Authority signal asks a simple question: where does the AI decide, where does the human decide, and is that boundary written down? In this case the boundary existed only in theory. In practice the AI decided, and a human carried the decision forward.
An organization rated Governed on Authority can point to the document that defines which AI outputs require source verification, who performs it, and what happens when time pressure and verification collide. An organization rated Exposed has a human-in-the-loop policy and nothing beneath it. Most organizations we assess sit in the second group and believe they sit in the first.
Questions for the board
- For each AI system that informs a consequential decision, who is named as responsible for verifying its output against the source?
- Do our policies distinguish between reviewing AI output and verifying it?
- Where have we chosen speed over verification, and was that choice made by leadership or left to the person in the moment?
Adding a human to the loop does not close an accountability gap. It moves it. The board's job is to know where it moved to.
Before any AI system informs a consequential decision, name the person responsible for checking its output against the original source. If you cannot name them, the human in your loop is a witness.
An earlier version of this piece appeared in Northern Signal, Tim Reed's briefing on AI governance and security.