The Human Was in the Loop. That Was the Problem: Why Human Oversight Does Not Close the AI Accountability Gap

A reported military near-miss shows what boards keep missing about AI oversight. Putting a person in the decision path does not tell you who owns the decision.

A special operations analyst asked an AI chatbot to assess the manifest of a Chinese vessel. The output said the ship carried components for a nuclear weapons program. The analyst read it, believed it, and passed it on. By the time anyone caught the error, aircraft were airborne and armed personnel were preparing to board. The chatbot had misidentified the cargo.

Every AI governance policy written in the last three years would have scored this workflow as compliant. A human was in the loop. A human read the output. A human made the call.

That is the problem. The human was there. The control was not.

Oversight is not the same as review

Most boards have accepted "human in the loop" as the default answer to AI risk. It sounds like a control. In practice it often describes a person who reads a conclusion and decides whether it feels right. That is review. Verification means checking the conclusion against its source, and nothing in most policies requires it.

The analyst in this case did not lack judgment. The analyst lacked a step that required going back to the source data before treating the finding as fact. The AI produced a hypothesis. The workflow handled it as a finding. Nothing in the process forced the difference to matter.

Oversight that only rereads the output checks whether the conclusion is plausible. AI output is almost always plausible. That is what these systems are built to produce.

Three gaps worth checking in your own organization

1. Provenance, not review

Real oversight traces a finding back to its inputs. Which document did the model read? Does the source say what the output claims it says? If no one in the workflow is required to answer those questions, the oversight is cosmetic.

2. A named owner for verification

After the fact, no one could say who owned the decision to validate the finding before the operation moved. That is the most common gap we see. Organizations assign an owner to the AI system and an owner to the business decision. The step between them, checking the output against its source, belongs to no one.

3. A stated answer on speed versus verification

Speed was an explicit priority in the strategy behind this workflow. That is a legitimate choice. It becomes a governance failure when it is never written down, because then each analyst makes the tradeoff alone, under pressure, in the moment. Leaders should decide in advance which wins when the two conflict, and for which decisions.

Where this sits in the Four Signals Framework

This is an Authority failure. The Authority signal asks a simple question: where does the AI decide, where does the human decide, and is that boundary written down? In this case the boundary existed only in theory. In practice the AI decided, and a human carried the decision forward.

An organization rated Governed on Authority can point to the document that defines which AI outputs require source verification, who performs it, and what happens when time pressure and verification collide. An organization rated Exposed has a human-in-the-loop policy and nothing beneath it. Most organizations we assess sit in the second group and believe they sit in the first.

Questions for the board

Adding a human to the loop does not close an accountability gap. It moves it. The board's job is to know where it moved to.

Operational Takeaway

Before any AI system informs a consequential decision, name the person responsible for checking its output against the original source. If you cannot name them, the human in your loop is a witness.

An earlier version of this piece appeared in Northern Signal, Tim Reed's briefing on AI governance and security.


Tim Reed, CPP

Tim Reed, CPP, is the founder of The Reed Group and author of Signals in the Noise: Security, Technology, and the Hidden Patterns of Modern Risk (ISZ Press, 2026). With 28 years of experience across corporate security, executive protection, and intelligence programs at Apple and Uber, he built the firm to bring operational security judgment to AI governance and board advisory work. He writes at Northern Signal.

Is Your Authority Boundary Written Down?

The AI Readiness Review rates your organization against all four signals, including where AI decides and where people do, and delivers a board-ready report with a risk tier rating.

Request an AI Readiness Review