Human in the Loop: How to Tell If the Review Is Real

Human oversight is the only control in an AI program that can stop working while producing exactly the same evidence as when it worked. A failed encryption control throws errors. A monitoring pipeline that breaks stops delivering alerts. A review step that has become a formality still generates approvals, timestamps and sign-offs, and the compliance file looks identical.

‍

The asymmetry makes the design question secondary to the measurement one. Plenty of material explains how to place a human in a workflow. Considerably less addresses how an organization would discover that the human stopped reviewing, which is the failure that matters because it is the one nobody notices.

‍

The Regulation Already Rejects the Formality

‍

European rules on high-risk systems are explicit that oversight cannot be symbolic. The requirements set out five things the arrangement has to deliver rather than describing a review step and leaving the rest to interpretation.

‍

The person overseeing must understand the system's capacities and limitations, remain conscious of the tendency to over-trust automated output, be able to interpret how an output was produced, hold genuine authority to disregard or reverse a decision, and have access to a mechanism that halts the system safely. Automation bias is named directly, which is unusual in a legal text and tells you the drafters anticipated the failure mode. The wider high-risk obligations place this alongside risk management and record-keeping duties.

‍

Authority Has to Be Real

‍

Override authority on paper means little where a reviewer who disagrees with the model faces a performance conversation about throughput or a requirement to justify each departure. The test is whether overriding is as easy as approving, and in most implementations it is not, since approval is a single action and disagreement requires an explanation.

‍

Why Review Degrades

‍

Two mechanisms operate together and both worsen as a deployment succeeds.

‍

Accuracy Erodes Vigilance

‍

A reviewer working with a model that is right most of the time learns, correctly, that scrutinizing each output rarely changes anything. Attention drops accordingly. The behavior is rational at the level of the individual case and produces exactly the wrong outcome on the rare case that matters, which is why automation bias strengthens as model quality improves rather than weakening.

‍

Throughput Pressure Finishes the Job

‍

Where volume rises and review time per item is not protected, the arithmetic resolves itself. A queue sized on the assumption of rapid approval cannot accommodate genuine consideration, so the organization has decided the question by capacity planning rather than by policy. Anyone measuring reviewer productivity in items per hour has already chosen speed over scrutiny whatever the governance document says.

‍

Six Signals That Tell You Whether Review Is Real

‍

All of these are computable from data most systems already produce, and none requires asking reviewers how carefully they are working.

‍

• Override Rate: A rate at or near zero means either a perfect model or an absent reviewer, and the second is more common.
• Time on Decision: Median seconds per item compared against the time needed to read the inputs at all.
• Variance Between Reviewers: Wide spread in override rates across people seeing comparable cases indicates some are reviewing and others are not.

‍

Three further signals require slightly more effort and produce sharper answers. Override rate over time, where a rate declining as familiarity grows is automation bias visible in the data rather than inferred. Performance on seeded cases, meaning known-incorrect outputs inserted into the queue deliberately, which is the only direct test of whether anyone is looking. Override accuracy completes the set, covering whether the reviewer was right when they disagreed.

‍

A Zero Override Rate Is a Finding

‍

Programs frequently report low override rates as evidence the model is performing well. Read alongside median review time it usually says something else, and the combination of a very low override rate with a very short review time is the strongest available signature of a control that has stopped operating. Neither number means much alone.

‍

Design Choices That Prevent It

‍

The strongest intervention is to stop putting a human on everything, because attention is finite and spreading it uniformly guarantees it is thin everywhere.

‍

Reserve Review for Consequence

‍

Routing every output through a person produces uniform inattention. Routing the consequential minority through a person with time to consider them produces review that happens. Selecting on impact rather than on volume means defining which decisions warrant it, which is a governance decision rather than a technical one, and categorizing systems by consequence is where that line gets drawn.

‍

Show the Basis, Not the Score

‍

An interface presenting a confidence figure invites deference. One presenting the factors that drove the output, including the ones that cut against it, invites evaluation. The regulatory requirement to enable interpretation of output points the same way, and it is a product decision that governance teams often assume is out of their hands.

‍

Require a Reason for Approval

‍

Most workflows demand justification for overriding and accept approval as a click, which prices disagreement above agreement. Requiring a brief reason for both makes the cost symmetric and produces a record that shows whether the reviewer engaged with the specific case or repeated the same phrase forty times.

‍

When Human Review Makes Things Worse

‍

An uncomfortable possibility deserves stating, because programs rarely test for it. Oversight adds value only where the reviewer is more accurate than the model on the population being reviewed, and that is an empirical question rather than an assumption.

‍

Where a reviewer overrides frequently and is usually wrong, the arrangement is injecting error while satisfying a control requirement. Measuring override accuracy against later ground truth is the only way to find out, and a record that supports later comparison is what makes the measurement possible, and the answer sometimes argues for better reviewer training or narrower routing rather than for more review. Regulatory obligations may still require the human step regardless, in which case the finding argues for improving it rather than removing it.

‍

What to Record Per System

‍

Three fields make oversight auditable rather than assumed, and their absence is what turns a question into an investigation.

‍

Whether human review applies at all, recorded explicitly rather than left blank, since an unspecified field and a considered decision that review is unnecessary look identical later, and the unit the register uses determines whether the field is even asked per deployment. The decision threshold that routes an item to a person, so a change in that threshold is visible as a change in the control. The measured signals above belong on a regular cadence, since oversight quality moves and a one-time assessment describes the week it was performed. Periodic testing leaves exactly the intervals in which this control decays.

‍

Attestation Rates Should Be Plausible

‍

Where review is recorded as an attestation, the attestation data itself is worth examining. Rates at or near one hundred percent across a large volume, sustained over months, describe either an exceptionally disciplined team or a checkbox. Comparing attestation volume against time available is a five-minute exercise that settles it.

‍

Measure the Control, Not the Diagram

‍

Human oversight is easy to specify, easy to document and unusually easy to lose without noticing, because a formality generates the same paperwork as a genuine review. Override rate, time on decision, variance between reviewers, movement over time, performance on seeded cases and override accuracy turn an assumption into a measurement. Regulation already requires the arrangement to be capable of catching and reversing errors rather than merely existing, so the evidence has to show it working. Kovrr's AI Security and Governance Platform records whether oversight applies per system and holds the assessment history that shows whether it still does.

‍

To see which AI systems in your environment have human review recorded, and which have the field left blank, book a demo mapped to your own estate.