AI Governance When the Output Is a Hiring Decision

A recruitment system faces two obligations at once. AI rules classify it as high-risk and require the governance that follows. Employment law has required attention to discriminatory effect for decades, independent of any technology.

‍

They are not two versions of the same requirement. The statistical test is conducted on the tool and the legal test is conducted on the outcome, so they have different subjects rather than different thresholds, and a tool can pass one while the process fails the other.

‍

Why Is the Vendor's Audit the Wrong Unit?

‍

Because it aggregates across a population and liability attaches to a specific employer and a specific role.

‍

A vendor testing its model across millions of applicants spanning many industries can demonstrate that selection rates across groups sit within the four-fifths rule, the eighty percent benchmark federal enforcement agencies generally treat as the line where adverse impact becomes evidenced. The result describes the vendor's aggregate population. Your exposure is assessed at the level of the job you are filling, in your applicant pool, with your other criteria applied.

‍

Which Is the Same Problem an Average Always Has

‍

An aggregate conceals the distribution it came from. A model performing evenly overall can perform unevenly for a particular role where the applicant pool differs from the average, and the vendor certificate says nothing about that case because the case was one row in the dataset.

‍

Can Every Stage Pass While the Process Fails?

‍

Yes, and the arithmetic is worth stating because it is checkable and almost nobody runs it.

‍

The Guidelines direct the analysis at the total selection process for a job rather than at any component, requiring component-level records only where the total shows adverse impact. The direction matters, because selection effects compound across stages. Two sequential steps each passing at a selection ratio of eighty-five percent produce a combined ratio around seventy-two percent, which sits below the threshold either one satisfied individually. Three stages at ninety percent each land near seventy-three percent.

‍

So an audit certifying the AI component and a separate check on the human review stage can both return a pass while the pipeline from application to hire does not, which is the level the Guidelines point at. The employer carries the consequence of the final effect, whatever each component demonstrated on its own.

‍

The Audit Boundary Is the Problem

‍

An audit scoped to the automated tool is answering a question about the tool. The obligation attaches to the outcome of the whole process, so scoping the audit to match the technology rather than the decision leaves the compounding effect unmeasured by design.

‍

What Evidence Does Each Test Need?

‍

Almost entirely different sets, which is the part that surprises programs expecting one exercise to serve both.

‍

A statistical audit produces selection rates by group for the component under test, with the methodology and the population described. It is a measurement exercise and what the AI-facing obligation largely asks for.

‍

A legal position needs something else. Counsel will tell you the question turns on whether the criteria are related to the job and justified by business necessity, and whether a less discriminatory alternative was available and considered. None of that is a fairness metric. It is documentation about why the criteria were chosen and what else was evaluated.

‍

Can Optimizing for One Weaken the Other?

‍

A specific and underappreciated trade, and it appears whenever a model is adjusted to pass a threshold.

‍

Reweighting features until selection rates even out produces a statistical pass. Where nobody can then explain why the resulting criteria predict job performance, the justification argument has been weakened in exchange. A model tuned to a metric with an unexplainable criteria set is in a worse legal position than one with a modest disparity and a documented rationale.

‍

What Does That Argue For?

‍

Recording the reasoning at the point criteria are selected rather than reconstructing it after an audit result. The rationale for including a criterion is available while somebody is deciding to include it and considerably harder to produce eighteen months later, which records that survive an audit covers as a general requirement.

‍

What Does the Proxy Problem Add?

‍

A failure mode that survives the obvious remedy, which is why removing demographic variables is necessary and not sufficient.

‍

Scrubbing explicit attributes lets a model pass a blind parity test while correlated features continue to carry the same information. Postcode, education history, employment history breaks and activity patterns all correlate with protected characteristics to varying degrees, so a model with no demographic input can still produce a demographically uneven outcome. Categorizing which systems carry this exposure is the prior step.

‍

How Is That Detected?

‍

Test the outcome rather than the input, which requires holding demographic data in order to measure disparity against it. The result is an awkward position where measuring fairness requires collecting the attributes the model is not permitted to use, and resolving it is a data protection question as much as a fairness one, which what data an obligation requires runs into.

‍

Who Owns This Inside an Organization?

‍

Three functions with no shared view, which is why the compounding effect goes unmeasured.

‍

Whoever procured the tool holds the vendor audit and reads it as evidence of compliance. Employment counsel holds the legal exposure and rarely sees the model documentation. Recruiting operations holds the pipeline data that would reveal the combined selection ratio and has no reason to compute it. Each position is reasonable and the union leaves the real question unanswered, which an obligation split across functions describes.

‍

What Is the Smallest Fix?

‍

Computing the end-to-end selection ratio once, which recruiting operations can do from data it already holds. The figure establishes whether the compounding effect is a live problem or a theoretical one, and it costs an afternoon rather than an engagement.

‍

Does Human Review Fix the Problem?

‍

Not on its own, and the assumption that it does is what leaves the compounding effect unexamined.

‍

A recruiter reviewing model output is frequently presented as the safeguard that makes an automated screen acceptable. Where the recruiter sees a ranked list and works down it, the review reproduces the model's ordering rather than correcting it. Where the recruiter applies their own judgment freely, they introduce a second selection stage with its own effect, which compounds with the first rather than offsetting it.

‍

Which Version Is Better?

‍

Neither reliably, and the answer depends on measurement rather than on design intent. A review stage that rarely departs from the model's ordering is documenting the model's decisions with a person's name attached. One that departs frequently is making decisions the model's audit says nothing about, and whether the review is doing anything is the measurement that distinguishes them.

‍

What Should Be Recorded About the Review?

‍

The departure rate, meaning how often the human outcome differs from the model's ranking. The figure tells you whether you have one selection stage or two, which determines whether the compounding arithmetic applies at all and is available from data recruiting already holds.

‍

What Should Be Established First?

‍

Four things, in this order, because the first frequently makes the rest urgent.

‍

The end-to-end selection ratio from application to offer, per role rather than in aggregate. Whether the vendor audit covers your role and your applicant pool or a global population. Whether the rationale for each criterion is recorded anywhere. Then whether the two obligations have separate owners who have spoken to each other. AI compliance readiness assessed per requirement rather than per framework is what keeps the two tests distinguishable.

‍

Two Tests, Two Subjects

‍

The statistical test examines the tool and the legal test examines the outcome, so a vendor demonstrating even selection rates across a global population has said nothing about the role you are filling. Selection effects compound across stages, so a pipeline can fail while every component in it passes, and an audit scoped to the automated part leaves that unmeasured by design. The evidence sets barely overlap, since one is a measurement and the other is documentation about why criteria were chosen and what alternatives were considered. Tuning a model to pass a threshold without being able to explain the resulting criteria trades a statistical pass for a weaker justification. Kovrr's AI Security and Governance Platform records which systems influence decisions about people and what evidence exists for each.

‍

To see which AI systems in your environment influence decisions about people and what each can evidence, book a demo mapped to your own estate.