Healthcare AI Governance: When Workflow Decides Device Status

An AI governance framework classifies systems by what they are. Healthcare regulation classifies them by how clinicians use them, and the same software can fall inside or outside device regulation depending on the clinical workflow it sits in.

‍

The distinction makes healthcare AI governance different from the sector-agnostic version rather than harder. A decision made by a clinical team about where a tool appears in a workflow, and how much time a clinician has to consider its output, can change the regulatory status of a product nobody modified.

‍

Device Status Turns on Four Criteria

‍

Software supporting clinical decisions is excluded from the definition of a device only if it satisfies all four criteria in the relevant section of the Food, Drug and Cosmetic Act. Failing any one puts the software into device territory.

‍

It must not acquire, process or analyze a medical image, a signal from an in vitro diagnostic device, or a pattern from a signal acquisition system. It must display, analyze or print medical information of a kind normally communicated between health care professionals. It must support or provide recommendations about prevention, diagnosis or treatment. It must also enable the clinician to independently review the basis for those recommendations, so that the clinician does not rely primarily on the output.

‍

The First Criterion Is Absolute

‍

Software taking a medical image, an in vitro diagnostic signal or a waveform from monitoring equipment as input remains a device regardless of how the other three are satisfied. No amount of transparency or human oversight recovers the exclusion, which makes the input type the first thing to establish about any clinical tool, and structured categorization should capture it at intake.

‍

The Fourth Criterion Is About Workflow

‍

Independent review is where the governance question lives, because it depends on presentation and context rather than on the algorithm. The regulator's position is that software intended for critical, time-sensitive decisions generally does not meet this criterion, on the reasoning that clinicians in those settings lack the time to review the basis for a recommendation.

‍

The Same Tool Can Land Either Side

‍

A sepsis prediction tool presented in a chart review workflow, where a clinician can see which inputs drove the output and has time to weigh them, sits differently from the same model firing an alert into an emergency department requiring immediate action. The vendor's classification describes the intended use they declared, and the deployment decides whether that description still holds. European rules apply a related test, since intended purpose drives classification there too.

‍

What Satisfying It Requires

‍

The regulator recommends the software or its labeling identify the intended use and users, the patient population, the required inputs, and a plain-language description of how the algorithm was developed and validated. A hospital cannot supply those on a vendor's behalf, which turns them into procurement questions rather than internal documentation tasks.

‍

A Recent Revision Loosened One Criterion

‍

Guidance in this area was revised recently and the change runs in the opposite direction to what most summaries suggest. The earlier interpretation required software to present recommendations as a list of options, which excluded any tool giving a single specific output.

‍

The earlier reading forced developers to construct artificial choice even where clinical guidelines pointed to one appropriate path, and it has been relaxed. The practical effect is that a tool recommending a single course of action is no longer disqualified on that ground alone, which widens the population of software that can sit outside device regulation while leaving the independent review requirement untouched. Anyone working from a summary written before the revision should check it against the current text.

‍

Certified Health IT Gives You a Procurement Lever

‍

Certification requirements for health IT now include a decision support criterion covering predictive interventions, defined as technology producing a prediction, classification, recommendation, evaluation or analysis from models trained on data.

‍

Thirty-One Source Attributes

‍

Certified technology supporting predictive decision support must surface a defined set of source attributes, substantially more than the set required for evidence-based interventions. Those cover the kind of information a governance committee needs and rarely receives, including how the model was developed, what it was validated against and what its known limitations are.

‍

The governance value is that this information becomes available by right rather than by negotiation, at least for tools inside certified health IT. Building an intake process that collects these attributes for every clinical AI tool, certified or not, gives a committee a consistent basis for comparison, and an inventory built to hold them is where the answers belong.

‍

Three Regimes Overlap and None Subsumes the Others

‍

A health system running clinical AI answers to several authorities at once, and the obligations are different in kind rather than in wording.

‍

• Device Regulation: Governs whether the software may be marketed and used at all, and turns on the four criteria and intended use.
• Health Information Privacy: Governs the data flowing through it, including vendor arrangements where a model processes protected information.
• State AI Statutes: Govern transparency and human review, with several states now imposing requirements on clinical documentation tools specifically.

‍

Privacy obligations carry their own enforcement pattern, and regulators have moved from asking whether an organization performed a risk analysis to what it did about the findings, which healthcare risk analysis and enforcement covers in detail. A tool can be entirely outside device regulation and still generate substantial exposure under the other two, which is why compliance readiness assessed per requirement rather than per framework is the workable structure.

‍

Scribes Are the Volume Problem

‍

Ambient documentation tools are the most widely deployed clinical AI in most health systems and the least governed, because they are framed as productivity software rather than as clinical tools.

‍

The framing holds only while the output stays out of the record unreviewed. A note generated by a model and signed without substantive review becomes part of the clinical record and the basis for downstream decisions, which is a different object from a draft. Several states have moved to require human review explicitly. Recording review as an attested step rather than assuming it, and monitoring whether attestation rates are plausible, is the control that separates the two situations.

‍

Data Flow Deserves the Same Attention

‍

These tools process highly sensitive conversation in volume, frequently to a third-party model. Whether that arrangement is covered by an appropriate vendor agreement, what retention applies and whether recordings are used for model improvement are questions with definite answers that are rarely asked before deployment. How regulated data reaches AI tools applies with more force here than in most sectors.

‍

What a Governance Committee Should Record

‍

Four attributes per clinical AI tool answer most of the questions that arrive later, and none requires specialist input to capture.

‍

Intended use as the vendor declared it, stated precisely enough to compare against actual deployment. Input types, since one of them settles device status on its own. The clinical workflow position, covering how much time a clinician has and whether the basis for the output is visible. A named clinical owner completes it, distinct from the technical owner, since the two answer different questions. Programs treating intended purpose as the unit of inventory rather than the tool will find this natural, because the same product deployed twice produces two records.

‍

Re-examine When Deployment Changes

‍

The classification is only as current as the workflow it describes. Moving a tool into a faster clinical setting, removing a review step or extending it to a new department all change the analysis without changing the software, so a workflow change should trigger re-examination in the same way a version change does.

‍

Classify the Deployment, Not the Product

‍

Healthcare AI governance breaks when it inherits the assumption that regulatory status is a property of software. Device status depends on input types and on whether a clinician can genuinely review the basis for an output in the time available, which makes it a property of the deployment. Recording intended use, input types, workflow position and a named clinical owner per deployment is what lets a committee answer the question correctly and notice when the answer changes. Kovrr's AI Security and Governance Platform holds those attributes against each system, so a workflow change surfaces as a governance event rather than as an operational one.

‍

To see your clinical AI estate recorded by deployment and intended use rather than by product, book a demo mapped to your own environment.