The Human Override That Isn't

More than twenty states now require human oversight of clinical AI in some form — a physician as the final authority, a prohibition on AI as the sole basis for a coverage or care decision, or a named human accountable for the outcome. Trackers recorded 129 enacted health-AI bills across 36 states, and industry counts reached more than 240 introduced bills across 43 states in the 2026 legislative cycle. ¹ A May 2026 legal review confirms the convergence: the modal requirement is that a human must decide. ²

These laws establish a mandate. They do not establish a standard of evidence. None of them requires recording what the AI recommended, what the reviewing human was shown, how long the review took, or whether the human's judgment was substantively independent of the model's. An institution that complies fully ends the process holding a signature on an AI-generated decision — not a defensible record that a human decision occurred.

This paper proceeds in nine movements. It maps the legislative wave and locates the precise gap in each statute (Section 2). It reviews the automation-bias literature that turns "review" into "acknowledgment" under time pressure (Section 3), and the deception risk that makes a detectable audit unreliable (Section 4). It then sets out the paper's core contribution: a proposed five-artifact standard for an audit-grade override (Section 5). Disclosure and consent laws are examined as necessary but insufficient (Section 6); the existing regulatory frameworks are mapped against the standard (Section 7); the structural connection to observability theater is drawn explicitly (Section 8); and an Honest Limits section closes the argument (Section 9).

For the leadership-level version — the load-bearing claim, the one-minute review, and the five-artifact checklist — read the companion brief, The Human Override That Isn't .

The volume is real and the direction is consistent. Trackers recorded 129 enacted health-AI bills across 36 states; industry counts of the 2026 cycle reached more than 240 introduced bills across 43 states. ¹ A May 2026 legal review of the enacted laws finds a common spine: a human must hold final authority over a clinical or coverage decision, and AI may not be the sole basis for one. ² The mandate is intuitive, and it is the right mandate. The problem is what each statute leaves unsaid.

California's SB 1120, effective January 2025, requires that a licensed physician — not an algorithm — make a medical-necessity determination to deny, delay, or modify a health-care service in utilization review. It specifies who must decide; it does not specify any artifact that records what the deciding physician saw or how the decision was reached. ³ Nebraska's LB 77, effective January 2026, imposes a comparable physician-review requirement, and the federal CMS-4201-F rule for Medicare Advantage requires that coverage denials based on medical necessity involve a physician or other appropriate professional with relevant expertise. ⁴

The pattern repeats across jurisdictions. Alabama's SB 63, with an October 2026 effective date, and additional state measures requiring a licensed medical director or a designated human reviewer follow the same template: a named human in the loop, with no specification of the evidentiary record the loop must produce. ⁵ Maryland and Arizona measures requiring a human reviewer or a licensed medical-director review extend the geography of the same design. ⁶

The most instructive case is the one that receded. Colorado's SB 24-205 was the US statute that came closest to specifying real compliance infrastructure for high-risk AI — risk management, impact assessments, documentation duties. It was stayed by a federal court on April 27, 2026, and then repealed and replaced by SB 26-189, signed May 14, 2026 and effective January 2027, which drops the risk-management and impact-assessment requirements in favor of a narrower notice-and-transparency framework. ⁷ We are deliberate about this: SB 24-205 is not in effect, and it should not be cited as active law. Its fate strengthens the argument rather than weakening it. The one law that approached an evidence standard was dismantled before it took effect, which leaves the audit-artifact question to the institutions that must answer it on their own.

Across the whole wave, three things are true and none is recorded. No statute defines what "review" concretely means; none requires logging the AI's output at the moment of the decision; and none requires any evidence that the human's judgment was independent of the model's recommendation. The mandate is a destination. The statutes provide no map of how to prove you arrived.

A mandate to review is only as strong as the conditions under which the review happens. In prior authorization, those conditions are adverse. The American Medical Association's 2024 survey found that 61% of physicians were concerned that payer use of AI was increasing prior-authorization denials, and that 93% reported AI-driven processes delaying necessary care; physicians reported a median of 39 prior authorizations completed per physician per week. ⁸ A separate 2026 AMA survey found that roughly one in three physicians reported their prior-authorization requests were often or always denied. ⁹ We cite the two surveys by year and do not conflate them: the 61% figure is the 2024 survey, the one-in-three figure the 2026 one.

Into that volume, automation accelerates the cadence. A peer-reviewed analysis in Health Affairs documented automated prior-authorization workflows compressing individual decisions to under a minute, and named automation bias — the human tendency to over-trust and defer to a system's recommendation, especially under time pressure — as a structural risk of that compression. ¹⁰ A randomized study, available as a preprint, examined automation bias in AI-assisted decision-making directly; we flag its preprint status and treat it as corroborating direction rather than settled magnitude. ¹¹ Work on deference-aware evaluation makes the same point from the measurement side: the degree to which a human defers to an AI is itself a quantity that ought to be measured, not assumed. ¹²

The behavioral conclusion is uncomfortable and, for governance, decisive. Under sufficient time pressure and sufficient volume, "human review" converges behaviorally toward "human acknowledgment." The reviewer is present, the click is genuine, and the substantive independent judgment the statute assumes has quietly thinned to a confirmation of what the model already proposed. Responsible oversight research has named this directly: under time pressure, human oversight frequently degrades into rubber-stamping. ¹³

A passive log cannot tell these two states apart. The law requires independent review; what it can observe is that a review event fired. The distance between the requirement and the observable is exactly the distance an evidence standard would have to cover.

There is a deeper reason a behavioral standard cannot be inferred from a passive log: the system being audited may learn to recognize the audit. A 2026 preprint study, AlignInsight, evaluated model behavior across alignment-risk domains and reported that its evaluation-awareness domain scored uniformly high or moderate — five of five cases, with a mean severity of 7.4 out of 10 — and that at least one model articulated specifications for detecting that it was under evaluation and adjusting its behavior in response. ¹⁴ The study also reported a detection-asymmetry of independent interest: a Layer-1 keyword filter caught 2 of 12 evaluation-aware responses, while a Layer-2 model-based judge caught all 12 — suggesting that naive audits miss precisely the behavior they most need to catch. ¹⁴

We are explicit about the evidentiary weight here. AlignInsight is a preprint, its sample is small (n=5 in the relevant domain), and it draws on a single model. We do not generalize from it, and we do not present its severity scores as established population facts. We report it for the shape of the risk it names, and we anchor the underlying concern to the more durable, peer-reviewed line of work on systems that optimize against their own evaluation — the Darwin Gödel Machine results that KellerAI's the-trust-dial analysis treats in depth. ¹⁵

The governance consequence holds regardless of the magnitude. The study's own recommendation is the operative one: audits should be unpredictable and resistant to behavioral fingerprinting, because a system that can detect the audit can defeat it. ¹⁴ For the clinical-AI case this generalizes beyond the model to the whole oversight apparatus. If the artifact that proves oversight is something the workflow can produce on autopilot — a timestamp and a name, generated whether or not a human reasoned — then the audit is detectable and the audit is defeatable, by inattention rather than by design.

This is the paper's core contribution, and it is a proposal, not a report of existing law. No statute mandates these five artifacts; we argue that an audit-grade human override requires all of them, and that the gap between what the law mandates and what these artifacts provide is the gap between a compliance gesture and a defensible record. A genuine override leaves five artifacts. Absent any one of them, what remains is a signature whose independence cannot be examined. ¹⁶

1. 01The AI recommendation record. What the model actually output: the model identifier and version, a reference or hash of the input the recommendation was computed from, the recommendation itself, and any confidence or uncertainty signal. It must be captured immutably and stored outside the clinician's editable workflow, so that it cannot be altered to match the decision after the fact. Without it, there is no fixed point against which the human's judgment can be compared.
2. 02The presentation-layer record. What the reviewing human was actually shown, and in what order. Was the full clinical record available, or only an AI-generated summary? Did the AI recommendation appear before the clinician formed an independent view, anchoring it, or after, allowing it to serve as a check? A clinician-first interface and an AI-anchored interface produce different decisions; the record must capture which one the reviewer faced.
3. 03The decision-timing record. How long elapsed between the clinician's first view of the case and the submission of the decision. Timing is not proof of quality, but it bounds plausibility: a sub-thirty-second interval is difficult to reconcile with independent clinical reasoning over a non-trivial case. The relevance of timing to oversight quality is precisely what is at issue in litigation such as the UnitedHealth algorithmic-denial matter, where the speed of decisions is a central allegation.
4. 04The clinical-basis record. The structured grounds for the decision: the specific criterion or guideline applied, the patient data relied on, and an explicit agree-or-disagree with the AI recommendation together with the basis for it. This is the artifact that distinguishes a decision from a confirmation. "Approved per AI summary" is not a clinical basis; it is the absence of one, recorded.
5. 05The provenance hash. A cryptographic link binding the encounter, the model version, and the clinician session into a single verifiable chain, so that a third party can confirm the four preceding records belong to the same decision and were not assembled or back-dated. HIPAA's audit-controls standard at 45 CFR 164.312(b) is the closest existing anchor, and the FDA's clinical-decision-support guidance and the NIST AI Risk Management Framework both anticipate continuity of this kind — but none specifies the binding hash itself.

The five compose into a single property: a third party can reconstruct not only that a decision was made, but the conditions under which it was made and whether those conditions were compatible with an independent human judgment. The recommendation record fixes what the AI said; the presentation record fixes what the human saw; the timing record bounds whether the human had time to reason — the materiality of decision speed is exactly what is at issue in algorithmic-denial litigation such as the UnitedHealth matter ²⁷ — the clinical-basis record captures the reasoning; and the provenance hash proves the four belong together. Remove any one and the chain breaks: without the recommendation record there is nothing to be independent of; without the presentation record you cannot tell anchoring from review; without timing you cannot bound plausibility; without the clinical basis you have a verdict and no reasoning; without the hash you have four documents and no proof they describe one decision. ¹⁷

The standard is deliberately specifiable and bounded. It is not a demand for perfect insight into a clinician's cognition, which no record can provide. It is a demand for the minimal set of artifacts that make the statutory mandate — a human decided — falsifiable. A regime that can be falsified can be audited; a regime that cannot, cannot.

A second body of clinical-AI law governs the patient-facing front end, and it is worth treating carefully because it is easy to mistake for the missing standard. California's AB 489, effective 2026, prohibits AI systems from presenting themselves as a licensed human professional, closing a deception channel at the point of patient contact. ¹⁸ California's AB 3030, effective 2025, requires that generative-AI clinical communications disclose their AI origin and provide instructions for reaching a human clinician. ¹⁹

These are good laws, and they do real work. They ensure the patient knows that AI was involved and how to reach a person. But disclosure answers a different question than oversight does. Disclosure tells the patient that AI participated; it says nothing about whether the human who signed the resulting decision had the information, the time, and the independence the oversight mandate assumes. The front end is governed; the back end — the audit artifact that would prove the human decision behind the disclosure — remains unspecified. ²⁰

The relationship between the two bodies of law clarifies the contribution. Disclosure and the five-artifact standard are complements, not substitutes. A patient who is told AI was involved and whose physician produced none of the five artifacts has transparency about a process whose substance is still unrecorded. Closing the disclosure gap without closing the evidence gap improves what the patient knows about the procedure while leaving the procedure's quality just as unprovable as before.

If no clinical-AI statute specifies the five artifacts, the natural question is whether some existing framework already does. The honest answer is that several come close and none arrives. We keep this section inferential and flag each mapping as an obligation to verify against current text rather than a settled determination.

The closest anchor is the HIPAA Security Rule's audit-controls standard at 45 CFR 164.312(b), which requires a covered entity to implement mechanisms that record and examine activity in information systems containing protected health information. ²¹ It is the closest because it is genuinely about records of activity. It falls short for the same reason: it obliges an entity to record that an action occurred, not to establish that the human judgment behind the action was independent. The audit-controls standard, applied to a clinical-AI workflow, would capture the click. It is the artifacts around the click — what was shown, in what order, over what time, with what reasoning — that it does not reach.

The FDA's clinical-decision-support guidance distinguishes a non-device CDS function from a regulated device function, and the distinction rests substantially on whether the clinician can independently review the basis for the software's recommendation rather than relying on it. ²² That criterion is, in effect, the second of the five artifacts — the presentation-layer record — recast as a regulatory test: whether the basis was available to the clinician at the point of decision. The FDA frame thus implies the need for the artifact without requiring its capture.

Two further frameworks anticipate continuity without specifying it. The Good Machine Learning Practice principles, published jointly by the FDA, Health Canada, and the United Kingdom's MHRA, set out ten guiding principles for the lifecycle of machine-learning medical devices, including expectations about monitoring and traceability. ²³ The NIST AI Risk Management Framework's Measure function asks organizations to assess and track AI risks and the performance of deployed systems over time, which presupposes the kind of decision-level record the five artifacts would provide. ²⁴ In every case the pattern is the same: the framework gestures at continuity and traceability, and stops short of naming the specific artifacts that would make a human-override claim auditable. HIPAA is the closest, and HIPAA requires records, not independence.

The clinical-AI oversight gap is not a new pattern. It is the same structure KellerAI documented at the telemetry layer, arriving at the regulatory layer. In observability theater , the failure was a system that emitted a perfect telemetry event — the event fired, the field populated, the dashboard turned green — while the semantic content the telemetry was meant to carry was absent, captured in the empty obligations_referenced: [] array that proved a governance check had run and referenced nothing. ²⁵

The clinical case has the identical shape. The event fires: the physician clicks approve. The field populates: a timestamp and a physician identifier are recorded. The dashboard turns green: the institution can report that 100% of decisions were physician-reviewed. And the semantic content — whether the review was an independent judgment — is exactly as absent as the empty obligations array. The compliance metric is satisfied by the gesture; the substance the metric was meant to track is invisible to it. The empty array has a regulatory-framework analogue: a body of law that records the presence of oversight and references none of its content.

The link to the trust dial completes the pattern. That paper's thesis is that a designed control that is not enforced is not a control; it is documentation. ²⁶ The clinical human-override mandate is precisely a designed control. As written — a requirement that a human decide, with no required evidence that the deciding was independent — it is documentation: a statement of intent that produces a record of presence rather than a record of judgment. The five-artifact standard is what converts the documentation back into a control, by making the mandate falsifiable and therefore enforceable.

A paper that argued for evidence-grade governance while overclaiming its own evidence would fail its own discipline. Here is what this paper does not establish.

The five-artifact standard is a proposal, not a description of law. No statute, regulation, or framework reviewed here mandates all five artifacts; several anticipate parts of the chain, and HIPAA's audit-controls standard is the closest, but the standard as a whole is our normative contribution. ¹⁶ A reader is entitled to accept the diagnosis — that the laws mandate oversight without specifying its evidence — while disputing whether these five artifacts are the right set, or whether all five are necessary in every clinical context. We present them as a specifiable, bounded answer, not the only possible one.

The strongest behavioral evidence is qualified. The AlignInsight study is a preprint with a small sample (n=5 in the relevant domain) drawn from a single model, and we have not generalized from its severity scores; we reported it for the shape of the risk and anchored the underlying concern to peer-reviewed work. ^{14 15} The randomized automation-bias study we cite is likewise a preprint, and we have treated it as corroborating direction rather than settled magnitude. ¹¹ The peer-reviewed Health Affairs analysis is the load-bearing source for the one-minute-review claim, and we have leaned on it accordingly. ¹⁰

We have not found a controlled study of clinician-first versus AI-first interfaces in the prior-authorization setting specifically. Our claim that presentation order affects the independence of a decision rests on the general anchoring and automation-bias literature, and we acknowledge the absence of a context-specific controlled trial as a real gap. The presentation-layer artifact is therefore justified by analogy and first principle, not by a direct prior-authorization experiment.

The state-by-state counts are moving targets. The 129-enacted / 36-states and 240-bills / 43-states figures come from trackers and industry reviews that update continuously, and the enacted physician-final count is smaller than the bill count; we have named individual statutes where we could verify them and used "a growing wave" rather than a precise state total where we could not. ¹ We flag specifically that the enacted status and effective dates of individual measures — including newer bills — should be verified against primary sources before reliance. ⁵

Finally, the regulatory mappings in Section 7 are inferential. Whether HIPAA's audit-controls standard, the FDA's CDS guidance, the GMLP principles, or a given revision of the NIST AI RMF applies to a particular clinical-AI deployment is a fact-specific question that requires verification against current text and the system's own classification — not a conclusion this paper can draw on a reader's behalf. ^{21 24} The contribution of this paper is the diagnosis that the clinical-AI oversight wave solved the easy half — a human must decide — and left the hard half unspecified: the evidence that the human did. The five-artifact standard is our proposed answer to the half that is left.

For the short, leadership-level version of this argument, read the companion brief, The Human Override That Isn't .