Aviation and Banking Already Solved Hallucination

“Hallucination” is a cognitive metaphor dressed up as an engineering term. It imports the assumption that the problem lives inside the model — in how a neural network generates tokens — and that fixing it requires making the model “smarter” or more “truthful.” This framing is both technically incorrect and practically paralyzing. It is incorrect because recent theoretical work establishes that generation and verification are asymmetric: verifying a claim is never harder than generating a correct one, and for the checkable class it is strictly easier. It is paralyzing because it directs attention toward an unsolvable single-model problem instead of a solvable system-level one.

The productive reframe is: a hallucination is the absence of a certificate. An output that a system cannot stand behind with a traceable guarantee is an output that should not be emitted, or should be emitted with an explicit, honest uncertainty signal. The question shifts from “will the model get this right?” to “what does the system guarantee about what it sends?” That second question has a known solution shape. The model is the untrusted generator; the architecture around it is the certifying layer. Users never experience a forward pass — they experience a delivered output. The delivered output is what must carry a guarantee.

This reframe does not deny that some errors are irreducible. It denies that irreducibility is an excuse for uncontrolled error. Aviation does not claim that sensors never fail; it claims that the system will detect a sensor failure and alert the crew before they act on bad data. That is a different and achievable guarantee. It is also, structurally, exactly what a governed AI system provides.

Once the question is reframed as a system guarantee, a clean split emerges. Some claims have an answer that can be checked exactly at the moment of answering: schema conformance, arithmetic, whether a cited document contains the sentence attributed to it, whether a request violates a written policy. For these claims, an independent verifier either confirms or rejects the model’s output. The generator’s opinion is irrelevant; only the verifier’s verdict is emitted. Under a gate that is sound (it never passes a false claim it can check) and default-deny (it abstains when it is uncertain whether a check applies), the rate of falsely asserted claims on this checkable class is zero — relative to the verifier’s specification. Not low. Zero. (A wrong verifier is a named, fixable defect, not a hallucination.)

The second bucket contains everything else: open-world facts, judgments, predictions, claims about contexts the system cannot fully verify. No exact oracle exists for these, so elimination is not available. The correct target here is not zero error but a bounded, measured, controlled error rate. The system must quantify what it does not know, commit to a tolerance, and abstain rather than fabricate when it cannot meet that tolerance. This is not vagueness; it is a mathematically precise regime with a distribution-free finite-sample guarantee, developed independently in the statistical learning literature.

The boundary between the two buckets is itself an engineering problem, and it is the most critical one: a checkable claim misrouted to the probabilistic path exposes a guarantee that should have been exact. The correct discipline is default-deny at the boundary — when in doubt about which bucket a claim belongs in, treat it as uncheckable and abstain. This is the same conservative posture aviation applies when an aircraft exits its operational envelope, and the same one banking applies when a model encounters inputs outside its calibration range. When the information required to certify a claim simply does not exist, “I don’t know” is not a failure mode. It is the only sound response.

Civil aviation has reduced the rate of catastrophic system failure to roughly one in a billion flight hours. The discipline that achieved this is not, primarily, about making components more accurate. It is about guaranteeing that when a component is wrong, the system detects it and says so before anyone acts on the bad data. Aviation calls this distinction — between how often a system is wrong and how often it is wrong without warning — the difference between accuracy and integrity. In navigation, the specific failure mode called Hazardously Misleading Information (HMI) is wrong data delivered without an alert. The entire integrity apparatus — redundant sensors, independent monitors, mandatory alerting, forced reversion — exists to prevent exactly this.

The architecture divides along the same two-bucket line described above. Systematic failures — defects in software logic, requirements errors, design flaws — are not assigned a probability. They are driven to zero through design assurance: requirements-based verification, complete bidirectional traceability from requirement to test, and structural coverage analysis. Any residual defect is reattributed to a named requirement, making it auditable and fixable rather than diffuse and deniable. Random failures — hardware wear, sensor noise, irreducible physics — are assigned a probability budget scaled to consequence. A failure that could cause a catastrophic outcome is held to a tolerance of one in a billion per flight hour. A failure with minor consequences tolerates a much higher rate. The threshold is set by what a failure costs, not by what is convenient to achieve.

What aviation calls integrity, a governed AI system calls the undetected error rate. The target is not zero false outputs — that is unachievable on open-world claims. The target is zero false outputs that escape the monitor undetected and reach the user as confident assertions. An output that is uncertain but labeled as uncertain is not a hallucination in the meaningful sense. An output that is wrong and delivered without warning is. Aviation built a global safety record on this distinction. The AI field has largely ignored it.

Banking arrived at a parallel answer through a different route and under different regulatory pressure. The 2008 financial crisis traced partly to unchallenged model outputs — risk estimates that nobody was structurally positioned to question — propagating through large institutions until the damage was irreversible. The regulatory response in the United States was SR 11-7, a Federal Reserve and OCC guidance document on model risk management that remains the governing spine of how regulated banks treat any quantitative model, including, explicitly, AI and machine learning systems.

The core principle of SR 11-7 is effective challenge: the party that validates a model’s outputs must be structurally independent of the party that built it. A model cannot validate itself; a team cannot mark its own homework. This is not bureaucracy — it is the same independence principle aviation builds into its command-monitor architectures and independent verification teams. A language model grading its own factuality inherits its own blind spots — a failure aviation and banking both treated as axiomatic and engineered around. The in-depth companion takes up the evidence.

Banking also formalized what it means to have a validated bound rather than an asserted one. Value-at-Risk is not just a number a bank reports; it is a number subject to daily backtesting that compares the model’s predicted loss quantile against actual outcomes. Regulators count the exceptions and apply a traffic-light regime: too many exceptions over a rolling window trigger mandated recalibration and capital add-ons. The bound is not a claim; it is a measured, regularly verified commitment. The same discipline — state a tolerance, measure it against reality, recalibrate when it drifts — is exactly what a governed AI system must apply to its probabilistic output bucket. An AI vendor who asserts a low error rate without a validation regime to back it is, in banking terms, an unvalidated model in production. That is an examiner finding, not a feature.

One further lesson from banking is worth naming: you cannot outsource the obligation to govern. SR 11-7 covers vendor and third-party models explicitly. Buying an AI system does not transfer the regulatory duty to the vendor. The bank that deploys an AI model it cannot explain, validate, or audit owns the risk, regardless of who built the model. This is the correct posture, and it is applicable far beyond banking.

The synthesis is concise. A governed AI system makes three guarantees, not one. First, zero false assertions on anything that can be deterministically checked — schema, arithmetic, policy conformance, citation resolution — because these are governed by an exact verifier, not by the model’s judgment. Second, a bounded and validated error rate on everything else, stated as a measured commitment subject to ongoing backtesting, not as an aspiration. Third, every output traced to the rule or data that authorized it — so that any output can be reconstructed, explained, and audited after the fact. These three guarantees map precisely onto what aviation calls design assurance, integrity monitoring, and bidirectional traceability, and onto what banking calls deterministic controls, VaR backtesting, and audit trail. The disciplines are not analogies. The underlying mathematical structure is the same object, independently rediscovered under regulatory compulsion.

The architecture is also, it turns out, over-determined by ethics. Two ethical traditions that ordinarily reach opposite conclusions — consequentialism and Kantian deontology — converge on the same design. The Kantian argument is that emitting a confident false claim treats the user merely as a means, violates a duty of honesty that cannot be traded against aggregate accuracy, and fails to universalize. The consequentialist argument is that undetected false assertions in high-stakes decisions produce concentrated harm that accurate-on-average statistics systematically hide. Both arguments, from different starting points, arrive at the same requirement: bound the undetected error, not the average one; emit calibrated uncertainty, not manufactured confidence; abstain when the evidence is insufficient. An architecture that two rival ethics traditions both demand is robust to your choice of moral framework. That is the strongest available validation of a design.

This is not a constraint on capability. It is the precondition for deploying capability where the stakes are real. Aviation’s safety record did not come from flying less; it came from engineering a system that could fly reliably because every component was accountable to a defined guarantee. Banking’s model risk discipline did not constrain trading — it made the risk legible enough to be managed. The same is true of AI: govern it like a safety-critical system, audit it like a regulated one, and the capability you deploy is capability you can stand behind.

The in-depth companion to this brief develops the full argument: the mathematical framework for bounded undetected error, the exact regulatory and standards cross-references, the ethics in formal detail, and the architectural primitives that implement the discipline in a deployable system. Read it at Governed Like Aviation, Audited Like Banking .