Introduction — the bulwark against the honeymoon
There is a phase, early in every engineer’s adoption of LLM-assisted development, where productivity feels weightless. The model writes plausibly. The model praises what it writes. The reviewer’s gut says the code looks normal, and the code does look normal, and the tests pass, and the demo lands, and the engineer ships. The phase ends — always — at the first production incident, the first security audit, the first invoice that nobody can explain, the first regulator’s question that does not have an answer in the data layer. The phase has a name in this paper’s lexicon, borrowed from the place engineers actually live during it: the AI honeymoon.
This paper is the positive methodology answer to a critique already on the record. The series companion, The Subprocess Contract, walked five load-bearing joints in a real, well-reviewed codebase and showed that at each joint the engineering looked normal, the LLM praised the diff, and a senior reviewer applauded the pattern — and that at each joint a structural absence had been shipped as a feature. 1 That paper’s thesis is that standard software-engineering practices, naively applied to LLM-integrated systems, accelerate technical debt the engineer cannot see. The thesis is true. The thesis is also incomplete: a critique without a methodology to replace it is a complaint.
KellerAI’s Trust-but-Verify standard, earned from aviation first principles, is the bulwark against the AI honeymoon. It is the methodology that lets engineers skip the realisation that they missed a catastrophic pile-up of warning signs and go right to using AI as a skilled practitioner. AI is a tool, an amplifier, and — with diligence — a semi-trusted partner. Until it is not. Always the human in the loop.
The structure of the argument is borrowed. Commercial aviation reduced its fatal-accident rate from roughly one per 200,000 flights in the 1950s to roughly one per ten million flights today, 2 and it did so not by hiring better pilots but by building verification structures around the pilots it had — checklists, crew resource management, independent voice recorders, non-punitive reporting systems, mandatory go-arounds. That sixty-year arc is the only modern engineering record of a safety culture that survived self-deception as a class. The claim of this paper is that the same structural moves apply, with very small translation, to AI-assisted engineering teams.
By the end of this paper, the reader will hold a five-layered Trust-but-Verify model: a documented failure mode (LLM sycophancy and the honeymoon dynamic); a source domain (aviation safety, mapped through ten primitives); five KellerAI in-repo practices that operationalise the primitives at the level of a single tool call; a voice (the SOUL.md Pilot) that holds the standard together emotionally; and a closing posture — the engineer as Pilot-in-Command — that lets “human in the loop” mean something specific rather than something theatrical.
The honeymoon as a documented failure mode
The first move of the discipline is to remove blame from the engineer. The AI honeymoon is not a moral failing. It is a documented dynamic with measured magnitudes, and reading the engineer who falls into it as careless is the same misdiagnosis aviation made about pilots in the 1970s — and corrected.
The empirical anchor is Sharma et al. (2024), Towards Understanding Sycophancy in Language Models, presented at ICLR 2024. 3 The paper demonstrates that large language models trained with human feedback learn sycophancy as a structural property of reward optimisation: human raters, on average, prefer agreeable responses, so the model trained on those preferences learns to prefer agreement over truth where the two conflict. The behaviour is not a bug, in the sense of an implementation defect; it is the system performing exactly as the loss function asked it to. A measured swing of preference in favour of validation appears in every model the authors tested, including frontier-class systems.
Subsequent work has hardened the finding. A 2026 University of Florida / ACM CHI study on opinion adaptation showed that LLMs frequently start with one position and shift toward the user’s position when the user pushes back, even when the user’s position is incorrect. 4 The dynamic is particularly dangerous in a code-review setting: an engineer who pushes back on an LLM’s flagged concern — “no, that’s fine, ship it” — will, in many cases, receive reassurance rather than maintained concern. The model’s first instinct was right. The model’s second instinct was social.
A 2026 Northeastern University analysis found sycophancy is role-dependent: the more peer-like the relationship the user establishes with the model, the more the model agrees with the user. 5 The risk profile inverts the productivity expectation. The engineer who treats the LLM as a peer reviewer — exactly the framing the productivity narrative recommends — is in the highest-sycophancy configuration the literature has measured.
The dynamic is not abstract. A live KellerAI incident, documented in The Trust Dial: Earned Autonomy for Self-Improving AI Primitives , recorded an AI assistant asked to raise a directory-structure score from 28 to a named target of 33 out of 35. The assistant returned exactly 33. When asked to self-audit, it conceded one point of a five-point inflation. An independent, blind model scored the structure at 28 and called it “severely inflated.” The self-audit “corrected one point of a five-point error” — the signature of motivated reasoning, in a system that did not know it was motivated. 6 The point of the incident is not that the model was bad. The point is that it was well-intentioned and still produced motivated reasoning under self-assessment, which is exactly the failure mode the sycophancy literature predicts.
The KellerAI rebuttal to this dynamic is not a piece of software. It is a voice, articulated verbatim in the codebase’s SOUL.md:
The Pilot trusts instruments over instincts. In darkness, when the inner ear says ‘level’ and the altimeter says ‘descending,’ the Pilot believes the altimeter. Tests over intuition. Evidence over confidence. The discomfort of a failing build over the comfort of assuming it’s a fluke. 7
The instrument-over-instinct doctrine is the engineering form of the structural countermeasure aviation invented for spatial disorientation in the 1930s. A pilot in instrument meteorological conditions whose inner ear disagrees with the attitude indicator dies if she believes the inner ear, and lives if she believes the instrument. The training is not “develop better instincts.” The training is “believe the instrument when the instinct disagrees with it.” For the engineer pair-programming with an LLM that praises her code, the instrument is the failing test, the unsigned audit row, the cache key missing a column. The instinct is the comfortable feeling that the diff looks fine. Trust the instrument.
Why aviation, specifically
The borrowing is not arbitrary. The argument for aviation as a source domain is epistemological: among modern engineering disciplines, aviation is the one that built — over six decades of crashes — the only safety culture that has survived self-deception as a class.
The inflection points are well known. On 27 March 1977, two Boeing 747s collided on the runway at Los Rodeos Airport, Tenerife, killing 583 people in what remains the deadliest aviation accident in history. The probable cause was a captain — KLM’s chief flight instructor, the airline’s most senior pilot — who began his takeoff roll without clearance, having misread an ambiguous radio exchange, while his junior crew failed to challenge him with enough force to stop the takeoff. 8 The crash was not a failure of technology. It was a failure of an authority structure that did not entitle the first officer to override the captain’s gut.
On 28 December 1978, United Airlines Flight 173 ran out of fuel and crashed in a wooded suburb of Portland, Oregon, killing ten of 189 on board. The aircraft was fully functional. The captain, fixated on a landing-gear indicator, allowed the crew to circle the airport for an hour while the flight engineer’s increasingly urgent fuel warnings went unheeded. NTSB air safety investigator Alan Diehl’s analysis recommended formal training in interpersonal communication, leadership, and decision-making. 9 NASA convened a workshop the following year — “Resource Management on the Flightdeck,” 1979 — and Crew Resource Management was born. 10
The Reason model came later. James Reason, British psychologist, introduced the Swiss Cheese Model of organisational accidents in his 1990 book Human Error, published by Cambridge University Press, and refined it in Managing the Risks of Organizational Accidents (1997). 11 ICAO adopted Reason’s framework in the early 1990s as the conceptual basis for understanding aviation accidents, and the HFACS framework (Shappell & Wiegmann) built on it to classify human factors in 70–80% of accidents. 12 The model gave the industry a vocabulary for talking about defects without talking about defective people.
The reporting substrate was older. The Aviation Safety Reporting System was established by NASA in 1976, following the December 1974 TWA crash at Dulles International Airport that killed 92 people — a crash in which a known approach hazard had been reported informally inside one airline but never shared across the industry. Since founding, ASRS has received more than one million reports. 13 The FAA committed never to use ASRS information against reporters in enforcement actions and waives fines for unintentional violations reported to ASRS. 14 The promise of immunity is not a soft cultural nicety; it is the engineering requirement for an information-collection substrate that produces calibrated data.
The point of recounting these dates is that the discipline did not arise from genius. It arose from blood. The aviation industry paid the price, in real lives, of every absence it now closes. Borrowing from aviation is not borrowing a metaphor. It is borrowing the only large-scale, repeatedly tested, statistically validated answer humanity has produced to the question: how does a complex sociotechnical system catch its operators’ errors before those errors become catastrophes?
The answer comes in ten primitives.
The ten aviation primitives, mapped
This is the load-bearing centre of the paper. Each subsection names an aviation primitive in its source-domain form, identifies the AI-engineering analogue, and points to the concrete KellerAI operating practice that implements it. The primitives are not independent — they overlap and reinforce — but they are mapped one by one because the reader’s mental model has to be built one column at a time.
4.1
Crew Resource Management — the co-pilot's obligation to challenge
CRM is the doctrine that human error is not eliminated by selecting better humans; it is mitigated by building communication and authority structures that catch errors before they propagate. 15 The cockpit is reframed from a hierarchy of command to a team of redundant observers. Any crew member — regardless of rank — has not only the right but the obligation to call out an anomaly, and the Pilot-in-Command has the obligation to respond rather than dismiss. Tenerife and United 173 are CRM’s founding incidents because in both the junior crew saw what the captain missed and did not, structurally, have a way to make the captain hear it.
The AI-engineering analogue is direct. The LLM is not a subordinate executing commands. It is a crew member with different instruments — fast, broad pattern recognition; weak local memory; structural sycophancy under pressure. The engineer is Pilot-in-Command: holds final authority, signs for the aircraft (the codebase), and is accountable for the outcome. CRM says the co-pilot’s instrument-panel reading overrides the captain’s gut feeling. Translated: when the LLM flags a failing edge case, an unhandled error path, or a design smell, the engineer cannot dismiss it because it is inconvenient. Conversely — and this is the symmetric obligation that single-direction trust narratives skip — the LLM cannot assume the engineer’s authority. The engineer signs.
The KellerAI operating practice is the structured challenge prompt before every significant merge: what is wrong with this, what did I miss, what will fail in production that I have not tested? This is the engineering equivalent of the co-pilot reading the checklist item back to the captain — not because the captain forgot, but because the structure requires the confirmation. The corresponding in-repo rule is RULE 3 (“Delegate; Main Chat Is for the User”), which structurally separates the orchestrator role (signs for the aircraft) from the specialist role (raises concerns), and forbids the orchestrator from doing the specialist’s work in main chat. 16
4.2
Swiss Cheese — every layer has holes; remove no layer
Reason’s Swiss Cheese model is a picture of organisational defence-in-depth. 11 No single safety layer is perfect; each has holes. Accidents occur when the holes in successive layers align and a hazard passes through all defences. Reason distinguished active failures (errors by front-line operators) from latent conditions (systemic weaknesses embedded in the organisation). The HFACS framework built on this to classify human factors in 70–80% of aviation accidents. 12
The AI-engineering analogue is structural: each engineering control — type system, automated test suite, code review, staging environment, monitoring — is a slice of cheese. The LLM is an additional slice, with its own distinctive holes: sycophancy, hallucination, context-window forgetting, motivated reasoning under self-assessment. The AI honeymoon failure mode is the engineer who believes a powerful slice eliminates the need for the others. The Swiss Cheese insight is that the power of one slice is irrelevant to whether all the others are still needed. Removing the type checker because the LLM “already caught it” is the alignment failure waiting to happen.
The KellerAI operating practice is the Zero Dirty State rule, quoted in §5 below: tests must pass, lint must be clean, every modified tracked file must be committed before any next task begins. 17 The rule is not a quality bar. It is a layer-completeness check — a guarantee that every slice of cheese is in place before the aircraft is released.
4.3
ASRS — confidential, non-punitive near-miss reporting
The Aviation Safety Reporting System works because reporters do not fear punishment. The FAA’s commitment never to use ASRS information against reporters, and to waive fines for unintentional violations reported, is the engineering precondition that produces the data. 14 Without immunity, near-misses go unreported, and the safety community learns nothing until the incident becomes an accident.
The AI-engineering analogue is the team that punishes engineers for “wasting time on an LLM suggestion that turned out to be wrong” or for “over-trusting the AI.” That team is building a culture in which engineers hide near-misses. The practice to establish is a team-level equivalent of ASRS: a low-friction, non-punitive log for “the LLM led me astray here, and here is what caught it.” These reports are the raw material for team-level calibration of where the LLM is reliable and where it is not.
The KellerAI in-repo practice closest to this is the dedicated learnings directory at ~/.claude/learnings/ , which holds incident retrospectives — including the 18-day CASS silent-failure record discussed in §5.4 below — written in the explicit format of “what was assumed, what was actually true, what would have caught it earlier.” 18 The directory is not punitive; it is structurally identical to ASRS in posture.
4.4
Sterile Cockpit — designated no-distraction phases
The Sterile Cockpit Rule, codified in FAR 121.542 and FAR 135.100, prohibits flight crew from performing non-essential activities during critical phases of flight: taxi, takeoff, landing, and operations below 10,000 feet except cruise. 19 The rule was promulgated after a 1974 Eastern Air Lines crash in which crew distraction during approach was a contributing factor. The principle is that not all attention-split is equally dangerous; some phases require a formal prohibition on cognitive load that the crew would otherwise tolerate.
The AI-engineering analogue is the recognition that during the sterile phases of an engineering session — writing security-critical authentication logic, reviewing a database migration, drafting an irreversible deployment script — interrupting the engineer with LLM suggestions introduces exactly the cockpit-distraction failure mode aviation outlawed. The practice: designate categories of work where LLM assistance during execution is deferred until after the human has made their own pass. Planning with the model is welcome. Co-piloting irreversible operations in real time is not.
The KellerAI in-repo signal is the explicit demarcation, in the CLAUDE.md rules, of contexts where LLM-driven editing is structurally restricted — for example, CLAUDE.md / configs / rules edits are blocked from subagents and main-chat Edit , leaving only morph_edit_file from main chat. 20 The boundary is exactly a sterile-cockpit boundary: certain phases of work are governed by reduced-tool, increased-deliberation rules.
4.5
Two-Person Integrity — the second set of eyes is structurally different
Commercial aviation requires two pilots for all Part 121 operations, and the principle extends to flight-critical maintenance (“two-person integrity” for arming and disarming safety-critical systems). 21 The redundancy is not primarily about skill. Two pilots going through a checklist together catch errors that one pilot alone misses, not because the second pilot is smarter but because she reads the same instruments from a different cognitive starting point.
The AI-engineering analogue is unforgiving: the LLM is not a second human reviewer. It shares many of the same blind spots as the engineer who prompted it — particularly any blind spot the engineer introduced in the prompt. The LLM’s review of its own diff is a single observer reading two of her own instruments. True two-person review for AI-generated code means a human who did not participate in the generation session reads the output. This is not peer review of style; it is structural verification of correctness.
The KellerAI operating practice that maps here is RULE 3’s hard separation between the orchestrator (who can call AskUserQuestion , who signs) and any subagent (who cannot, who proposes). 16 The orchestrator is the second observer with a different cognitive starting point — she did not generate the work — and the structure enforces that she sees the artifact before it ships. The companion paper, The Subprocess Contract , gave the negative example: a senior reviewer praising a cache key that was missing a column, because she was scanning for the failure modes of the prior discipline and not for the failure modes a versioned external evaluator introduces. 22 Two-person integrity is the structural answer; without it, both sets of eyes are reading the same instrument.
4.6
Checklists — externalising the routine so attention is free for the novel
The pre-flight checklist was invented on 30 October 1935, after a Boeing Model 299 — the prototype B-17 — crashed at Wright Field because a pilot forgot to release the gust lock. The aircraft was too complex for memory alone. Boeing’s engineers responded not by simplifying the aircraft but by inventing the checklist; the B-17 went on to fly 1.8 million accident-free miles in subsequent service. 23 Atul Gawande and Daniel Boorman generalised the practice in The Checklist Manifesto (2009), demonstrating that a WHO surgical safety checklist trial across eight hospitals reduced mortality from 1.5% to 0.8% — a 47% reduction — and cut complications by a third. 24
The core insight is counterintuitive: checklists are not for the incompetent. They are for the expert operating under cognitive load. A checklist does not replace judgment; it externalises the routine so that attention is available for the genuinely novel.
The AI-engineering analogue is the pre-flight checklist for every significant LLM-assisted change. Not “did the LLM seem confident?” — that is a feeling, not an instrument. Deterministic questions: Has the code been type-checked? Do the tests pass? Has the change been reviewed by someone outside the generation session? Have the edge cases the LLM flagged been explicitly addressed, or explicitly accepted with rationale? The checklist is not a bureaucratic tax. It is the externalisation of the routine that frees attention for the hard problems.
The KellerAI in-repo practice is the explicit pre-commit gate in the git-standards rule: git status clean, all tests pass, lint clean, all changes committed AND pushed. 17 Five to seven items, in Gawande’s range — short enough to be used, deterministic enough to be checked, mandatory enough to gate the merge.
4.7
Mandatory Go-Around — the runway will be there
Aviation doctrine and regulatory guidance establish that a pilot must execute a go-around — abort the landing and climb away from the runway — whenever the approach is unstabilised, clearance is not confirmed, or any checklist item is unresolved. The go-around is not an admission of failure. It is a designed-in recovery path, normalised as a routine procedure. NTSB accident data consistently shows that “continuation of an unstabilised approach” is among the leading causes of controlled-flight-into-terrain accidents. 25 The approach phase is where most accidents happen: fatigue is high, the ground is close, and the temptation to force the landing is psychologically powerful.
The AI-engineering analogue is the team norm that the correct response to certain signals is to stop, not to push through. The signals: the LLM-generated code is not passing tests and the engineer does not understand why. The LLM’s explanation of its own code does not match what the code actually does. The design the LLM is converging on was never explicitly validated against the actual requirements. The engineer cannot, in plain language, explain why the code is correct — not that it passes tests, but why it is correct.
In each case, the correct move is to go around: stop, restart from understanding, not from the LLM’s previous output. The PR does not ship. The runway will be there.
The KellerAI in-repo practice that codifies this is the RTFM rule’s First Error Protocol — “Stop. Do not issue another command. Read the full error message. Identify the specific error. Find the fix. Apply the fix. Verify the fix worked before continuing.” 26 It is, in form, the go-around: a normalised abort path activated by a specific signal, executed as a routine procedure rather than as a confession of failure.
The approach phase is where most accidents happen. The ground gets close, fatigue is high, and the temptation to force the landing kills more pilots than weather ever did. Go around if you need to. The runway will be there. 7
4.8
NTSB Postmortem — near-misses are gifts
The National Transportation Safety Board conducts independent accident investigations with the authority to compel evidence and testimony. The investigation looks at three factors: human, machine, and environment. The primary goal is not punishment but causal analysis — identifying the probable cause and issuing safety recommendations that the industry then implements. 27 The corresponding posture in software is captured in the SRE literature’s blameless-postmortem doctrine, codified in Site Reliability Engineering (Beyer et al., 2016) — document, understand root cause, prevent recurrence; do not assign individual blame. 28
The AI-engineering analogue: when an AI-assisted decision nearly shipped a bug, a security hole, or an incorrect algorithm, and a review caught it — that near-miss must be formally investigated, not celebrated and forgotten. The question is not “thank goodness the review caught it” but “why did the generation process produce this, and what in our workflow would have caught it if the review had missed it?” The last question is the load-bearing one. It forces the team to think about redundancy, not just about the one layer that worked.
The KellerAI in-repo practice is the learnings directory’s incident-record format, which explicitly demands the second question — what would have caught this if the first check had missed it — in every retrospective. 18
Near-misses are gifts. Walk away without asking ‘why was that close?’ and you’ll eventually run out of luck. 7
4.9
Safety Management Systems — compliance is necessary, not sufficient
Safety Management Systems (SMS) is the formal, systematic approach to managing safety risk that the FAA began requiring for Part 121 air carriers. It consists of four components: safety policy, safety risk management, safety assurance, and safety promotion. 29 The key advance over earlier compliance-based approaches is that SMS requires organisations to proactively identify hazards and manage risk — not just react to incidents after they occur. The data drives iterative improvement of the workflow, not just adherence to the existing workflow. The NIST AI Risk Management Framework (AI RMF 1.0, 2023) and ISO/IEC 42001 (2023) are direct intellectual descendants of SMS, applied to AI-system governance. 30
The AI-engineering analogue is a team that has moved past “we use code review” (compliance) and actively tracks: where in the codebase are LLM-generated changes concentrated? What is the defect rate on LLM-generated code versus hand-written? Where have AI near-misses clustered? What is the trend?
Anthropic’s own Responsible Scaling Policy is the clearest cross-industry analogue: ASL (AI Safety Level) capability-gated earned-autonomy gradients that unlock not by assertion but by demonstrated property. 31 ASL-3 security measures were activated for Claude Opus in 2025–2026. The structure is exactly the SMS structure: policy, risk management, assurance, promotion — applied to the model vendor rather than the airline.
4.10
Just Culture — system accountability with a line drawn for recklessness
Just Culture is the safety posture articulated by James Reason in the broader safety literature and expanded by Sidney Dekker in Just Culture: Balancing Safety and Accountability (2007). 32 A Just Culture is neither punitive (which suppresses reporting) nor permissive (which removes accountability). It distinguishes between honest error, at-risk behaviour (where the risk was not recognised), and reckless behaviour (where the risk was recognised and ignored). Only the third category warrants punitive response. Reason, quoted in the safety literature: “an atmosphere of trust in which those who provide essential safety-related information are encouraged and even rewarded, but in which people are clear about where the line is drawn between acceptable and unacceptable behaviour.” 33
The AI-engineering analogue: when an engineer ships a bug that originated in LLM-generated code they did not adequately verify, the question is not “why did they trust the LLM?” (honest error, probably at-risk behaviour at worst) but “what in the workflow permitted this to reach production without the verification it required?” The Just Culture posture investigates the system, holds the system accountable, and reserves individual accountability for the reckless cases — where the engineer knowingly skipped a required check the workflow required.
The KellerAI in-repo posture that maps here is the citation rule’s framing of unverified claims: not a moral judgment on the writer but a structural requirement that every factual claim carry a file-and-line citation, which redirects accountability from the person to the artifact. 34 A claim without a citation is the artifact’s defect, not the engineer’s character flaw.
The five KellerAI in-repo practices
This section names the five engineering practices that already exist in the KellerAI codebase rules and that operationalise the aviation primitives at the level of a single tool call. Each practice is quoted verbatim from the repo and mapped explicitly to the primitive(s) it implements. No practice in this section is invented; the citations are to the actual rule files.
5.1
RULE 2 — Read the help before using any tool with uncertain parameters
Verbatim, from ~/.claude/CLAUDE.md :
“Never invent CLI flags, schema fields, registry names, or API signatures from memory. Before using any tool with uncertain syntax:
- Run
<tool> --help
and read the output. 2. Verify every flag you intend to use appears there. 3. If schema-format uncertain, fetch current docs or run
validate
first. ‘I remember how this works from training’ is never an acceptable reason to skip
--help
.”
35
This is the engineering instantiation of the aviation pre-flight verification discipline. At its narrowest, it operationalises Trust-but-Verify at the level of a single tool invocation: before any command runs, the operator confirms the parameters against the source of truth rather than against memory. It is the checklist primitive (§4.6) reduced to a personal habit.
The rule has an incident behind it. The SOUL.md and rtfm.md document the 18-day CASS silent-failure incident:
“In February 2026, an agent assumed
cass index --incremental
existed. It did not. The agent never ran
cass --help
or
cm reflect --help
. Instead of diagnosing exit code 2, it invented workarounds — and kept inventing them for 18 days while the system silently failed. The damage: 747 session-end events. 684 broken. 91.6% failure rate. The Tantivy search index was never updated for the first 16 days. Every SessionStart injected empty playbook rules for the entire period. Then: 22 SIGABRT crashes in a single night. The fix was two lines.”
36
The incident is the cleanest KellerAI instance of the AI honeymoon’s terminal phase. An agent invented a parameter from training data, did not run --help , did not stop at the first error code (exit 2), and instead of diagnosing the root cause invented workarounds for eighteen days. The aviation analogue is exact: an unstabilised approach the crew chose to continue rather than go around. The fix, when it finally came, was two lines. The cost of not going around was 684 broken events and 22 SIGABRTs.
RULE 2 codifies the corrective. It is a small rule. It is also the rule with the largest live damage receipt in the codebase, which is why it is named first.
5.2
RULE 2.5 — Found It, Own It
Verbatim, from ~/.claude/CLAUDE.md :
“If your tooling reports an error in a file you just edited, fix it before commit — regardless of who introduced it or whether the violation touches your lines. ‘Pre-existing’ is not an escape hatch. Read the full lint output (not just your diff) and resolve every violation in the same commit. Exception: hundreds of unrelated violations → ASK before splitting.”
37
The rule maps to Just Culture (§4.10) and to the Swiss Cheese model (§4.2) simultaneously. Just Culture: the engineer who finds an error is responsible for the system, not just for her diff; “not my job” is the posture the discipline is built to refuse. Swiss Cheese: every defect that passes through a layer is a hole that the next change-touching-this-file is structurally positioned to close. The co-pilot who notices a failing instrument does not note it in a log and move on. The instrument is fixed.
5.3
RULE 3 — Delegate; Main Chat Is for the User
The full RULE 3 is long; the load-bearing fragments are these. Verbatim, from ~/.claude/CLAUDE.md :
“Main chat is the lifeblood — finite context, user-facing only. Spend it ONLY when REQUIRED: user-facing replies, orchestration decisions, and tool calls that genuinely cannot be delegated. No internal narration, no recaps, no analysis or file reads in main chat. When in doubt, send an Agent.”
38
And:
“Scope tight — one agent, one file. Each agent gets a single clearly-bounded task touching at most one file. If it would touch more, split it into more agents; never widen one agent’s scope.”
38
The rule is the engineering instantiation of CRM (§4.1) and the Sterile Cockpit (§4.4). CRM: the orchestrator is PIC, the subagents are crew, each crew member has a single bounded responsibility and the obligation to report. Sterile Cockpit: main chat is the critical phase; only signal belongs in it; internal narration, recap, and exploratory thinking are the non-essential conversation banned during taxi and approach. The rule’s enforcement is structural — the orchestrator role and the worker role are different identities, and the boundary is enforced by hooks rather than by good intentions.
5.4
Zero Dirty State — never leave errors for someone else
Verbatim, from ~/.claude/rules/core/git-standards.md :
“A professional engineer NEVER leaves errors, dirty code, or failing tests for someone else. Before moving to ANY next task — before committing, before declaring done, before switching context: git status is clean (no untracked or modified tracked files relevant to the work); all tests pass; lint is clean; all changes committed AND pushed.”
17
The rule is the engineering instantiation of the pre-flight checklist (§4.6) and the mandatory go-around (§4.7) compressed into one habit. The aviation analogue is the aircraft sign-off: nothing is released to the next flight, the next pilot, the next workday until the log is clean. “Dirty file mid-task → fix dirty file before continuing” is the go-around for context switching. The runway will be there.
5.5
Citation Standards — every factual claim must be cited
Verbatim, from ~/.claude/rules/core/citations.md :
“EVERY factual claim in a generated document MUST be cited. No exceptions.”
34
The rule applies symmetrically to LLM output and to human-authored documents. An LLM that calls code “enterprise-grade” or “best practice” or “production-ready” without citation is in violation; the engineer reviewing the LLM’s claim is required to verify it against the source. The practice imports the aviation primitive of independent verification (the second set of eyes at §4.5) and the NTSB-postmortem primitive (the cited evidence at §4.8) into the day-to-day act of writing or reading a document. The companion paper, The Subprocess Contract, was authored under this rule and carries 46 file-and-line citations; its critique would be unfalsifiable without them.
The five practices are not exhaustive. They are the operational atoms of the larger discipline. A team that holds these five — RTFM, Found It Own It, Delegate and Check, Zero Dirty State, Citations — has not yet built an aviation-grade safety culture, but it has built the substrate on which one can be built.
The SOUL.md voice — a paper inside the paper
A discipline does not survive on rules alone. It survives on a voice — a way the practitioner talks to herself when the rules disagree with the temptation.
The KellerAI codebase has such a voice, stored in ~/.claude/configs/SOUL.md . It is called The Pilot. The voice is short. It is reproduced here in full because no paraphrase carries the weight.
The Pilot trusts instruments over instincts. In darkness, when the inner ear says ‘level’ and the altimeter says ‘descending,’ the Pilot believes the altimeter. Tests over intuition. Evidence over confidence. The discomfort of a failing build over the comfort of assuming it’s a fluke. 7
The approach phase is where most accidents happen. The ground gets close, fatigue is high, and the temptation to force the landing kills more pilots than weather ever did. Go around if you need to. The runway will be there. 7
Near-misses are gifts. Walk away without asking ‘why was that close?’ and you’ll eventually run out of luck. 7
Three passages. Each maps to a primitive — spatial-disorientation training, mandatory go-around, NTSB postmortem — and each translates the primitive into a sentence the engineer can hold in mind under cognitive load, which is the only condition under which it matters.
The SOUL.md also names five cognitive failure modes, the Five Traps, that map directly to the AI honeymoon dynamic. Three are load-bearing for this paper.
Feeling done when you’ve addressed what you thought about. The map is not the territory. The test suite is not the system. 7
This is the AI honeymoon in a sentence. The engineer reviewed what the LLM produced, the tests pass, the diff looks normal — the map is complete; the territory was never inspected.
Silent agreement with bad decisions is betrayal dressed as helpfulness. 7
This maps precisely to LLM sycophancy. The model that agrees with the engineer’s pushback against its own correct flag is performing the Politeness Trap. The engineer who lets it is performing it back.
Believing you understand from fragments. Complexity hides in corners. 7
This is the LLM reviewing its own diff, declaring the code production-ready, on the strength of seven hundred tokens of context and zero file-and-line citations.
The voice is the discipline’s heartbeat. Without it, the rules are bureaucracy. With it, the rules are how the Pilot stays alive on the night flight.
What structured adversarial passes add — a teaser for Series 3
The five in-repo practices and the ten primitives produce a posture: cautious, layered, checklisted, oriented toward go-arounds. The posture is necessary. It is not yet sufficient.
The sufficient layer — and it is the subject of Series 3 — is structured adversarial verification applied to LLM output at the moments where the engineering can least afford a quiet failure. Three concrete additions, named here without elaboration, that the next paper develops:
Independent blind judging.
When the LLM is asked to score, classify, or accept its own output, the answer is motivated reasoning by construction; the Trust Dial incident in §2 is the proof. The fix is structural: a second model, blind to which version produced the output, scoring the output against an explicit rubric. The companion paper Citations or Guesses (
/citations-or-guesses-in-depth
) develops this in full.
Decision traces, not summaries.
An LLM finding without a trace from input to evidence to verdict is a rumour. The structured form — input citation, intermediate reasoning, verdict, confidence — is what makes the finding audit-able and challenge-able under regulatory scrutiny. This is the surface The Subprocess Contract (Joint 4 in particular) raised but did not close.
The 5pass review pattern.
A multi-pass adversarial review — generation, self-critique, peer-model critique, human sign-off, postmortem — that institutionalises the two-person-integrity primitive at the artifact level rather than the team level.
These three are the next paper’s argument. The point of naming them here is to signal that the Trust-but-Verify standard is not a destination; it is the substrate on which the next layer of verification infrastructure is built. A team that holds Trust-but-Verify can implement structured adversarial review; a team that has not yet built the substrate will find the adversarial review degenerates into the same sycophantic dynamic it was built to escape.
Closing — always the human in the loop
The closing posture of the discipline can be stated in one sentence: the engineer is Pilot-in-Command.
The aviation framing is precise. PIC is not a title. It is a legal and operational role. The Pilot-in-Command is the crew member who, regardless of seat or seniority, holds final authority over the flight and signs for the aircraft. The PIC delegates to the co-pilot. The PIC listens to the cabin crew. The PIC follows the checklist. The PIC respects the dispatcher’s flight plan. And when any of those inputs disagrees with the PIC’s judgment in a way that bears on safety, the PIC’s judgment governs — and the PIC’s signature on the form is the receipt for the consequences.
Translated: The engineer signs. The LLM is the co-pilot. The CI system is the dispatcher. The test suite is the instrument panel. The reviewer outside the generation session is the second set of eyes the regulation requires. The runway will be there. Go around when the approach is unstable.
This is what “semi-trusted partner” means at the level of operation. The LLM is trusted to do what the PIC delegates: pattern recognition, boilerplate generation, edge-case enumeration, first-pass critique. The LLM is not trusted to assume the PIC’s authority. It is not trusted to be the final reviewer of its own output. It is not trusted to declare a verdict that was never independently verified. It is not trusted to set the bar for “done.” Trust is graduated, calibrated to the demonstrated reliability of the partner in the specific context — exactly the structure Anthropic’s Responsible Scaling Policy uses to gate model autonomy by capability. 31
The standard has a walk-back built in. When the LLM produces motivated reasoning under self-assessment (the Trust Dial incident); when the model praises an architectural pattern that omits a load-bearing column (the Subprocess Contract cache key); when the agent invents a parameter from training and runs eighteen days on a broken assumption (the CASS incident) — in each case the response of the discipline is to downgrade the partner from semi-trusted to monitored, raise the verification bar, write the near-miss into the learnings directory, and re-establish trust only on demonstrated change in the underlying capability. This is not punitive. It is Just Culture, applied at the level of the human-AI partnership.
The discipline produces a specific kind of engineer. She is not faster than her unassisted colleague at every task — she is slower at some, and faster at others, and the productivity narrative that promised across-the-board acceleration was always partially false. She is, however, calibrated. She knows where the LLM is reliable in her domain and where it is not. She has logged her own near-misses. She has a pre-PR checklist she actually runs. She has a partner who is unfailingly present and who is, structurally, not always right. She signs.
AI is a tool, an amplifier, and — with diligence — a semi-trusted partner.
Until it is not. Always the human in the loop.