Detecting Over-Simplification in Four Artifact Types

A growing class of production AI failures shares a single signature: an engineer simplifies an artifact — a prompt, a skill, an agent's system message, a plugin's tool descriptions — for clarity, cost, or maintainability. Aggregate evaluation metrics look stable. Users start hitting cliffs the eval doesn't see.

The published evidence is unambiguous. Subtle prompt formatting changes can swing accuracy by up to 76 points³ . Frontier models cap at ~68% accuracy under high instruction density ² . A STAR reasoning framework that scored 100% in isolation collapsed to 0–30% when embedded in a 60-line production prompt — the authors named this instruction dilution ¹ . SWE-Bench Verified pass rates for SWE-Agent + GPT-4 fell from 12.47% to 3.97% after stricter validation, with 32.67% of original successes attributable to solution leakage ⁴ .

The implication: most teams cannot tell when their artifacts have crossed a capability cliff, because the test set they built around the original artifact tests average competence rather than preserved capability frontier.

When a software function is simplified, regressions usually announce themselves: a test goes red, a build breaks, a type-checker complains. AI artifacts have no such property. They degrade smoothly across most inputs while collapsing on the narrow band of cases that justified the original complexity.

Three properties make this worse than ordinary regression:

This definition deliberately includes the eval harness in the loop. If an eval can't see the regression, the regression has, for operational purposes, happened.

"Advanced prompt, skill, agent, or plugin" covers four distinct surfaces with different failure modes. The detection framework has to adapt to each.

Across all four, the underlying mechanism is identical: specific tokens carry specific behaviors , and removing them removes the behaviors — but only on the inputs that triggered them. The detection challenge is identifying which inputs those are.

Six findings from the published literature anchor the framework. Each represents a constraint the system must respect.

Before designing tests, name the failures. Twelve modes recur across artifact types.

You cannot test for capability loss without naming the capabilities.

The literature and practice yield five candidates. Each is rated on signal quality and effort. The goal is selecting a combination, not picking one.

The recommended hybrid resolves into four components, each with concrete inputs, outputs, and thresholds.

A · Behavioral Contracts

Predicates can be regex, structural parsers, deterministic tool-trace inspections, or — sparingly — small LLM calls with high-agreement rubrics. Keep the set to 10–30 items.

B · Frontier Battery

Curated set at the edge of intended capability. Suggested composition:

Size: 50–200 items typical. Refresh production-derived slice quarterly per the data-flywheel pattern ¹⁴ .

C · Bias-Controlled Paired Judge

Naive LLM-as-judge fails. Position swap alone shifts pairwise outcomes by >10% ⁷ . The bias-controlled protocol:

# For each input in eval set:
# 1. Generate output from original artifact (A) and simplified (B)
# 2. Judge twice with positions swapped: (A,B) and (B,A)
# 3. If judgments agree   → record winner
# 4. If they disagree     → record TIE (position bias detected)
# 5. Normalize for length: if winner is >1.5× longer, downweight
# 6. Aggregate: simplified passes iff win_rate(B) ≥ 0.50

Use a judge from a different model family than the artifact's own model to reduce self-enhancement bias ⁸ . Calibrate quarterly against a small human-labeled set.

D · Triggered Bisection

Runs only when A, B, or C trips:

# Inputs: original, simplified, failing_eval_subset
# 1. Diff the two artifacts → list of removed/changed components
# 2. For each component, build a "partial" artifact:
#    simplified + just that component restored
# 3. Re-run failing items against each partial
# 4. The component whose restoration recovers failures = cause
# 5. If no single component recovers → interaction effect; pair-wise

O(n) for single causes, O(n²) for pair-interactions. Cap at pairs; report "interaction" rather than enumerate triples.

Six capability-carrying components below. Toggle them off to simulate simplification. Three scores reflect what each detection layer would see — and where they diverge.

Two patterns worth observing: aggregate pass rate degrades gradually while contracts can break cleanly on a single removal; and different components produce qualitatively different verdicts even at similar token cost.

A restaurant marketing prompt

Step 1 · Contracts

Build is blocked at Layer A. Both failed contracts are BLOCK severity. No need to proceed.

Step 2 · Bisection

Triggered. The bisector restores each removed component and re-tests:

Output: C3 was carried by the persona block; C4 by the refusal handling. Both must be restored or rewritten before the simplification ships.

Step 3 · A constructive next move

Knowing which components carry which behaviors changes the conversation from "we can't simplify" to "we can simplify these specific parts but not these." A second iteration might compress the three examples (which the bisector showed are weakly load-bearing) while preserving persona and refusal blocks. The framework gives an evidence-based path to a smaller artifact — not a flat veto.

The framework runs as a check on artifact changes. Suggested wiring:

# .github/workflows/artifact-eval.yml
name: artifact-evaluation
on:
pull_request:
  paths: ['prompts/**', 'skills/**', 'agents/**', 'plugins/**']

jobs:
contracts:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4
    - run: python -m eval.contracts --strict

frontier:
  needs: contracts
  steps:
    - run: python -m eval.frontier --threshold 0.85

paired:
  needs: contracts
  steps:
    - run: python -m eval.paired --sample 50 --swap --normalize-length

bisect:
  needs: [contracts, frontier, paired]
  if: failure()
  steps:
    - run: python -m eval.bisect --report-pr-comment

Suggested thresholds

Defaults — tune per artifact; safety-critical agents warrant stricter thresholds.

The data flywheel

Production failures should flow back into the eval suite. The pattern documented by LangChain, Braintrust, and others ¹⁴¹⁵ : monitoring identifies a production trace where the artifact misbehaved, a domain expert annotates the correct trajectory, the corrected example joins the frontier battery. The eval grows organically to match the failure modes the artifact actually encounters.

The framework is more expensive than "just look at pass rate." Four objections deserve direct response.

The framework is incomplete in known ways. Naming them honestly is part of operating it well.

• 01Stochasticity baseline. Any single eval run has noise. Threshold tuning must account for variance. Typical mitigation: run each item 3–5 times, use median or pass-at-k.
• 02Judge drift. The judge model may update between runs, changing scores without artifact change. Pin the judge version. Re-baseline on upgrades.
• 03Interaction effects. Components A and B may be load-bearing only together. Single-component bisection misses this; pair-wise is O(n²). Practical limit: stop at pairs and report "interaction" without enumeration.
• 04Generated frontier items. Using an LLM to generate frontier items risks them inheriting the LLM's blind spots. Human-curated items are higher signal but more expensive.
• 05Adversarial robustness ≠ over-simplification. This framework detects accidental capability loss; it does not specifically test for adversarial robustness, which deserves its own pipeline.
• 06Cross-artifact regressions. When a skill is simplified, the agent that calls it may degrade in ways the skill's eval doesn't see. End-to-end agent evals are needed in addition.