Two literatures, one problem, no citations between them

In 2024, Zhou et al. published "On Speeding Up Language Model Evaluation" (ICLR 2025): treat the cells of an LLM evaluation grid as bandit arms, sequentially pick promising method-example pairs, fill in the rest with low-rank matrix factorization, and identify the top-performing method using only 5-15% of the typical resources, an 85-95% saving in evaluations run. In 2026, a second, independent group published SySRs, a Successive-Rejects-style elimination bandit that identifies the best model on all fifteen benchmarks tested using at most 35% of the model/test-point pairs. That 35% is the paper's worst case, so it implies roughly a 65% floor on evaluation-work savings, not a cap. The two figures belong to two different papers and should never be blended into one range.

So: adaptive bandit pruning of an LLM evaluation grid under a cost cap is a published idea, twice over, by groups who did it before we thought about it. And it isn't LLM routing, either: RouteLLM and FrugalGPT allocate spend across models per production query at inference time; eval-grid pruning allocates an evaluation budget across a fixed grid of (model, task, effort) cells before anything ships. Adjacent problems, not the same problem under different names.

Budgeted multi-armed bandits were not an analogy imported into ad-tech. They are ad allocation's home turf. Slivkins (2013) poses pay-per-click ad allocation directly as a bandit problem where each ad has an uncertain click probability and a spending limit. Combes, Jiang and Srikant (SIGMETRICS 2015) prove regret lower bounds for the budgeted setting and show B-KL-UCB asymptotically optimal for the cost models they study. Xia et al. (IJCAI 2015) extend Thompson sampling to budgeted bandits with an O(ln B) regret bound. Hard per-arm dollar budgets and pacing discipline are exactly the cost regime an eval grid lives in, where a frontier model at maximum reasoning effort costs orders of magnitude more per cell than a small model at minimum; the guarantee boundary comes next.

Here is the striking part: a reference-list check of both eval papers found no citation to any of the three ad-tech bandit papers. Two research communities, a decade apart, converged on "budgeted bandit over a grid of costly options" without apparently reading each other. That is suggestive that the formalism fits the problem (though also consistent with budgeted-bandit theory simply being everyone's standard toolkit), and either way, the older literature answers budget questions the eval papers leave open.

The favorable, logarithmic-regret results hold in narrow regimes: essentially when one resource constrains you and the optimal policy is a single best arm. The moment the optimal policy must mix arms, playing different arms for different task types (the expected case for a heterogeneous eval grid, where different models win on different tasks), the lower bound degrades to Ω(√T). Concretely: the eval grid studied in this research (an internal multi-model agent-evaluation harness KellerAI builds and operates itself) works out to 34,320 cells, which gives each (subject, model, effort) arm at most 260 observations in a full exhaustive sweep. Enough for a bandit to prune obviously dominated arms early; nowhere near enough to certify a near-optimal mixture. The honest scope is adaptive cell pruning under a spend cap, never "optimal allocation," and a vendor pitch that promises the latter on eval-sized data is over-claiming.

Adoption also has a real build cost: an adaptive select-execute-update loop in place of a static sweep, a persistent spend ledger that most harnesses (including the one studied) don't have, and an audit story for why an eliminated arm was never measured. For small grids, cheap models, or strict audit regimes, the exhaustive sweep is the correct choice, not a fallback.

Our overlap review returned a clear verdict: the core mechanism is already published, with high confidence. What survives for a team building this is modest. Treating reasoning-effort level as its own explicit bandit axis, jointly with model choice, is bookkeeping, not an algorithm. Reaching for the classical budgeted-MAB family, with its proven regret bounds for heterogeneous per-arm dollar costs, instead of bespoke methods is a defensible algorithm choice. An engineering-adoption note, not a new problem formulation.

The in-depth companion carries the full treatment: every paper cited with verified figures, the bandits-with-knapsacks regime boundaries, the corrected grid arithmetic, the routing-versus-eval-allocation distinction, the adjacent IRT-filtering and example-compression approaches, and the Monday-morning adoption checklist.