Calibration as an objective function, not a slide

Marketing-mix modeling (MMM) is the observational layer of ad measurement: a regression-family model that estimates, from historical data, how much each marketing channel contributed to sales. Like an LLM judge, it is cheap to run, covers everything, and is systematically wrong in ways you can't see from the inside. The experimental layer is the ground truth: randomized lift tests, the ad-world equivalent of a randomized controlled trial. Expensive, narrow, trustworthy.

What Meta's Robyn does with these two layers is the part worth stealing. Robyn does not treat agreement with experiments as a validation slide to show after fitting. It makes calibration a literal term in the fit: its multi-objective hyperparameter search minimizes the calibration error against experimental lift results alongside its fit-to-history error and a plausibility term. Candidate models that fit history beautifully but disagree with the experiments lose the search.

Translate the architecture directly: your judge or DQI scorer is the MMM layer; a randomized, human-graded or verified-outcome subset is the experiment layer. Disagreement with ground truth should be part of the loss you tune the cheap layer against, on equal footing with whatever accuracy metric it optimizes natively.

Robyn's documentation ships with warnings born of calibrations gone wrong, and both map onto judge calibration with almost no force applied. The first is scope matching: align the two measurement layers on granularity, metric, and period. If your judge scores individual responses but your human grading happens at the conversation level, or your judge measures helpfulness while your ground truth measures task completion, you are calibrating one instrument against a different instrument and calling the residual "judge error."

The second is subtler and, in our reading, the single most reusable idea in the underlying research: estimand mismatch. Robyn's authors warn that experiments measure short-term impact on matchable outcomes while the model estimates longer-term impact on all outcomes, so naive calibration systematically underestimates. An estimand is simply the quantity a measurement is actually estimating, and two measurements of "the same thing" routinely aren't. Judge calibration has this exact disease waiting for it: a verified-outcome label is a long-horizon, all-causes measurement; a judge score on response text is a short-horizon, response-scoped one. Force the judge to match outcome labels without correcting for the gap and you aren't removing the judge's bias. You're teaching it a new one.

There is also a data-discipline floor worth carrying over: Robyn's published guidance amounts to roughly two years of weekly observations and a 1:10 variable-to-observation ratio (the in-depth companion carries the primary source). The numbers don't transfer to evals; the posture does. A ground-truth subset too small to support the correction you're fitting produces confident miscalibration.

Three seams, honestly. First, MMM calibrates against randomized experiments; expert human grading is stronger than an automated judge but still not a randomized experiment, so the analogy's strength degrades accordingly. Second, this is an architecture, not a product experience report: our research found no current hook for the pattern in the eval framework it studied (an internal multi-model agent-evaluation harness KellerAI builds and operates itself), and the missing experiment-design primitives are the highest-effort item in our research shortlist. Third, we are not claiming to have invented judge calibration; that literature is active. The narrower observation, that MMM's specific discipline of calibration-as-fit-objective, scope matching, and estimand correction transfers cleanly as an architecture, is what this pair of papers contributes.

The in-depth companion carries the full treatment: Robyn's three objectives with the primary-source quotes, the estimand-mismatch warning verbatim, the worked calibration-loss sketch, the judge-evaluation literature's reference points, and an explicit accounting of which strands of the ad-tech analogy remain unverified.