Calibration as an objective function, not a slide

Marketing-mix modeling (MMM) is the observational layer of ad measurement: a regression-family model estimating, from historical data, how much each marketing channel contributed to sales. It is cheap to run, covers every channel continuously, and is systematically wrong in ways invisible from inside the model: confounded by seasonality, correlated spend, and everything the regression cannot see. The experimental layer is the ground truth: randomized lift tests, which randomly withhold ads from a control group and measure the outcome difference against a treated group. A randomized controlled trial for advertising: expensive, narrow, trustworthy.

The eval-platform mapping is almost embarrassingly direct. The LLM judge, heuristic scorer, or data-quality index (DQI) is the MMM layer: an automated number attached to every output, systematically biased in ways the judge itself cannot report. The human-graded or verified-outcome subset is the experiment layer: narrow, costly, credible. Most teams connect the two with a periodic spot-check and a judgment call. The interesting question is what it looks like to connect them the way Robyn does.

Robyn does not treat agreement with experiments as a validation slide produced after fitting. It makes calibration a literal term in the fit. The Analyst's Guide describes the mechanism: Meta's Nevergrad gradient-free optimization platform allows the inclusion of "the MAPE(cal,fb) as a third optimization score besides Normalized Root Mean Square Error (NRMSE) and decomp.RSSD ratio."1 Unpacking the three objectives: NRMSE is the fit-to-history error, meaning how well the model reproduces the sales it trained on. Decomp.RSSD is a plausibility term penalizing implausible channel-contribution splits. MAPE(cal) is the calibration error: the gap between the model's channel estimates and the experimental lift results for the same channels. Because the search is multi-objective and gradient-free, adding the third score is mechanically straightforward, and its effect is blunt: candidate models that fit history beautifully but disagree with the experiments lose the hyperparameter search.

Translated to judges, the architecture reads: disagreement with ground truth should be part of the loss the judge is tuned against, on equal footing with whatever accuracy metric it optimizes natively. A worked sketch, labeled as a sketch rather than a benchmarked recipe: tune the judge to minimize its native scoring loss plus a weighted calibration-error term (say, the absolute gap between judge pass-rate and human-verified pass-rate, computed per matched bucket of task type and time window) rather than the native loss alone. The bucketing is not decoration; it is where the next section's warnings live.

Robyn's documentation carries warnings born of calibrations gone wrong, and the first is scope alignment between the two measurement layers: "ensure that the incrementality studies align with what the MMM is measuring e.g. the same level of granularity, same metrics measured, and within the same period."1 Granularity, metric, window. Miss any of the three and the calibration term is fitting the model to a different instrument's reading of a different quantity.

Each dimension translates without force. Granularity: if the judge scores individual responses but human grading happens at the conversation level, the two layers are not measuring the same object, and no weighting fixes that. Metric: a judge measuring helpfulness calibrated against ground truth measuring task completion learns to predict task completion while wearing a helpfulness label. Window: judge scores from this month calibrated against human labels from last quarter import whatever drifted in between (model version, traffic mix, task difficulty) as "judge error." The residual between mismatched instruments is not bias to be corrected; it is a category error wearing a number.

The subtler warning comes from Gufeng Zhou, Robyn's author at Meta Marketing Science (writing with Skokan, Chen and Lares), and in our reading it is the single most reusable idea in the underlying research. Even with scope matched, the two layers measure different estimands, where an estimand is the quantity a measurement is actually estimating.

The consequence follows immediately: "a naive calibration comparing MMM and experiment outputs directly is likely to underestimate the calibrated media."2 Two measurements of "the same channel" are not measuring the same thing, and forcing agreement without correcting for the gap does not remove the model's bias; it installs a new, quieter one.

Judge calibration has this exact disease incubating. A verified-outcome label, such as "did the customer's issue actually resolve?", is a long-horizon, all-causes measurement: it bundles the response quality with everything else that happened downstream. A judge score on the response text is a short-horizon, response-scoped measurement. They are different estimands wearing the same name, and a team that forces the judge to match outcome labels without an explicit, reasoned correction for the horizon gap is teaching the judge to predict things the response did not cause. The Robyn lesson is that the correction must be a deliberate modeling decision, written down, not an assumption that the gap is zero because nobody looked.

Robyn also carries a data-discipline floor worth importing as posture rather than as numbers: "a minimum of two years of historical weekly data" (about 104 weekly observations, stating the derived figure explicitly) and a recommended ratio of "1 independent variable : 10 observations."1 The eval translation is not the figures but the fact of a floor. A calibration layer has sample-size requirements of its own, per bucket, and an expensive ground-truth subset too small to support the correction being fitted produces the worst of both worlds: confident miscalibration with a methodology slide.

Three seams, stated at full strength. First, the evidence-class gap. MMM calibrates against randomized experiments, the strongest evidence class there is. Most eval teams' ground truth is expert human grading, stronger than an automated judge and still not a randomized experiment. The analogy's strength degrades accordingly, and it should be said plainly that the experimental-design half of the original ad-tech analogy (geo-lift testing, ghost ads, IAB/MRC measurement standards as a template for judge validity) was never verified in the research behind this series. It remains an unverified hypothesis, marked incomplete in our source research when the verification budget ran out, and this paper cites none of it as evidence.

Second, the implementation gap. This is an architecture, not a product experience report. Our research found no current hook for the pattern in the eval framework it studied, an internal multi-model agent-evaluation harness KellerAI builds and operates itself. What is missing is experiment-design primitives: randomized subset selection, matched-bucket bookkeeping, and the calibration-term plumbing. The source spike rates that gap L on its small/medium/large effort scale, the single highest-effort item in its shortlist, while noting the statistical backend itself needs no changes. We are not aware of a published end-to-end deployment of Robyn-style calibration for LLM judges; if a reader knows of one, we would genuinely like to read it.

Third, the originality boundary. We are not claiming to have invented judge calibration. That literature is active, and Zheng et al.'s "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" is its standard entry point3; we did not survey the judge-calibration sub-literature exhaustively in this research pass, and we say so rather than imply coverage. The narrower observation this pair of papers contributes, that MMM's specific discipline of calibration-as-fit-objective, scope matching, and estimand correction transfers cleanly as an architecture, was the strongest "not already covered" candidate in our overlap review precisely because it is architectural rather than bandit- or routing-shaped. That is a statement about our review's coverage, not a certificate that no one has written it down.

If you're deciding whether to act on this:

• Ask the estimand question before comparing any two numbers. What horizon and what outcome does the judge score actually estimate, versus the ground-truth label? Write the answer down; the gap is the correction you owe, and "zero" is a claim, not a default.

• Match scope first. Same granularity, same metric, same time window between judge scores and ground-truth labels, or the calibration measures the mismatch, not the judge.

• Size the ground-truth subset like a statistician, not an auditor. Per-bucket sample-size requirements come before the loss function. If the subset cannot support the correction, a calibrated-looking judge is just a confident one.

• Then, and only then, move calibration into the objective. The Robyn pattern's whole point is that agreement with ground truth is a term the cheap layer is optimized against, not a slide reviewed after the fact.1

The distilled takeaway travels well. Treat your cheap scorer as an MMM, treat your ground truth as the experiment, make their agreement part of the objective, and before comparing the two numbers at all, ask what each one actually estimates. Two decades of marketing measurement suggest that gap is where calibration quietly fails.