Calibration as an objective function, not a slide

Marketing-mix modeling (MMM) is the observational layer of ad measurement: a regression-family model that estimates, from historical data, how much each marketing channel contributed to sales. Like an LLM judge, it is cheap to run, covers everything, and is systematically wrong in ways you can't see from the inside. The experimental layer is the ground truth: randomized lift tests, which randomly withhold ads from a control group and measure the outcome difference against a treated group (the ad-world equivalent of a randomized controlled trial). Expensive, narrow, trustworthy.

What Meta's Robyn does with these two layers is the part worth stealing. Robyn does not treat agreement with experiments as a validation slide to show after fitting. It makes calibration a literal term in the fit: its optimizer runs a multi-objective hyperparameter search that includes "the MAPE(cal,fb) as a third optimization score besides Normalized Root Mean Square Error (NRMSE) and decomp.RSSD ratio."1 Unpacking the jargon: NRMSE is the fit-to-history error; decomp.RSSD is a plausibility term that penalizes implausible channel-contribution splits; MAPE(cal) is the gap between the model's estimates and experimental lift results; and the search runs on Nevergrad, Meta's gradient-free optimizer, which is why bolting on a third objective is mechanically straightforward. Candidate models that fit history beautifully but disagree with the experiments lose the search.

Translate the architecture directly: your judge or DQI scorer is the MMM layer; a randomized, human-graded or verified-outcome subset is the experiment layer. The Robyn pattern says the expensive layer shouldn't merely audit the cheap one after the fact. Disagreement with ground truth should be part of the loss you tune the cheap layer against, on equal footing with whatever accuracy metric it optimizes natively. Concretely, a judge tuned this way minimizes its native scoring loss plus a weighted calibration-error term (say, the absolute gap between judge pass-rate and human-verified pass-rate, computed per matched bucket of task type and time window) rather than the native loss alone.

Robyn's documentation ships with warnings born of calibrations gone wrong, and both map onto judge calibration with almost no force applied.

The first is scope matching. Robyn: "ensure that the incrementality studies align with what the MMM is measuring e.g. the same level of granularity, same metrics measured, and within the same period."1 The eval translation: if your judge scores individual responses but your human grading happens at the conversation level, or your judge measures helpfulness while your ground truth measures task completion, or the two layers sample different weeks of traffic, then you are calibrating one instrument against a different instrument and calling the residual "judge error."

The second is subtler and, in our reading, the single most reusable sentence in the underlying research. Gufeng Zhou, Robyn's author at Meta Marketing Science, writing with Skokan, Chen and Lares, warns that the two layers measure different estimands: "experimental estimates are usually the short-term last dollar impact of ads on some match-able outcomes, while MMM measures the long(er) term average impact of ads on all outcomes." The post's conclusion follows: "a naive calibration comparing MMM and experiment outputs directly is likely to underestimate the calibrated media."2 An estimand is simply the quantity a measurement is actually estimating, and two measurements of "the same thing" routinely aren't.

Judge calibration has this exact disease waiting for it. A verified-outcome label ("did the customer's issue resolve?") is a long-horizon, all-causes measurement; a judge score on the response text is a short-horizon, response-scoped one. Force the judge to match outcome labels without correcting for the estimand gap and you aren't removing the judge's bias. You're teaching it a new one. The Robyn lesson is that the correction has to be an explicit, reasoned adjustment, not an assumption that the gap is zero.

There is also a data-discipline floor worth carrying over. Robyn's guidance is "a minimum of two years of historical weekly data" (about 104 weekly observations, to state the derived figure explicitly) and "1 independent variable : 10 observations."1 The numbers themselves don't transfer to evals; the posture does. A calibration layer has sample-size requirements of its own, and an expensive ground-truth subset too small to support the correction you're fitting produces confident miscalibration.

Honesty about the seams, because there are three.

First, MMM calibrates against randomized experiments, the strongest evidence class there is. Many eval teams treat expert human grading as a stronger reference signal than an automated judge, and it is, but it is still not a randomized experiment, so the analogy's strength degrades accordingly. The experimental-design half of the original ad-tech analogy (geo-lift testing, ghost ads, IAB/MRC measurement standards) was never verified in our underlying research; it remains an unverified hypothesis, and we cite none of it as evidence here.

Second, this is an architecture, not a product experience report. Our research found no current hook for the pattern in the eval framework it studied, an internal multi-model agent-evaluation harness KellerAI builds and operates itself. The missing piece is experiment-design primitives: a large gap, the highest-effort item in our research shortlist, though the statistical backend itself needs no changes. We are not aware of a published end-to-end deployment of Robyn-style calibration for LLM judges. If a reader knows of one, we would genuinely like to read it.

Third, on originality, the position of this whole series applies here too: we are not claiming to have invented judge calibration. That literature is active and has its own reference points: "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" is the standard entry3, and we did not survey the judge-calibration sub-literature exhaustively. The narrower observation — that MMM's specific discipline of calibration-as-fit-objective, scope matching, and estimand correction transfers cleanly — was the strongest "not already covered" candidate in our overlap review precisely because it is architectural rather than bandit- or routing-shaped. That is a statement about our review's coverage, not a certificate that no one has written it down.

If you're deciding whether to act on this:

• Ask the estimand question before comparing any two numbers. What horizon and what outcome does the judge score actually estimate, versus the ground-truth label? Write the answer down; the gap is the correction you owe.

• Match scope first. Same granularity (response vs. conversation), same metric, same time window between judge scores and ground-truth labels, or the calibration is measuring the mismatch, not the judge.

• Size the ground-truth subset like a statistician, not an auditor. If it can't support the correction you're fitting, a calibrated-looking judge is just a confident one.

The distilled takeaway travels well. Treat your cheap scorer as an MMM, treat your ground truth as the experiment, make their agreement part of the objective rather than a slide. Two decades of marketing measurement suggest the gap between what the cheap layer estimates and what the expensive layer estimates is where calibration quietly fails.