Skip to main content
kellerai.blog

The industry that invented positional credit rules retired them

Google deprecated rule-based attribution. Step-level agent credit assignment is about to re-run that experiment.

KellerAI White Paper · Engineering Discipline & Verification · Jul 2026

Context

Agent evals are converging on step-level credit assignment: which step of a twelve-step trace deserves blame for the failure? Positional heuristics (weight the first step, the last step, or spread credit evenly) are the cheap, tempting first answer, and teams are building them now.

The Finding

Ad-tech already ran this experiment. Google no longer supports first-click, linear, time-decay, or position-based attribution; the default is a counterfactual model that compares converting against non-converting paths. Google's own docs record the two costs it accepted: a 200-conversion/2,000-interaction volume recommendation and an algorithm described only as 'Google AI.' The transfer to evals is a negative constraint: most agent harnesses don't yet capture the step-level, outcome-labeled traces the surviving method needs.

Tags:
Credit AssignmentAgent EvalsMeasurement Discipline
Paper Details
CategoryEngineering Discipline & Verification
AudienceML/agent-platform engineers building step-level credit assignment or process-reward-model evals; eval-infra leads weighing positional heuristics.
MethodAnalytical · primary-source read of Google Ads attribution documentation mapped against agent-eval trace instrumentation
Length~1,450 · 6 min
Sections4
DateJul 2026
AuthorsKellerAI
Read the full paper
In plain language

The problem on your desk

The problem on your desk: an AI agent took twelve steps, got the wrong answer, and someone has to decide which step deserves the blame. The tempting fix is a rule. Blame the last step, or the first, or spread it evenly. Teams are writing exactly these rules right now, because they ship in an afternoon and produce a tidy number. The trouble is that when your rule and your colleague's rule disagree, there is no principled way to say who is right. The number is a convention wearing the costume of a measurement.

What the solution is

This brief doesn't sell you a fix. It hands you the receipt from the industry that already ran the experiment. Online advertising spent two decades assigning credit across customer journeys with rules exactly like these. Google has now retired nearly all of them in favor of a method that compares the paths of customers who convert against those who don't, and lets the difference assign the credit. Google's own documentation also records what that upgrade cost: a real appetite for data before the answers are trustworthy, and a credit split you have to take on trust.

Why it works

It works as a warning precisely because it isn't a vendor's promise. It's the published track record of the one industry that ran positional credit rules at production scale, with revenue on the line, and walked away from them. The mapping is structural, not poetic: a chain of steps ends in an outcome, outcomes are scarce and late, and per-step credit is what everyone wants. That's attribution. Agent evals are simply the newest team to inherit the problem.

The bottom line

Don't hard-wire step-blame rules into your evals as the permanent system; the industry that invented them deprecated them. Instrument your traces so each step and outcome is recorded first. That's the prerequisite for the principled version, and it pays for itself in debugging even if you never build the rest. If you need a rule meanwhile, label it temporary and give it a review date.

Section 01

An industry deprecates its own heuristics

Multi-touch attribution (MTA) is ad-tech's name for the credit-assignment problem: a customer saw eight ads before buying, so which ads get credit for the sale? For years the standard answers were rule-based models with names that will sound uncomfortably familiar to anyone building agent evals: first-click (all credit to the first touch), last-click, linear (spread it evenly), time-decay (recent touches matter more), and position-based (bonus credit to first and last).

Google no longer supports most of them. Its Ads documentation states plainly that "the first click, linear, time decay, and position-based attribution models are no longer supported by Google," and that conversion actions using the deprecated four "have been upgraded to use data-driven attribution," now the default model for most conversion actions. Last-click is the one rule-based holdout still available. Google gives no date for the deprecation, only the fact of it, so treat this as the industry's standing position rather than a datable event.

What replaced the rules is worth reading closely. Data-driven attribution (DDA) is a counterfactual method: "by comparing the paths of customers who convert to the paths of customers who don't, the model identifies patterns among those ad interactions that lead to conversions," then assigns more credit to the interactions its path comparison associates with conversion. Not "the last touch matters most because we said so." Instead: compare journeys that ended in a conversion against journeys that didn't, and let the difference assign the credit.

One caveat before leaning on this story. Google is not a neutral witness to its own deprecation. A single-vendor model is easier to sell than an auditable rule an advertiser could second-guess, and that commercial motive is a real alternative explanation, distinct from "the empirics settled it." That doesn't make DDA's counterfactual logic wrong. It does mean "the market leader retired the rules" is weaker evidence than a controlled comparison, and this brief leans on the former.

Section 02

The two costs Google's own documentation exposes

The deprecation is only half the lesson. The other half is what Google's own documentation admits the replacement costs.

First, data volume. Google recommends "at least 200 conversions and 2,000 ad interactions in supported networks within a 30-day period" for DDA to assign credit precisely, while noting the model "will still function with less data." That is a precision recommendation, not an eligibility gate, and the distinction matters: counterfactual credit assignment degrades gracefully but needs volume to be trustworthy. There is a floor below which you are back to guessing, just with more machinery.

Second, auditability. On the Ads help pages, the mechanism behind DDA is described only as credit "as determined by Google AI": no named algorithm, no inspectable math. The practitioner-facing product documentation asks advertisers to accept the credit split on trust. Google judged that trade worth making. Whether you can make it, in a regulated eval pipeline where someone may ask you to justify why step 7 was blamed, is a different question, and an honest reading has to leave it open.

Section 03

Agent evals are standing where ad-tech stood

Step-level credit assignment for agents is structurally the same problem MTA was solving. That includes the current wave of process reward models, which score each intermediate step of a reasoning or tool-use trace rather than only the final outcome. A trajectory of touches ends in an outcome; you want per-touch credit; outcomes are sparse and delayed; and rule-based positional heuristics are the cheap, tempting first answer.

The record above is the strongest argument we know of for not adopting those heuristics permanently. Not because a linear or last-step weighting won't ship; it will, and that is its appeal. But the one industry that ran positional credit rules at production scale retired nearly all of them in favor of a counterfactual alternative. When your first-step-weighted eval metric and your colleague's last-step-weighted one disagree, there is still no principled way to pick a winner.

Section 04

You probably can't build the good version yet either

Here is where this brief refuses to become a feature pitch, because the underlying research is blunt about it. Our research spike rendered its verdict on the MTA-to-eval transfer as "inverted": the transfer's entire value is a "do not build this" constraint, not a feature spec. The reason is data plumbing. The system studied in that research (an internal multi-model agent-evaluation harness KellerAI builds and operates itself) captures exactly one trace event per subject invocation: no internal tool-call sequence, no list-of-steps field anywhere in the result schema. DDA-style counterfactual credit needs step-level, outcome-labeled path data for successful and failed runs alike, and you cannot run a converting-versus-non-converting path comparison on traces that don't record paths.

The honest sequencing: don't ship positional heuristics as the permanent credit system; instrument step-level, outcome-labeled trace capture first; and budget for both published costs (a data-volume floor and an auditability story) before building the counterfactual version. A cheap heuristic that is explicitly labeled temporary, with a review date, is fine. An "interim" rule that quietly survives a decade is the failure mode ad-tech just finished living through.

The in-depth companion develops the full argument: the deprecation record and its incentive caveats, both published costs with Google's own numbers, the process-reward-model mapping with the field's reference citations, the trace-instrumentation gap, and the staged adoption path.

End of paper

↑ Back to top