The Last Step Gets Too Much Credit

The verifier passes after the final patch.

The patch gets the applause.

But the decisive move may have happened twelve calls earlier, when retrieval found the one migration note that invalidated the original plan.

Long-horizon agents create a credit problem: success arrives late, many steps contributed, several were wasted, and the most visible action is not necessarily the most valuable one.

If training gives the final step all the credit, the system learns to finish. It may not learn how to become finishable.

Credit has three dimensions

Temporal credit asks which earlier action caused a delayed outcome.

Structural credit asks which component helped: retriever, planner, model, tool, memory, verifier, router, or human.

Multi-agent credit asks which participant changed team success when agents explored, reviewed, delegated, or voted.

These dimensions interact. Agent B may find evidence early, Agent A may use it later, and a verifier may reject two branches before Agent C produces the accepted artifact. One scalar team reward does not identify the contribution path.

Represent a trajectory as a causal graph, not only a list:

evidence E3 -> plan P4 -> patch A7 -> test failure O8 -> revision A11 -> success
                  \-> rejected branch B6 -> diagnostic D9 -----/

The rejected branch may have information value if its failure localized the bug. A cancellation may have capacity value if it stopped redundant work. A safe refusal may preserve expected value by avoiding an irreversible loss.

Equal return is simple and noisy

The simplest Monte Carlo assignment gives every action in a successful episode the terminal return. If the task succeeds with reward 1, all decisions receive a positive learning signal. If it fails, all receive zero or negative signal.

That is unbiased under some formulations and enough for short tasks with many samples. For long agent runs it has high variance:

• good actions appear in failed trajectories because a later action was wrong;
• bad actions appear in successful trajectories because a later recovery rescued them;
• irrelevant stylistic tokens inherit the same reward as a decisive tool selection;
• costs incurred after success was already inevitable look helpful;
• environment failures punish the policy.

Discounting earlier reward can make the problem worse. It favors late actions precisely because the terminal signal is nearer, even when an early decision made success possible.

RUDDER proposes return decomposition and redistribution for delayed rewards. Its reported experiments cover artificial delayed-reward tasks and Atari settings, not language-agent software workflows. The transferable mechanism is to learn which state changes predict a difference in eventual return, then move reward information closer to those changes while preserving the return objective.

Do not reward the chain of thought as ground truth

Process supervision gives feedback on intermediate steps. In “Let's Verify Step by Step”, Lightman et al. compare outcome and process supervision on mathematical reasoning and report benefits for their process-supervised setup. That is primary evidence for a specific task and model regime.

Agent workflows add a crucial distinction:

model reasoning text = proposed explanation
environment transition = observed effect
verifier evidence = scored claim about progress

A plausible statement such as “I have isolated the root cause” should receive no process reward until a reproducible test, dependency relation, or counterfactual patch supports it.

Reward machine-checkable process when possible:

• a query retrieved the required authoritative document;
• the selected repository revision matches the task;
• a new test fails before the fix and passes after it;
• a tool receipt proves one idempotent effect;
• the failure set shrank without suppressing tests;
• a compensation restored the declared invariant;
• the verifier cites the exact evidence for a subgoal.

Process rewards can still be gamed. Counting files opened rewards opening files. Rewarding fewer failing tests rewards disabling tests. Rewarding agreement with a critic can train both systems into the same blind spot.

Subgoals must describe state, not activity

Bad subgoal:

inspect the authentication code

Better subgoal:

identify the function that validates session expiry,
with file range and a failing test that reaches it

The first rewards an action. The second defines evidence.

For multi-hop work, create a subgoal graph:

G1 establish task and base revision
G2 locate behavior and executable reproduction
G3 identify violated invariant
G4 construct bounded candidate
G5 pass focused behavior checks
G6 pass regression and policy checks
G7 commit verified artifact

Credit a transition only when its postcondition becomes newly true. Re-reading the same file does not earn G2 again. A later observation can invalidate a subgoal, in which case the trace records reversal and downstream credit should be reconsidered.

Counterfactual credit asks what changed

The strongest question is not “Was this step in a successful run?” It is “Would the run still have succeeded without this step, with everything else held as comparable as possible?”

Useful counterfactuals include:

• remove one retrieved document and replay selection;
• replace one model decision with the baseline policy;
• deny one tool and observe whether another path succeeds;
• remove one candidate from the verifier set;
• replace one agent's message with a null or default action;
• stop the trajectory before an extra retry and compare outcome;
• keep the action but change its timing or branch order.

Language-agent environments are stochastic and path-dependent, so one replay is weak evidence. Run paired seeds or multiple matched rollouts and report a distribution.

For a component i, a simple difference estimate is:

credit_i = E[return with i] - E[return with baseline_i]

The baseline must be plausible. Removing a required argument and crashing the workflow exaggerates credit. Replace the component with the best legal default or comparison policy.

Multi-agent teams need a counterfactual baseline

A shared team reward encourages cooperation but gives every agent the same signal. In a team of five, a passive agent receives success credit when four others solve the task. An agent that discovers a decisive failure may be punished if the team ignores it.

COMA introduces a counterfactual multi-agent actor-critic method that uses a centralized critic and an agent-specific counterfactual baseline. The evaluated domain and neural architecture differ from language-agent swarms. The key idea is directly relevant: compare the chosen action with alternatives for that agent while holding other agents' actions in context.

For an agent team:

credit(agent B at turn 4)
= team value with B's evidence message
- expected team value if B sent a legal baseline message

That requires a trace of who observed what and when. If Agent A independently saw the same evidence before B sent it, B's message has less marginal value. If B's message prevented a dangerous merge, it may have high value even though no new patch was produced.

Shapley-style attribution is principled and expensive

Cooperative game theory suggests measuring a participant's marginal contribution across different coalitions. For three agents A, B, and C, evaluate team value with subsets such as {A}, {B}, {A,B}, and {A,B,C}, then average the marginal value of adding each agent across orders.

This handles redundancy better than counting messages. If A and B independently find the same evidence, neither receives full standalone credit when both are present.

Exact evaluation requires many coalitions and becomes infeasible as team size grows. Agent workflows also violate clean assumptions: removing one agent changes messages, context, timing, and later decisions. Use sampled coalitions or structured ablations, and label the result as modelled contribution rather than causal fact.

Credit is also not payment. A component can have negative marginal outcome value while still being required for governance. Independent review may reject many candidates and reduce throughput, yet preserve expected value by preventing rare losses.

Worked trace: visible work loses to decisive evidence

Consider 100 matched illustrative tasks with three components:

• R retrieves versioned evidence;
• P proposes and revises a patch;
• V verifies behavior and policy.

Observed success under repeated ablations:

Configuration	Successes / 100
R + P + V	78
P + V, no R	51
R + V, baseline patcher	22
R + P, no V, externally adjudicated	58 accepted

Naive “last action” credit favors P because it writes the patch. Simple marginal differences show other contributions:

R marginal in full context = 78 - 51 = 27 points
P marginal versus baseline patcher = 78 - 22 = 56 points
V observed acceptance difference = 78 - 58 = 20 points

The V difference is not pure success contribution because “no V” changes the acceptance mechanism and external adjudication. It may include false accepts and false rejects. This is why every ablation needs an admissible baseline and equivalent terminal scoring.

Now add costs:

R costs $0.20/task and adds 27 successes per 100
incremental cost per added success = $20 / 27 = $0.74

extra P search costs $1.50/task and adds 18 successes over a cheaper patcher
incremental cost per added success = $150 / 18 = $8.33

These are constructed numbers. They demonstrate how credit and economics combine: allocate the next inference dollar to the component with positive marginal verified-outcome value, not the component that produces the final artifact.

Credit the stop decision

Continuing is an action. Cancelling is an action. Escalating is an action.

An agent that stops two losing branches after a verified candidate appears saves cost and capacity. A coordinator that detects repeated equivalent failure prevents a runaway loop. A reviewer that abstains under insufficient evidence avoids a false accept.

Add explicit counterfactual value:

stop credit = avoided expected cost and risk
              - probability that cancelled work would improve outcome × value

Estimate it from historical continuation experiments, not from the stopping agent's confidence alone.

Credit models drift with the workflow

A process reward model trained on one tool interface may mistake new behavior after tools change. A retriever improvement can make an old “search again” step redundant. A stronger verifier can shift value from human review to automatic rejection.

Version credit models and test them on bifurcation tasks where one controlled decision changes the outcome. Monitor:

• rank correlation with controlled ablations;
• false positive reward for non-progress activity;
• credit mass by position, component, and agent;
• stability under paraphrase and harmless reordering;
• agreement with delayed production outcomes;
• whether cost-saving stop decisions receive credit.

If 80 percent of positive credit always lands on the last 10 percent of tokens, investigate positional bias before celebrating “efficient reasoning.”

Some outcomes arrive after the training window

A patch can pass today and create maintenance cost next month. A support answer can satisfy the user and cause a refund later. A migration can complete and corrupt a rare record during the next batch.

If the episode closes at immediate verification, delayed value never enters the return. Extend outcome identity beyond the training run:

task accepted
candidate verified
artifact merged
deployment observed
7-day rework window
30-day rollback and incident window
90-day maintenance sample

Do not keep an RL worker open for 90 days. Join delayed events to the immutable task and artifact IDs, then update an evaluation dataset or delayed reward model. Protect against survivorship bias: deleted, reverted, abandoned, and never-deployed artifacts remain in the denominator.

Delayed feedback is confounded. A later incident may involve many changes. A maintainer may rewrite a patch for unrelated style. Use causal evidence where possible:

• bug-fix commits that cite the generated artifact;
• rollback receipts tied to the exact release;
• review comments linked to changed lines;
• paired canaries or feature flags;
• ownership and dependency graphs;
• human adjudication with uncertainty labels.

Credit can be negative. If an early shortcut yields an immediate pass and later rework, the full return should reflect both. Version the attribution window so “success” does not silently change between training runs.

Credit uncertainty should control autonomy

Teams often store a single credit score, such as 0.73, and route optimization as if it were observed truth. Credit is an estimate. It depends on the baseline, replay seeds, judge calibration, missing alternatives, and environment stability.

Report an interval or distribution:

retrieval marginal success: +27 points
95% bootstrap interval: [+18, +35]
matched tasks: 100
baseline: same policy without retrieved document
environment: repo-eval.v12

The interval above is illustrative. In real analysis, preserve paired task outcomes and use a method appropriate to the design.

Uncertainty changes decisions:

• high positive credit, low uncertainty: invest or route more work;
• near-zero credit, low uncertainty: simplify or remove the component;
• positive credit, high uncertainty: collect targeted bifurcation tasks;
• negative credit on a protected slice: block promotion;
• unknown credit for irreversible work: retain human or independent verification.

This prevents an attribution model from becoming a hidden authority system. Credit can recommend where to spend inference or which agent to include. The runtime still enforces permissions and consequence boundaries.

Credit without novelty rewards meetings

Agent swarms can inflate activity through repeated agreement. Ten agents restating one diagnosis generate more tokens and more apparent consensus without adding information.

For messages or evidence, separate:

validity: is the claim supported?
novelty: was it already known to the team?
decision impact: did it change an action, verifier, or stop?
outcome impact: did the changed decision improve expected terminal value?
cost: what inference, latency, and coordination did it consume?

A duplicate can still have value by increasing confidence under independent evidence. But agents using the same model, prompt, context, and retrieved source are not independent votes. Measure source and failure correlation before rewarding consensus.

The next article applies this discipline to swarms: adding agents increases search capacity, but also communication edges, correlated error, stale state, and verification load.

Builder playbook

1. Preserve causal trajectory graphs with states, observations, eligible actions, effects, and terminal evidence.
2. Separate temporal, structural, and multi-agent attribution.
3. Begin with honest terminal reward and explicit cost.
4. Define subgoals as newly verified state, never mere activity.
5. Keep process judgments tied to executable or cited evidence.
6. Build legal counterfactual baselines and run matched repeated replays.
7. Use agent-removal, message-removal, tool-removal, and candidate-removal ablations.
8. Measure marginal verified outcome and marginal cost together.
9. Credit cancellation, abstention, recovery, and loss prevention.
10. Version and recalibrate credit models when workflows, tools, or verifiers change.

The last step deserves credit for what it changed.

It does not deserve the whole story merely because it was standing beside success when the reward arrived.

References

• Arjona-Medina et al., “RUDDER: Return Decomposition for Delayed Rewards,” NeurIPS 2019. Return decomposition and reward redistribution for delayed-reward problems.
• Foerster et al., “Counterfactual Multi-Agent Policy Gradients,” AAAI 2018. Agent-specific counterfactual baselines under a centralized critic.
• Lightman et al., “Let's Verify Step by Step,” 2023 preprint. Primary comparison of process and outcome supervision in mathematical reasoning.
• Sutton and Barto, “Reinforcement Learning: An Introduction,” second edition. Primary textbook treatment of returns, value, temporal difference, and policy learning.
• Wang et al., “Agent Lightning,” 2025 preprint. A framework addressing training and credit assignment across multi-step agent executions.
• Li et al., “Multi-Agent Credit Assignment with Pretrained Language Models,” AISTATS 2025. Peer-reviewed work on sparse-reward cooperative multi-agent credit assignment.