Multi-turn reinforcement learning gains in LLM agents depend heavily on interaction budgets, study finds

A new paper from the Chinese Academy of Sciences Institute of Automation finds that the gains from multi-turn reinforcement learning (RL) for AI agents depend heavily on how many turns the agent is allowed to take. When interaction budgets tighten, the advantage of RL-trained agents over base models shrinks or disappears.

The research, published on November 4, introduces a metric called Pass@(K,T) to evaluate agent performance across both sampling breadth (K) and interaction depth (T). This dual-axis view reveals something that standard success-rate metrics hide: trajectory-level RL benefits require sufficient turn budgets to materialize.

Where RL actually helps - and where it doesn't

The team found that agentic RL does not uniformly improve single-step decision quality. Instead, its strength concentrates in two areas: error recovery and resilience over long sequences of interactions. When an agent has room to course-correct, RL training shows clear value. When the turn budget is cut short, that edge erodes.

One striking example from the paper's BFCL benchmark: at a 25% turn budget, a GRPO-Full model on Qwen3-4B scored 2.5 compared to 3.0 for the base model. The RL-trained agent performed worse than the untrained version when constrained. This budget sensitivity creates deployment risk - real-world systems often face unannounced limits like timeouts or cost ceilings that the agent cannot anticipate.

BudgetMix RL as a countermeasure

To address this fragility, the researchers propose BudgetMix RL, a training method that varies interaction budgets during the learning process. The approach exposes the agent to different constraints so it learns policies that work across a range of execution settings rather than optimizing for a single generous budget.

In the same 25% turn budget test where GRPO-Full fell behind the base model, BudgetMix RL reached a score of 8.5 - a substantial improvement over both. The method includes the total and remaining budget directly in the prompt, which raises an open question about how the model behaves when no budget cap is announced, a common scenario in production deployments.

The paper also includes a preliminary discussion of process supervision signals, weighing their benefits and limits as an alternative way to reduce budget dependence.

Why this matters for IT, product, and support teams

For teams deploying AI agents in customer support, IT operations, or internal tooling, the finding is practical: an RL-fine-tuned agent that performs well in testing with generous turn limits may underperform in production where latency budgets or cost controls restrict interaction depth. Testing should measure performance across multiple turn budgets, not just a single success rate. For professionals evaluating or building agent workflows, the research suggests that training methods like BudgetMix RL could produce agents that hold up better under real-world constraints - though the open question about unannounced caps means teams should still benchmark their specific deployment conditions.

The paper is available on arXiv under ID 2611.06483, with a project page at rethinking-agentic-rl.github.io.