A controlled study of nine models finds that task success decays geometrically with the number of dependent steps. On a real tool-use loop, every model fell from near-perfect to near-zero within sixteen steps. A shorter context made the decay steeper, not gentler.
Agent benchmarks report one number: the fraction of tasks completed. Production teams report that this number does not predict live behaviour. An agent passes every offline gate, then fails on a long multi-step job, and nobody can say in advance when the failure starts.
The author, Shubhra Mittal, an engineer at Microsoft AI, argues that the missing variable is the horizon. The horizon is the number of dependent steps the agent must complete correctly to finish a task. Benchmarks use short and medium horizons. Production workloads run about ten times longer. If success decays with horizon, benchmarks measure only the region where success is still high.
Assume each step succeeds independently with probability r, the per-step reliability. A task with horizon H then succeeds with probability close to r multiplied by itself H times. This is a geometric law. For a short task the result stays near one. As the horizon grows, the result falls toward zero.
A pipeline of stages with no retry follows the same law. If each stage has 99 percent uptime, a 100-stage pipeline completes only about 37 percent of the time.
The paper asks two questions: does agent success follow this shape, and is the driver the step count or the context length?
The study uses four synthetic task families. An exact simulator checks every answer, so no LLM judge adds noise.
Three families are streaming tasks: operations arrive one after another. Ledger tracks several account balances through a sequence of operations and asks for one final balance. Refchain tracks one named variable through a sequence of assignments with distractors. Cipher applies a sequence of ordered edits to a short string. The fourth family, ToolQA, is a real agent loop. The agent must reach a target node a fixed number of hops along a hidden chain. It issues its own tool calls to inspect nodes, and each hop reveals only the next node.
To separate step count from context length, the streaming tasks run in three regimes on the same task instances. In the natural regime, one instruction arrives per turn and the full history stays in context. In the compressed regime, the same operations arrive across the same number of turns, but early turns are replaced by a carried state and a short window of recent history. In the padded regime, all operations for a horizon arrive in one single turn.
Nine instruction-tuned models from six vendor families took part: six open models from 1.2 billion to 671 billion parameters, and three hosted proprietary systems. The dataset holds 10,664 analysed trajectories.
Success falls monotonically with horizon wherever the task is not trivial for the model. Of geometric, threshold, and linear decay shapes, the geometric shape fit best in 28 of 36 model-task cells. In the remaining cells, strong models held near-perfect accuracy until a critical horizon, then dropped suddenly.
Task structure mattered almost as much as model size. Refchain decayed slowest, Ledger decayed moderately, and Cipher decayed fastest for every model. In Cipher, one wrong edit early in the sequence invalidates every later step.
ToolQA produced the most severe result. At horizon 2, seven of nine models scored 1.00. By horizon 16, every model had fallen to a small fraction of its starting success. Qwen2.5-72B scored 1.00 at horizons 2 and 4, then 0.13 at horizon 16. DeepSeek-V3, the strongest model at horizon 16, scored 0.27. GPT-4o-mini fell from 1.00 at horizon 4 to 0.07 at horizon 16. A readiness review that tested only horizons 2 and 4 would have approved these models and seen no sign of the drop.
Per-step reliability rose with model scale, but with diminishing returns. The correlation with the log of parameter count was weak, at 0.36. The proprietary models clustered with the strongest open models. No model reached perfect per-step reliability on any non-trivial task, so every model falls below any fixed reliability target at some horizon.
Decay also accelerated inside a trajectory. Mean per-step accuracy fell from 0.58 in the first third of long trajectories to 0.44 in the last third. The first error usually occurred about one third of the way through, and after an error the agent rarely recovered. Format and tool-call drift, where a turn cannot be parsed as a valid action, affected 21 percent of trajectories and rose with horizon.
Three caveats. The tasks are synthetic; the paper does not test open-ended real workflows. The ToolQA runs stopped at horizon 16 because the API budget ran out, so the collapse at 16 is a lower bound. The model roster is dated: the proprietary systems are GPT-4o-mini, Gemini 2.5 Flash-Lite, and Claude 3 Haiku, all small hosted models, not current frontier models. The study has a single author, and three small models were excluded because they did not follow the response format.
The natural regime performed best at every horizon. The compressed regime decayed faster. On the study's log-odds scale, success fell by 0.69 for each doubling of horizon, against 0.44 for natural, with a p-value of 0.000003. The padded regime fell by 0.40 per doubling. That slope is statistically the same as natural, but it starts from a lower level.
If models lost information in long contexts, a shorter context would help. Instead, it hurt. The author's reading: the compressed regime removes the agent's own earlier reasoning text, so the agent loses the working notes it wrote for itself. The padded result rules out multi-turn conversation as the cause. What remains is the step count itself.
Teams truncate context to save cost and latency. This data says that shortcut is likely to reduce reliability on long tasks.
Measure per-step reliability on your own tasks, not the pass rate. Run the same task at several horizons and fit the curve. A pass at horizon 8 tells you almost nothing about horizon 32 unless you have the curve.
Budget horizon from measured reliability. The author projects a mean per-step reliability of 0.61 onto common benchmark lengths and reports 0.42 success at 8 steps, 0.30 at 30 steps, and 0.24 at 100 steps. These projected values do not match a plain geometric law: 0.61 multiplied by itself 8 times is about 0.02. The paper does not explain the difference, so treat the projected numbers with caution and fit your own curve. Because the per-step failure rate rises inside a trajectory, budget below the geometric estimate.
Insert step-level checks. Errors propagate silently, so end-to-end monitoring sees a failure only after it has happened. A cheap consistency check at each step catches the failure earlier. Parse every tool call strictly and treat a malformed action as an incident, because interface drift grows with horizon.
Do not truncate context as a reliability fix. Keep the agent's own reasoning trail available, or replace it with a compaction that preserves that trail, and then measure the effect at long horizons.
The code, task generators, prompts, seeds, raw trajectories, and analysis scripts are public, with an evidence trail that maps each reported number to its source.