AI-TechScienceGovernmentBusinessSportsEntertainment
AI-Tech

Autonomous software agents stumble when tasks require longer chains of tool calls

Testing across more than one hundred live servers reveals that current artificial intelligence systems fail as sequences lengthen, even when given the correct digital tools.

15 statements added by Jerzy Kamiński · 29 Oct see what was added
Autonomous software agents stumble when tasks require longer chains of tool calls
Source: Christopher Bowns (CC BY-SA 2.0)
Published29 Oct 2026, 13:54 Last updated29 Oct 2026, 13:54 Sources
Source mode Marks each sentence drawn from a source or a contributor

When an artificial intelligence agent interacts with external software, grading its success is surprisingly difficult. In static programming benchmarks, an automated grader simply compares the model's final text output to a fixed answer key. In the physical or digital world, however, software environments change constantly, records update, and multiple different sequences of actions can accomplish the exact same goal. Grading an agent purely on the exact words it outputs or the precise sequence of functions it calls often misclassifies valid solutions as failures.

Software systems increasingly connect large language models to external data sources and application programming interfaces through standardized interfaces like the Model Context Protocol, an open standard that allows models to discover and invoke external tools. If these autonomous systems are deployed to handle administrative workflows, data lookups, or system maintenance, developers need dependable ways to measure whether an agent actually achieved the requested outcome in a live environment.

To solve a task in a changing digital environment, an agent must complete a clear chain of actions. First, it receives a user request and inspects available tools across connected servers. Next, it must select an appropriate tool, supply the correct arguments, and execute the call. It then reads the returned data, updates its internal plan, and triggers subsequent tool calls until the desired change takes place. Evaluating this process requires verifying whether the necessary operational effects occurred, regardless of the intermediate path chosen.

In a preprint posted to the arXiv server, computer scientists Jerzy Kamiński, Ilya Galyukshev, Artem Kuznetsov, Sergey Chuprin, Kirill Redko, Aidar Shumbalov, and Anna Kalyuzhnaya introduced DynamicMCPBench, an automated evaluation framework designed to score agents on live, stateful servers.1 Rather than checking a static final text response, the framework records live successful trajectories, distills them into path-agnostic effect checkpoints, and grades an agent based on whether it reproduces those required system effects.12

How do current language models perform on multi-step tool tasks?

Large language model agents struggle significantly when required to execute multiple tool steps in succession across live servers. In an evaluation across 24 models tested on 121 Model Context Protocol servers and 750 tasks, Kamiński and colleagues found that even the top-performing agents solved only about half of the total tasks.1 The benchmark divided tasks evenly across 15 distinct task categories, with 50 tasks in each category.12 Performance was measured using a strict pass^3 metric, where a task counts as solved only if a model succeeds across three independent attempts.12 Under this standard, 31 percent of the 750 tasks were solved by no model at all.12

Autonomous software agents stumble when tasks require longer chains of tool calls
Source: Original article

Jerzy, a researcher on the project, explained that the failure pattern across systems pointed to an inherent bottleneck in coordination rather than simple model mistakes. Jerzy said in response to questions from Primary that "the tasks that stump everyone are the ones that string many steps together across different services." Jerzy noted that the team had expected agents to pick the wrong functions, but the systems rarely made that error. "Instead they run out of steam partway through: they start pulling the right information but don't finish putting it together into the result the task actually asked for," Jerzy said, adding that "they aren't picking the wrong tool so much as losing the thread of a long job before it's done."

The authors reported a steep decline in reliability as tasks demanded longer sequences of tool calls.12 Agent accuracy dropped from 39 percent on tasks requiring the shortest tool chains down to 13 percent on tasks requiring the longest chains of tool calls.12 Jerzy told Primary that "every extra step is another chance to slip, so the longer the job, the worse the odds of getting all of it right, and accuracy falls off steeply as the chains grow." Jerzy observed that in longer sequences, an agent "either never gets around to a step it needed, or it collects the pieces and then stops short of assembling them into the final answer."

The researchers also addressed the difference between single-attempt testing and repeated success. Jerzy explained to Primary that "let a model try once and it looks clearly better than if you make it succeed three times in a row on the same task." Jerzy emphasized that this variance justified the stricter standard, noting that "a single lucky run makes an agent look more dependable than it is, and for anything you would actually deploy, dependability is the point."

How does effect scoring work across live servers?

DynamicMCPBench operates as an extensible framework rather than a frozen dataset, allowing engineers to generate new evaluation sets directly on their own tool servers.12 The system automatically creates realistic operational goals, runs an explorer model to find a successful path live, and extracts essential milestones, termed effect checkpoints, from that trace.14 When a candidate model attempts the same task, the framework evaluates whether each required effect was produced, ignoring superficial variations in tool choice or phrasing.12

Jerzy explained the philosophy behind effect checkpoints, telling Primary, "We never grade the agent on the exact path it took or the words it wrote; we grade whether the real result happened." Jerzy compared the evaluation to "checking that the room actually got cleaned rather than which hand held the broom." When building tasks, the researchers map equivalent routes so alternative solutions still count. "We have an optional AI helper that can judge the trickier 'different tool, same result' cases, but we switched it off for every result in the paper, so no score depends on a model's opinion," Jerzy said.

Autonomous software agents stumble when tasks require longer chains of tool calls
Source: Original article

To confirm that this automated grading matches human judgment, the authors conducted a human validation study, which measured a chance-corrected agreement score of 0.76 between the automated metric and human reviewers.12 Jerzy noted that the automated grader was intentionally tuned to avoid false passes, saying that "it almost never rewards a bad answer, and it pays for that caution by occasionally flunking a good one." Jerzy added that "when the grader and our human reviewers disagreed, it was almost always the grader being too harsh, never too generous."

What are the limits of dynamic benchmark scoring?

The reported benchmark results reflect performance on synthetic and recorded task trajectories across a specific collection of 121 servers, rather than an exhaustive test of every enterprise tool environment.12 The study is a preprint that has been posted online and has not yet completed formal peer review. In addition, while the chance-corrected agreement of 0.76 shows strong alignment with human raters, automated effect scoring can still occasionally misinterpret unconventional but valid problem-solving strategies.1

Jerzy cautioned that these scores should not be assumed to reflect private corporate environments. Jerzy told Primary that the servers were mostly English-language and openly available online, adding that "in our own work the real wall was tools locked behind logins we couldn't get past." Furthermore, the benchmark provided candidate agents with the necessary tools directly rather than requiring them to search through thousands of candidates. Jerzy noted that "when we buried the tools in a big catalog, scores dropped further." The authors also observed that most tasks involved reading information rather than modifying databases, limiting what the findings say about transactional systems.

These findings indicate that current language models remain fragile when coordinating actions across multiple interconnected systems. As developers build autonomous workflows that depend on chains of external tools, benchmarking frameworks that measure real operational state changes will be necessary to identify failure points that static tests overlook.

This piece was prepared from the arXiv preprint and public records together with answers from Jerzy to 6 questions from the Primary news team, completed September 2026.

The record

28 sentences trace to 3 sources and 1 contributor.

  1. 1 JK Jerzy Kamiński, Ilya Galyukshev, Artem Kuznetsov, Sergey Chuprin, Kirill Redko, Aidar Shumbalov, Anna Kalyuzhnaya Paper · arxiv.org · 1 Sep 2026 DynamicMCPBench: A Trace-Grounded, Effect-Scored Benchmark for LLM Agents over Live MCP Servers Preprint · may not have been peer reviewed See the source
  2. 2 A arXiv Third-party reporting · alphaxiv.org · 31 Aug 2026 DynamicMCPBench: A Trace-Grounded, Effect-Scored Benchmark for LLM Agents over Live MCP Servers See the source
  3. 3 JK Jerzy Kamiński Contributor · interview 11 Sep 2026 · in the article since 29 Oct 2026 Contribution — Jerzy Kamiński 15 statements added to this article
  4. 4 H huggingface.co Third-party reporting · huggingface.co TokenWasteGroup/DynamicMCPBench · Datasets at Hugging Face See the source

Article history

  1. Published 29 Oct 2026, 13:54
    Assembled by the Primary desk from 3 sources · 1 contributor · 28 cited sentences
  2. Jerzy Kamiński, computer scientists 15 statements added 49 days before publication
    1. Go to the sentence: Jerzy said in response to questions from Primary that "the tasks that stump everyone are the ones that string many steps together across different services."
    2. Go to the sentence: "Instead they run out of steam partway through: they start pulling the right information but don't finish putting it together into the result the task actually asked for," Jerzy said, adding that "they aren't picking the wrong tool so much as losing the thread of a long job before it's done."
    3. Go to the sentence: Jerzy told Primary that "every extra step is another chance to slip, so the longer the job, the worse the odds of getting all of it right, and accuracy falls off steeply as the chains grow."
    4. Go to the sentence: Jerzy observed that in longer sequences, an agent "either never gets around to a step it needed, or it collects the pieces and then stops short of assembling them into the final answer."
    5. Go to the sentence: Jerzy explained to Primary that "let a model try once and it looks clearly better than if you make it succeed three times in a row on the same task."
    6. Go to the sentence: Jerzy emphasized that this variance justified the stricter standard, noting that "a single lucky run makes an agent look more dependable than it is, and for anything you would actually deploy, dependability is the point."
    7. Go to the sentence: Jerzy explained the philosophy behind effect checkpoints, telling Primary, "We never grade the agent on the exact path it took or the words it wrote; we grade whether the real result happened."
    8. Go to the sentence: Jerzy compared the evaluation to "checking that the room actually got cleaned rather than which hand held the broom."
    9. Go to the sentence: "We have an optional AI helper that can judge the trickier 'different tool, same result' cases, but we switched it off for every result in the paper, so no score depends on a model's opinion," Jerzy said.
    10. Go to the sentence: Jerzy noted that the automated grader was intentionally tuned to avoid false passes, saying that "it almost never rewards a bad answer, and it pays for that caution by occasionally flunking a good one."
    11. Go to the sentence: Jerzy added that "when the grader and our human reviewers disagreed, it was almost always the grader being too harsh, never too generous."
    12. Go to the sentence: The study is a preprint that has been posted online and has not yet completed formal peer review.
    13. Go to the sentence: In addition, while the chance-corrected agreement of 0.76 shows strong alignment with human raters, automated effect scoring can still occasionally misinterpret unconventional but valid problem-solving strategies.
    14. Go to the sentence: Jerzy told Primary that the servers were mostly English-language and openly available online, adding that "in our own work the real wall was tools locked behind logins we couldn't get past."
    15. Go to the sentence: Jerzy noted that "when we buried the tools in a big catalog, scores dropped further."