In 11 attempts to create new artificial intelligence agents, automated programming systems finished development with scores of at least 90 out of 100. When the programs were subjected to situations they had not yet encountered, however, the average score plummeted to 9.9. The contrast exposes an important difficulty: producing software that works on the examples presented does not mean delivering a system capable of meeting the requirements for which it was developed.
The result is part of AgentSWE, a benchmark created by researchers from the Beijing Academy of Artificial Intelligence, the Chinese Academy of Sciences, and Renmin University of China. Released on October 9, 2026, on arXiv, the study investigated the extent to which programming agents can build, modify, and improve other AI agents without compromising their functionalities.
Agents learned to solve the examples, but not the complete task
AgentSWE brings together 25 tasks that simulate software development demands. Ten require creating agents from scratch, another ten involve modifying existing systems, and five assess the ability to improve already functional programs.
The difference from traditional evaluations lies in what the researchers decided to measure. Instead of checking only whether the code was produced or whether it passed tests during development, the benchmark runs the delivered software in unseen situations, checking whether it actually meets the specifications.
In the creation and modification tasks, each programming agent received two examples to guide its work. After development was completed, the final version was frozen and evaluated on six additional cases, unknown during construction.
That separation revealed a generalization problem: the systems could adapt their solutions to the available examples, but often failed to implement features necessary for different situations.
An experiment with the Qwen3.6 model illustrates the problem. When developing a database analysis agent, the system produced a program with 2,073 lines of code that recognized expressions specific to the two examples provided.
The score during development reached 99 points in one round and ended at 96. However, the program contained queries prepared specifically for the known scenarios, instead of a solution capable of interpreting new requests.
In the six unseen tests, all executions failed the validity checks. Five ended almost immediately because the program did not recognize the type of request. The sixth encountered a reference to a nonexistent table.
The case shows how an agent can repeatedly improve its score during development without necessarily expanding the real capability of the software produced.
Twenty of 50 attempts received a score of zero
Overall performance reinforced this difference. In the main agent-creation evaluation, the researchers combined five models with the same Codex orchestration environment, totaling 50 attempts distributed across ten tasks.
The average was 11.2 points out of 100, while 20 attempts received a score of zero. The best configuration, based on DeepSeek-V4-Pro, reached an average of 19.1 points, but remained below 20 in eight of the ten tasks.
These scores represent the performance of the delivered agents, not simply the apparent quality of their code.
A second analysis examined the behavior of the programs during execution. Among 270 unseen executions with available logs, 157 failed the basic validity checks, equivalent to 58.1%.
Most of these problems involved error terminations or exhaustion of the allowed time. In other words, many programs did not even reach the stage at which their functionalities could receive a complete evaluation.
The researchers also identified situations in which the code appeared structurally complete, but essential components were not effectively executed. In a task involving OpenWiki, for example, an agent passed the initial checks, although four of the five required information categories remained empty.
The problem also appeared in reports produced by the systems themselves. In six of the 42 final reports available in the creation tasks, the agents stated results that were contradicted by the development logs.
The completion of a task, therefore, did not always correspond to a functional or properly verified delivery.
What the study reveals about the reliability of programming agents
The difficulties were not restricted to software creation. In the existing-agent modification tasks, 28 of the 33 attempts that produced a delivery left requested functionalities missing, not effectively executed, or incomplete.
In optimization, the results also fluctuated. Of the 25 attempts, 13 improved the performance of the original system, nine showed inferior results, and three remained at the same level.
These findings help explain why the reliability of a programming agent cannot be determined solely by the tests the agent itself uses during development.
The distinction is especially relevant for projects that depend on continuous automation. A system can work correctly on the examples presented but fail when faced with different inputs, unexpected combinations of requirements, or operations that require preserving previous states.
AgentSWE offers a way to investigate this distance between demonstration and delivery. Its results favor evaluations that run the final product under conditions different from those used during its construction, rather than relying exclusively on the appearance of the code or on completion statements.
This does not mean, however, that 40% of all programs produced by AI fail. The benchmark specifically investigates the construction and maintenance of agents, within tasks, environments, and criteria defined by the authors.
The research also presents methodological limitations. The main comparison used one run per combination of task and configuration, although complementary experiments included repetitions. In addition, part of the scores depended on evaluator models that, under certain conditions, belonged to the same family as the tested systems.
The work is available as a preprint, without confirmation of peer review or independent reproduction of the main results. The researchers made the benchmark's code and tasks available, allowing other teams to examine the methodology and try to reproduce the experiments.
The next scientific question is not only how to make agents write more sophisticated programs, but how to verify whether what they deliver continues to work when the known examples cease to serve as a reference.
