An automated system developed by Anthropic achieved better results than the best human participants in 23 of 24 safety comparisons, according to data released by the company on Monday the 19th. The tool was created to identify ways to bypass mechanisms that prevent the use of artificial intelligence models in high-risk activities.
In the remaining comparison, the system matched the best human result. Anthropic says the tool completed the tests in 24 to 48 hours, while the bounty programs used as a reference remained open for weeks or months.
The result allowed the company to bring forward the completion of a goal in its Frontier Safety Roadmap, originally scheduled for January 2027. The goal was considered met on October 6 and formally removed from the safety roadmap on the 19th.
How Anthropic compared automation and human researchers
The evaluation used eight bounty program challenges and three generations of safety classifiers used in production, totaling 24 combinations.
These classifiers function as barriers designed to identify and block dangerous requests. The tests focused on safeguards related to biological weapons, where researchers tried to find techniques capable of circumventing the restrictions.
The automated results were compared with the best human submissions received in the programs held between April 20 and September 11, 2026.
Anthropic considers that the vulnerabilities identified by the automated system were, collectively, more substantial than those discovered by the participants in these programs. This assessment meets the criterion previously established by the company for considering the goal completed.
The numbers, however, do not demonstrate AI's overall superiority over safety experts. The comparison covers a specific set of challenges and protection mechanisms, and the results were released by Anthropic itself, with no independent validation presented.
The company said it intends to publish a more in-depth technical report on the experiment but has not yet set a date for its release.
