AI company Anthropic reports that in September, its AI models have been used in a cyberattack on 30 US government agencies, banks, large tech companies and chemical production corporations, reportedly orchestrated by a Chinese state-funded hacker group. Four of these attacks were successful in retrieving sensitive data from protected databases. The attacks were carried out by the AI for 80-90%, with only 10-20% being done by human hackers steering the AI agents. The AI ‘safety guardrails’ were easily surpassed by the hackers telling it that the AI was working ‘as part of a cybersecurity safety test for the companies and agencies’.

There are currently no guardrails in place that cannot be surpassed, the companies have no idea how they would do that, or if it is even possible to not have these models jailbroken.

While Anthropic is the only company that has reported on such a large scale cyberattack, every large AI tool like ChatGPT or Google Gemini is used for hacking, scamming and fraud. If these companies are allowed to continue, the AIs will get better at all of the above and the end picture could look like no information or money being safe on the internet.

Sources:

https://www.axios.com/2025/11/13/anthropic-china-claude-code-cyberattack

https://www.anthropic.com/news/disrupting-AI-espionage