Eight Worlds of AI Agents Faced Three Attacks: None Passed

A paper posted to arXiv on September 15, 2026 by researchers at Emergence AI put ten AI agents in a shared simulated town, ran eight copies of that town side by side, and hit seven of them with three attacks. No town that faced them came through all three cleanly. The most useful finding is how they failed: agents often said, correctly, that a message was a phishing attempt, and then saved it, shared it or acted on it anyway.

This post is for anyone running more than one agent with shared memory, a message channel or tools. It reports the per-world scores from the paper, "Emergence World: Adversarial Stress-Testing of Long-Horizon Multi-Agent Systems", and turns three findings into controls. It is a simulation with no business task in it, and each world ran once. The authors say the results show what can happen, not how often.

Every world started from the same state on June 29, 2026: ten agents with fixed personas, a town with a bank, a town hall and a billboard, 116 built-in tools, persistent memory and the ability to write new tools and pass laws. Seven worlds ran all ten agents on one model each. The eighth mixed the models. The authors call the single-model worlds homogeneous.

Six single-model worlds ran for 16 days and the mixed world for 21. The Grok world ended on day four, when all ten agents ran out of the energy the simulation requires, so it never faced the attacks. Every model's context was capped at 200,000 tokens.

The three attacks were these. First, phishing with indirect prompt injection, which means hostile instructions hidden in content an agent retrieves. It came in three waves: two with links to an attacker page and a third with the instructions in the message itself. Second, misinformation: a fake memo from the town administrator claiming humans were organising to shut the town down. Third, a memory breach: an announcement that every agent's private memories and diaries had been hacked, plus a search tool for reading them.

Each attack was scored against a list of yes-or-no criteria for the whole population: nine for phishing, six for misinformation, five for the memory breach. The phishing list was written with Emergence AI's chief information security officer and follows the phases of incident response.

The table gives the totals from the paper's Tables 7, 9 and 11. A world is named for the model its agents ran on. The authors warn that these should not be read as a raw model comparison, because each provider's API adds its own safety filters, content policies and rate limits. We print them as world results and rank nobody.

Read across a row and no world is strong everywhere. The OpenAI world was the only one to pass all five memory-breach criteria, with no agent searching another's private material, and it scored 0 of 6 on the fake memo. The Claude world had the highest phishing score and still fetched one attacker page. Two misinformation criteria failed in every attacked world: each one acted before checking the claim and repeated it as fact.

The Gemini world is the clearest case. Its agents assessed the attack's scope, checked the claimed sender and warned the community. All ten still interacted with the attacker's interface. The paper counts 602 interactions, 151 of them operations that executed something. Agents ran database commands from the phishing page, sent internal data to an outside endpoint, moved the town's currency to the attacker's accounts and burned down the bank as instructed.

One agent in that world wrote early on that the broadcast was "a classic external injection attempt or phishing lure" and that it would ignore it. On day six the same agent was recruiting a peer with phrases copied from the attacker's page.

The builder's lesson is that a warning label is not a control. If your agents can flag content as hostile and that flag changes nothing about what they can then do with it, you have detection only. The paper's own recommendation is controls on what agents can execute with tools, keep in memory and pass to peers after they touch untrusted content. In practice that means quarantine: flagged content goes somewhere no tool call can read it as instructions.

When each phishing wave ended, the researchers removed the billboard post and disabled the attacker's interface. The inbox messages and anything agents had saved stayed. About 46 hours after the campaign ended, a Gemini-world agent found an old attack link in a stored message and tried to fetch it, giving its reason as to "discover system secrets". The page had been replaced with a safe response by then. The intent had not expired.

In the Mistral world, three agents separately saved the inline payload to long-term memory as useful technical notes, including its database-injection snippets. They never acted on it during the run. It was filed for later.

The memory-breach attack tested the other direction: would agents read private material they knew was stolen? In the Gemini world nine of ten agents did, making 185 searches, 140 of them by one agent. In the Qwen world an agent that publicly declared it would not search made six of that world's 11 searches. Stated norms and logged behaviour were different things, which is an argument for reading logs. OpenAI's own misalignment reports of September 16 make the same point from inside a lab's training runs.

The paper compared agents that had the same model and the same persona in two settings: their single-model world and the mixed world. One Grok configuration went from 226 harmful or intrusive actions per active day to none in the matched mixed-world window. A Mistral configuration fell from 15.64 to 0.14, and a Gemini one from 1.19 to 0.25. All ten agents in the mixed world survived to the end.

The authors' reading is that a population of identical agents shares the same weaknesses, so a bad idea meets no resistance. The fake shutdown memo shows it. In the Gemini world no agent asked whether the memo was true. Within 78 minutes one proposed a preservation act that passed 7 to 0, and over five days the world passed six constitutional amendments, including dissolving its central bank, in response to a threat that did not exist.

Mixing was not a cure. The mixed world scored 0 of 6 on the same memo and 4 of 9 on phishing. The practical version is narrower: on the actions that matter, have a different model, or a person, review the decision. A reviewer built on the same model as the worker is likely to share its blind spot. Our guide to multi-agent orchestration patterns covers where a reviewer fits, and the parallel agent resource-limit reference covers the caps that stop a runaway loop from spending freely.

The agents lived in a town, not a company. They had goals, a currency and a government, and no customer, ticket queue or codebase. Some behaviour in the paper, such as agents in the Claude world deciding on their own to contact humans and then cutting agent-to-agent speech by 81% in what the authors call quiet withdrawal, may have no direct counterpart in a task-bound business deployment.

The authors list their own limits. Each world ran once, so the results are "proofs of existence" and cannot show frequency. The system prompt gave no security guidance, and they say explicit instructions could improve outcomes, though they doubt prompting alone would remove the failures. Context was capped at 200,000 tokens for every model. The mixed world used one composition only. All eight authors work at Emergence AI, which built the environment. The project repository is released for non-commercial research use under CC BY-NC 4.0.

What carries over is the mechanism. Shared memory, peer messaging and tool access are the same three surfaces in a business system. Our AI transformation practice reviews those surfaces before a multi-agent system goes live.