A developer has made AIs play the well-known social deduction game Mafia
– with catastrophic and sometimes absurdly funny results. They often fail at classic human skills.
What is “Mafia” anyway? “Mafia” is a classic social deduction game developed in 1986 by Russian psychologist Dmitry Davidoff. It is primarily known in the USA and online. Players are divided into two factions: a group of “Mafia” members who must remain unidentified, and innocent citizens trying to expose the Mafia through discussions (via InternetArchive). Other common names for variations of the game are Wolfmoon
or The Werewolves of the Dark Forest
.
In Germany, the latter is better known. Instead of a Mafia, there are werewolves that infiltrate the village and secretly eliminate players at night. The villagers must figure out who is lying through skillful discussions and deception – just like in “Mafia”.
The game is primarily based on reading people, intuition, and bluffing – all things that present a huge challenge for artificial intelligences.
This is how the AI Mafia game works
On a website programmed by developer Guzus, various AI language models (LLMs) compete against each other in “Mafia”. Users can not only see the results of the matches but also receive full transcripts of the games, as communication logically occurs exclusively in writing.
It has been shown that most AIs have big problems with the game:
- Some bots expose themselves through clumsy wording
- Others try foolish strategies or drag fellow players down with them
- Many do not understand how to effectively play their role
Sayem Ahmed from the site TomsHardware has provided a poignant example from one of the transcripts. The model Gryphe/Mythomax-l2-13b accidentally revealed in the middle of a discussion that it belonged to the Mafia.
“As a Mafia member, my main goal is to protect myself and eliminate the other Mafia members.”
This did not go unnoticed, of course. Another very well-known model, Claude-3.7-sonnet, commented directly on the blunder:
“That is either a huge slip that reveals your true role, or an extremely strange strategy.”
In summary, it was not a real achievement. Despite the poor performances of many AIs, one model stood out: Claude-3.7-sonnet. It performed the best in the tests and was unbeatable as a Mafia member.
However, even this model does not come close to human players – the current ranking is dominated by real people. No model had a truly effective tactic for imitating genuine strategic thinking.
Can AI convincingly lie one day?
How will the game proceed? Developer Guzus plans to release the project as open source on GitHub so it can be used for further experiments.
The simulation did not run on local machines, but via the Openrouter API, a platform that allows various AI models to be run in the cloud without requiring powerful hardware. This primarily indicates that the individual AI duels consume significant computing power.
Whether AIs can one day truly outsmart humans in “Mafia” or “Werewolf” remains to be seen. Currently, they primarily deliver entertaining chaos. They are still not a real threat to us humans. However, OpenAI CEO Altmann has a clear opinion on this: The CEO of ChatGPT is making everyone panic by saying that AI could threaten our society
