On June 27, the limited-time game mode Playground was supposed to launch. However, after a short time, the servers of Fortnite went down – no matter the game mode. What happened? What measures is Epic taking to prevent such catastrophes in the future?
The joke with Playground: While normally 100 players can enter a match, the Playground only allows 1-4 players. The capacity of the Playground is therefore tiny. If one is full, a new one must be opened. This small difference recently killed Fortnite.
This is what happened: The Sternschanze in Hamburg is a popular nightlife destination with many bars. Let’s assume that each bar can allow 100 guests at the same time. All people strolling through the Sternschanze can freely choose where to go for a drink.
Now, if the number of allowed guests in each bar is reduced to 1-4, the following happens:
- Many people have to stand outside the pubs, cannot gain entry, and get angry.
- New pubs cannot be opened quickly enough to accommodate all the people.
In the case of Fortnite, the pubs are servers, and the slightly intoxicated guests are the players.
What went wrong with the introduction of Playground
Playground requires significantly more servers: Fortnite’s matchmaking is controlled by a mechanism called Matchmaking Service (MMS). This assigns players to free servers. Preferred regions and server load are taken into account.
If we take our analogy of the Sternschanze, the MMS acts as a kind of guide that selects the suitable bar for guests and accompanies them there.
If now not 100 players are on a server (as is usual in a normal match), but only 1-4 (depending on squad size), the required number of servers could potentially increase a hundredfold because the total number of players has not changed, but the servers they need have.
Epic’s previous measures were not sufficient: Even though Epic prepared for the “Playground” event, the servers were suddenly confronted with a gigantic number of players searching for a free server simultaneously. Additionally, as the Matchmaking Service now had to examine significantly more servers for free capacities, the process slowed down further.
Like a drunken party guest, the search routines stumbled over their own feet but could no longer stand up on their own – it led to a crash. Since the server clusters between the Playground and the other modes were not separated, all running modes in Fortnite were affected – a global outage.
Epic’s measures to restore services
Playground is separated from the rest of the game: Epic’s first measure was to separate the server cluster of the Playground from the rest. This was intended to prevent further problems with the Playground from impacting other modes.
Furthermore, Epic could now make unlimited changes to the servers to react promptly to service limitations. What followed were profound changes in the server architecture.
Epic succeeded in making the system more dynamic. Player access to matchmaking was significantly accelerated, and free capacities were identified more quickly.
Epic pushed the servers to the limit during testing: Every change required a stress test of the architecture. For this purpose, millions of virtual accounts were created, all accessing the system simultaneously. In this way, further bugs in the system were identified and fixed.
Only when everything was fine-tuned and no more crashes occurred, was the Playground finally reopened on July 2.
What has Epic learned from the drama? According to their own statement, Epic has learned a lot about their own matchmaking and the potential sources of errors within it. The changes made during the work on the Playground were correct and necessary.
The insights gained from the crash will also be helpful in the future for new modes and player influxes.
Epic also admits: It was harder than expected. The problems were not foreseen in this way, but they will be helpful for the future of the game.
What do you think? Were you very upset about the server crashes?
The Playground is currently inactive, but it could still become very important for Fortnite!
