Recently, there have been significant issues with Destiny 2 after updates. Bungie is providing internal insights as the developer clarifies what went wrong and how they plan to avoid such situations in the future.
What problems are we discussing? With the reset on February 11, a small update was introduced intended to fix some issues and kick off a new event. However, after about an hour, emergency maintenance occurred and the servers were taken offline.
The issues even led to the last resort: a rollback. This has now happened for the second time in Destiny’s history. The last time was just two weeks ago when Update 2.7.1 caused significant problems. Bungie wants to provide clarity now and is revealing their hand.
Destiny developers address the problems
How did the original problem occur? Bungie is now addressing this in a letter to the community. It details the causes of the issues that led to the two emergency maintenances and the rollbacks.
To understand how the loss occurred in player inventories, Bungie mentions two causes:
- A bug in the inventory menu
- Problems with the servers
Problem 1: The Inventory Menu
This is how inventory management works in Destiny 2: Whether quests, items, or materials and currency, the entries in your inventory as well as quest logs are time-stamped. This helps the game know when you first received each item.
Each time you log in, Destiny checks all time stamps. It verifies whether all items or missions are correctly ordered and whether all resources do not exceed their maximum cap.
The described time stamp has caused several problems for a while now and has been investigated by the developers. The new fix was deemed critical and reviewed, but initially, no error could be identified by the domain experts.
When Update 2.7.1 went live two weeks ago, it was believed that the problem had been sufficiently resolved. However, issues still occurred sporadically. Combined with Cause Number 2, this led to the catastrophe.
Problem 2: The Servers
New servers cause trouble: At the launch of Shadowkeep last October, Bungie stated that significant investments were made in new servers. Through the new world servers, the developer has more capacity than ever before and wanted to be prepared for the influx of players for the expansion.
A small percentage of the new servers (under 1%) have been problematic since then. The affected servers crash during new updates, requiring manual restarts.
The solution to the problems?
How was the “2.7.1” situation resolved? To address the consequences of the problematic update, a patch was quickly applied to the servers instead of trying to deploy a full build of the game code.
This involved changing the server settings to override the game code used for processing character data and then restarting the world servers to implement this change.
To fix that issue quickly, we applied a patch to the servers instead of trying to get a full build of the game code deployed. This involved making a change to a server setting to override the game code used to process character data and then restarting the WorldServers to pick up that change.
Bungie
Why did the problem occur again? Unfortunately, as often happens, a small percentage of the world servers crashed when the update 2.7.1.1 went live. The affected servers were manually restarted, and everything seemed to be fine, the guardians played happily.
However, again there were complaints from a few players about missing resources. Instead of checking and fixing each affected player individually, emergency maintenance occurred, leading to the second rollback for everyone.
It turned out that the manually restarted servers had not correctly loaded the fix from the previous patch. Anyone playing on these servers experienced the problems. During internal tests, employees had only encountered the “good” servers (the 99%).
Into a secure future
New security measures: Bungie aims to ensure that such issues can be resolved more quickly in the future or, ideally, do not arise at all. These 7 precautions are intended to help:
- Starting today, there will be a new security measure for “hot patching” to ensure that servers cannot start with an incorrect version.
- From Season 10, faulty world servers should no longer crash during updates. The issue has already been identified but cannot be implemented sooner.
- A fix for characters affected by the timestamp issue will be made with the next hotfix.
- Rollback and recovery measures should happen faster in the future.
- In a future version, Bungie plans to address the issue that may cause servers to skip loading configuration data.
- More protection will be integrated into the code for clearing login accounts to prevent future errors from being introduced into such a critical area.
- Bungie is updating their development methodology to catch issues like this earlier in the release pipeline.
Currently, the Destiny team is feverishly working on a new hotfix that is planned to go live tomorrow. This will also address the warlock super glitch.
Developers’ apology: At the end of the report, Bungie apologizes to the guardians for the issues of recent times:
We know that today’s outage and the rollback of characters were frustrating for you, especially with the start of Crimson Days, just as it was frustrating for us to realize that this is a problem we could have avoided.
We apologize for the frustration and inconvenience this has caused, and we will continue to work to ensure that such things do not happen again.
Bungie
What do you think of the insights into Bungie’s internal procedures? Do you believe that the problems are a thing of the past due to the measures taken?
