Riot Games, the makers behind LoL, reflect on one of the hardest night shifts between the seasons.
An extreme recall bug forced Riot to disable ranked games in 2015. The LoL team worked night shifts to solve the issue.
That was today, two years ago, and the community is once again occupied with this on the second anniversary. How was it back then when the ranked mode was disabled due to this severe bug? Riot looks back on those intense 28 hours.
The Morning of July 15, 2015: “Oh my God, have you seen the video on Reddit?”
On July 15, 2015, a video from a user pointing out a serious bug reached Reddit. If you pressed keys in the “correct” order, Riven could instantly teleport back to base and skipped the actual cooldown (about 8 seconds) of the spell.
Those who exploited this could move across the map particularly fast and avoid any unwanted fights. A significant and unfair advantage over other participants in a match.
As the Riot team began working that day, more and more reports of this bug flooded into LoL headquarters. The message spread quickly within the team:
- Donna Mason, Release Manager “We are hearing in emails, pings, and from people on the team ‘Oh my God, did you see that thing on Reddit?’
- Tim Isenman, Live Producer “The first thing we saw was this Reddit post. Following that, we immediately started looking into it. There were also a few people from the champion team who were already looking for bugs.”
In the video, you see the Riven recall bug that was posted on Reddit:
When such a bug appears in League of Legends, the main goal is to reproduce it. When the developers themselves know how to trigger the bug, working on a solution is significantly easier. They also distinguish between “Is there a bug in the game? That’s not good” and “There is a bug in the game that gives an unfair advantage. That’s particularly bad!” Then they test whether they can exploit the bug themselves.
- Matthew Wittrock, Release QA “So I fiddled around and finally I could reliably reproduce the bug. ‘That’s really easy – that’s not good!’
As the video spread quickly and generated more clicks, the team decided to disable the champion Riven. There was too much danger for the community that games would be ruined by imitators. Riven was disabled worldwide, and the team began working on a hotfix.
Afternoon of July 15, 2015: Disappointment – Not only Riven is affected
It is now 14:50 and the team has disabled Riven worldwide. Work on the hotfix is already in high gear as more reports reach the team. Apparently, more than just one champion is affected by this bug.
- Tim Isenman “After we disabled Riven, we received more pings from Riot employees pointing to new videos. It became clear there that other champions in League of Legends were also affected by this bug. First Yasuo, then Graves, and soon many others …”
- Mark Sassenrath “More and more people wrote to us that they could reproduce the bug on Shen and champion X, Y, and Z. The list kept getting longer”
- Tim Isenman “When we then found out that the bug was applicable to about a quarter of all champions in LoL, our hearts sank“
But even here, the extent of the problem was not entirely clear. The team would soon discover that the bug was huge!
- Matthew Wittrock “At that point, we still underestimated the problem. We assumed that there were certain abilities triggering this bug. We did not know it affected every champion.
- Tim Isenmann “And then we found out that every champion using the item Tiamat or Hydra can trigger the same effect. Now any champion could exploit the bug.
Even without the mentioned items, there were 40 champions that could trigger this bug. When it became clear that any champion could exploit the bug and thus only advantages and no disadvantages for the players existed, a quick solution was necessary.
“We can’t disable all champions, what do we do instead?” It was considered to disable ranked games. However, most players at this point only knew that Riven could cause the bug. That it could also happen with other champions had not yet reached many players.
The developers decided to keep ranked games activated until the bug became a “real problem.” For now, only Riven was disabled.
The Riot team decided to contact the moderators of Reddit. They wanted to draw as little attention to this issue as possible but still wanted new information about the bug. Therefore, they suggested importing new videos into the already existing thread to collect the results. However, this led to the creation of a “mega thread” that listed all videos prominently and led to even greater attention than initially feared.
Now the Riot team was clear: We need to disable ranked games. All champions that could produce the bug were listed on Reddit. Additionally, there was also an instruction on how to execute the bug. The danger for ranked games had thus become too great, and the bug had become the mentioned “real problem.”
Evening of July 15, 2015 – The solution plan is implemented
Riot is now deploying multiple teams to work on a solution to this severe bug.
- Matthew Wittrock “I remember that we sat down with different teams. Then it was ‘We know this’, ‘What do we do about it?’ and then ‘Cool! Go, everyone starts on their work!’ – That was exciting. The bug is really bad, but we have a plan and are quickly working on a solution.”
Riot then had to explain to the staff how the bug works so that they could test together in custom games whether the bug still exists.
After four failed attempts at a solution, the team finally had a solution that satisfied everyone. This was then sent for testing to the quality assurance team. The bug was tested with all possible champions and item combinations.
The quality assurance team would require approximately 12 hours for this. It was now between 10:00 PM and 11:00 PM. Riot decided to send some employees to bed. The quality assurance team was told to wake everyone up if the hotfix does not work and the bug continues to occur.
02:00 AM, July 16, 2015 – Disappointment again – There are problems again
Around 2:00 AM on July 16, the team was informed that the hotfix was not working as desired. Everyone was awakened, the engineering team, the designers: all had to get up.
- Mark Sassenrath “You know something is wrong when your phone goes off at 2:00 AM. Either you forgot something, or you’ve made it worse.”
Two new hotfixes were developed in the next two hours. By around 4:00 AM, the mood was correspondingly tense. But now testing had to be done again.
By the afternoon of July 16, the ranked games had been disabled for already 19 hours. Players were promised that they would report back in a few hours and update them on the status of the bug. This gave the team more time to work on a solution.
- Tim Isenman “Around 4:00 PM, we were informed that the latest bug fix showed positive results for 80%-90% of the champions. That was good news. However, we had difficulty communicating it to our players because there remained the chance that the solution might not work for all champions and that the team would have to start from scratch again.” Therefore, they decided to remain calm and say nothing.
5:00 PM on July 16, 2015 – Ranked games down for a day
It has now been about 24 hours since the ranked games were disabled. The final bug fix has now been positively tested on all champions, and the Riot teams are beginning to deploy the fix worldwide and reactivate ranked games.
- Tim Isenman “I believe it was around 5:00 PM when we were told that the fix was 100% successful. One region after another began receiving the bug fix.
- Donna Mason “Since we have so many game servers, deploying the bug fix took a lot of time. But we hurried”
- Tim Isenman “In the end, our team in Vietnam took a long time to respond. All of us were gathered around the monitor, waiting for feedback from Vietnam. Then someone from there writes back, “I still have the problem” and we almost went crazy.
- Matthew Wittrock “That’s the biggest fear. No matter how much we’ve tested. We always wait for this one post saying, “Hey, I still have the bug.”
- Tim Isenman “We told him, ‘Restart the game and test it again.’ He did, and we waited and waited … . Finally, he said ‘I cannot reproduce it. I think it’s fixed!’ Before, we were really in a panic because if it still existed in one region, it would be present in all regions.
8:00 PM on July 16, 2015 – The ranked servers are operating again
After now 28 hours of downtime, the ranked servers are reactivated. The bug has finally been fixed.
- Tim Isenman “We kept the servers offline for five more minutes to test internally. Then we activated the ranked servers and were finally able to inform the players that everything was back up and running. The demon was defeated!
- Matthew Wittrock “Now was not the time to worry. If something didn’t work, we would find out in the next few hours. We have to take care of other things and move on.
And thus the work at Riot Games continued, addressing the next issues. Dozens of employees worked many hours on this “Great Recall Bug of 2015” to quickly reactivate the ranked servers and return players to ranked games. It is considered one of the most severe bugs in the history of League of Legends.
This player lists 180 bugs with Mordekaiser – It’s his hobby
