What happened. Late last night, a licensing service that sits outside of XBOX, but which XBOX depends upon, began failing. This caused some sign-in scenarios to fail, and it caused many scenarios that require an entitlement check to also fail, such as listing out your Full Library and launching games that you own. It also affected several of our publishing and store partners who depend on those same systems which is why some of you saw issues in specific games and not others.
What we did. Our on-call teams caught the first signals overnight through automated monitoring and declared a major incident. Once we isolated the failing infrastructure, we shifted traffic onto the healthy parts of the system while the team continued to work towards a root cause. Service came back unevenly across regions which is why some of you recovered hours before others.
What's next. We're running a full post-incident review. I care less about the one-line root cause and more about the real questions: why a failure in one service was able to take down this much, why recovery took as long as it did, and what we change so a single point of failure can't ruin your night again. That means hardening the dependencies underneath sign in and game launch, improving how we detect and contain this class of failure, and being faster and clearer with you when something breaks.
winfuture.de
