The Comedy of Errors That Let China-Backed Hackers Steal Microsoft’s Signing Key

Microsoft said in June that a China-backed hacking group had stolen a cryptographic key from the company’s systems. This key allowed the attackers to access cloud-based Outlook email systems for 25 organizations, including multiple US government agencies. At the time of the disclosure, however, Microsoft did not explain how the hackers were able to compromise such a sensitive and highly guarded key, or how they were able to use the key to move between consumer- and enterprise-tier systems. But a new postmortem published by the company on Wednesday explains a chain of slipups and oversights that allowed the improbable attack.

Such cryptographic keys are significant in cloud infrastructure because they are used to generate authentication “tokens” that prove a user’s identity for accessing data and services. Microsoft says it stores these sensitive keys in an isolated and strictly access-controlled “production environment.” But during a particular system crash in April 2021, the key in question was an incidental stowaway in a cache of data that crossed out of the protected zone.

“All the best hacks are deaths by 1,000 paper cuts, not something where you exploit a single vulnerability and then get all the goods,” says Jake Williams, a former US National Security Agency hacker who is now on the faculty of the Institute for Applied Network Security.

After the fateful crash of a consumer signing system, the cryptographic key ended up in an automatically generated “crash dump” of data about what had happened. Microsoft’s systems are meant to be designed so signing keys and other sensitive data don’t end up in crash dumps, but this key slipped through because of a bug. Worse still, the systems built to detect errant data in crash dumps failed to flag the cryptographic key.

With the crash dump seemingly vetted and cleared, it was moved from the production environment to a Microsoft “debugging environment,” a sort of triage and review area connected to the company’s regular corporate network. Once again though, a scan designed to spot the accidental inclusion of credentials failed to detect the key’s presence in the data.

Sometime after all of this occurred in April 2021, the Chinese espionage group, which Microsoft calls Storm-0558, compromised the corporate account of a Microsoft engineer. With this account, the attackers could access the debugging environment where the ill-fated crash dump and key were stored. Microsoft says it no longer has logs from this era that directly show the compromised account exfiltrating the crash dump, “but this was the most probable mechanism by which the actor acquired the key.” Armed with this crucial discovery, the attackers were able to start generating legitimate Microsoft account access tokens.

Another unanswered question about the incident had been how the attackers used a cryptographic key from the crash log of a consumer signing system to infiltrate the enterprise email accounts of organizations like government agencies. Microsoft said on Wednesday that this was possible because of a flaw related to an application programming interface that the company had provided to help customer systems cryptographically validate signatures. The API had not been fully updated with libraries that would validate whether a system should accept tokens signed with consumer keys or enterprise keys, and as a result, many systems could be tricked into accepting either.