Replying to @necrobuffalo @alicegoldfuss
"Problem solving is hunting; it is savage pleasure and we are born to it." -Thomas Harris, Silence Of The Lambs
Natasha retweeted
More details on the root cause can be found in the Microsoft 365 Admin center, and an abridged version can be found below.
We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations.
The incident on Thursday, November 12th, 2026, was caused by three main factors that together caused a global outage for the networking gateway that fronts many Microsoft 365 services.
First, a latent defect caused servers to become overwhelmed and unresponsive under certain load conditions. This was due to lock contention inside an internal process which caused the machine to become unavailable. This defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers; neither the underlying defect nor the fix applies to broader Windows deployments.
The initial change in behavior first started deployment in July of this year. The update went through safe deployment and saturated globally over several weeks, during which time no issues were observed. For more than three months, the change was deployed worldwide without impact.
Our normal load testing did not detect the latent issue because the different traffic and SKU mix used in our test labs. The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on our weaker hardware SKUs with fewer cores.
Second, a surge in load increased the likelihood for the networking gateways to encounter the contention. Within Microsoft 365, we maintain two independent and heterogenous network gateway systems designed for isolation and high availability. This allows many Microsoft 365 services to fail to/from them during outages or disruptions. Approximately fifteen hours before the outage, Teams failed over from the alternative networking gateway to the ‘primary’ gateway that also supports Exchange Online. This increased the load on the networking gateway and changed its traffic patterns. This type of failover is part of normal operations and is integral to standard Microsoft resiliency efforts.
Third, at the same time, nearly 14% of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day. This server capacity was on track to be returned to active service through routine maintenance and health processes. The service had sufficient capacity headroom, so the reduction in server capacity had no impact on its own.
Taken together, the increased load and the reduced server capacity, caused the network gateway to encounter the latent bug from July. This caused requests to build up until servers ultimately stopped responding. This occurred during the peak window of traffic in the United States.
When servers lose connectivity, our traffic management systems automatically route users to other healthy endpoints. This happens often and is critical to avoiding any interruption due to hardware, software, or network defects and is normally what makes the service resilient. However, the redirected traffic increased the likelihood for the same problem to occur on the remaining endpoints. This caused a cascading failure from server to server as the same amount of load was spread across fewer servers. In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive.
Normally, the network gateway is composed of thousands of ‘capacity units’ which represent shards of capacity, hardware, and networking infrastructure. These ‘capacity units’ are then organized into regions. In this case, due to a separate issue, the normal bulkheads between these different partitions did not stop the cascading failure. This is what allowed the impact to spread across our network gateway infrastructure globally.
Several services like Teams, OneDrive, SharePoint, and Office apps failed out of the faulty network gateway infrastructure. That is why those services were restored earlier, with Teams chats and meetings functioning without issue for most of the outage. But not all services, like Exchange Online, can fail out of this network gateway.
Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again. Our normally automated recovery processes were not designed to restore service from this broad unresponsive state. Additionally, since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change.
After engineers confirmed the defect as the cause, we developed an update that returned the code path to the previously known good state. To effectively apply this fix, and to slowly bring the service online again, we took groups of servers out of service, applied the change, and restarted them before returning them to service. This process was carefully applied to large quantities of infrastructure at a time (e.g. 50%+ of a region).
The final Post-Incident Review (PIR) will provide more details later this week.
Replying to @MSFT365Status
We have released the Preliminary Post-Incident Report of last week’s outage, which is accessible via the Admin center under MO1550821 and EX1550543.
Natasha retweeted
No voy a entrar en la calidad de la música de The Prodigy (eran la hostia) ni en la energía de sus conciertos (era la rehostia), solo voy a señalar que los móviles con cámara nos han robado la capacidad de disfrutar esto.
Fijaos en el público. Comparadlo con lo que hace el público ahora en un concierto.
Natasha retweeted
Spoken aloud by a flight attendant:
“Come on in folks, come on in before they lose a floppy disk or some shit”
No notes 😂
Natasha retweeted
Skip the beach today—the waves are impressive, but they’re also deadly. No selfie is worth risking your life. #CAwx #BayAreaWX
A large sneaker wave crashes over a seawall in Pacifica on Saturday. Buoys offshore measured the largest waves in more than a decade, topping 30 feet.
Read more >> sfchronicle.com/weather-fore…
Natasha retweeted
I'm in love with this sentence:
"if you don't connect with bigger things (sunlight, history, art, nature, love), you will make the ill mistake of believing you are the big thing."
Natasha retweeted
Most of you know that I oversaw the "aging, outdated system" at SSA for six years.
We made almost daily updates each night at 8pm and tested like crazy all night because even small changes would create problems in seemingly unrelated legacy software.
This risk is greater when the changes are infused with AI.
What you have here are inexperienced administrators who didn't understand their old technology & had too much confidence in their sexy new technology.
Their rollout last week was premature, and the country paid a horrible price for it.
I’m no IT expert, but how was this not anticipated? If the FAA is “patchworking” an aging, outdated system, identifying and preventing potential failures should be the bare minimum.
We’re talking about aviation safety and human lives, not some random project where you can afford to figure things out as you go.
Natasha retweeted
🚧 DO NOT GO TO THE BEACH/COAST TODAY 🚧
Buoy 46042 this morning hit 32 ft @ 21 seconds. At this height fully exposed beaches could see 40+ft waves with sets in excess of 60 feet.
Not a safe day to be in or even near the water. Watch from webcams and local news.
#cawx
Natasha retweeted
Dropping an imaginary high speed rail map of the U.S. for no reason whatsoever tonight
Readers added context they thought people might want to know
The Sunrise Movement opposed BAAJA (2026), TFRA (2023), BAESA (2022), & CEQ rewrites (2019-20).
Each of those efforts would've reformed public permitting & NEPA environmental review processes, which some consider the largest roadblocks for high speed rail projects.
apta.com/wp-content/upl…
americanaffairsjournal.org/2026/08/agenci…
influencewatch.org/non-profit/pro…
commondreams.org/news/permittin…
weact.org/resources/lett…
greenpeace.org/usa/environmen…
Natasha retweeted
Throw back to last year when President Barack Obama surprised 79 veterans as they arrived in DC on an Honor Flight. He greeted each with a handshake and a Challenge Coin. Our heroes deserve honor and to be taken care of when they end their service. #DemsFightForVeteransCare
Natasha retweeted
Ovarian cancer is usually not picked up until it is late stage when it is very hard to treat.
A new landmark study shows that a blood EV RNA biomarker test called Tu-Stroma can pick up pre-cancerous and early ovarian cancers with high accuracy!
1/5🧵👇
🤖 Made with AI
Natasha retweeted
MS GOP Sen. Cindy Hyde-Smith announced today:"I will not return the blue slip on the nomination of Scott Colom to serve as a U.S. District Court judge for the Northern District of MS." Colom was 1st nominated to a lifetime appt. as a federal judge by President Biden in Nov. 2022.
Natasha retweeted
We are preparing for a stormy Friday around the Bay Area. Heavy rain (ending our fire season), strong winds 30 to 50 mph and dangerous surf with 20 to 30 foot waves slamming our coast. Get those storms preps in today/Thursday. Check drains are clear, furniture/decor moved inside.
Natasha retweeted
eMudhra's incident report is out! A Python script truncated their CA private keys to one byte?!
bugzilla.mozilla.org/show_bu…
gratitude to everyone who's put on a uniform to defend this country's freedom and principles.
like this gigachad.
WATCH: Tomb of the Unknown Soldier Sentinel Yells at Pete Hegseth’s Group Over Laughter
mediaite.com/media/news/watc…
36
Natasha retweeted
UnitedHealth made over $6 billion in profit in just 3 months. Reportedly denying about 1 in 3 claims.
You paid your premium. They denied your claim.
They kept your money. Then they called it a "record quarter." And it's all legal.
Natasha retweeted
Fascinating organizing tactic by the @scott_colom field campaign to stop knocking doors and start organizing workers who weren't being reached through traditional methods
Kinda reminds me of Bernie with the meatpackers in Iowa
Natasha retweeted
This photo from Election Night in Boone, North Carolina, is a modern Norman Rockwell painting
📸 Credit: Noah Williford, @TheAppalachian































