Wild Horizon (@DGJFarrell) header
Wild Horizon (@DGJFarrell) profile picture
Joined June 2012
Wild Horizon retweeted
More details on the root cause can be found in the Microsoft 365 Admin center, and an abridged version can be found below. We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations. The incident on Thursday, November 12th, 2026, was caused by three main factors that together caused a global outage for the networking gateway that fronts many Microsoft 365 services. First, a latent defect caused servers to become overwhelmed and unresponsive under certain load conditions. This was due to lock contention inside an internal process which caused the machine to become unavailable. This defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers; neither the underlying defect nor the fix applies to broader Windows deployments. The initial change in behavior first started deployment in July of this year. The update went through safe deployment and saturated globally over several weeks, during which time no issues were observed. For more than three months, the change was deployed worldwide without impact. Our normal load testing did not detect the latent issue because the different traffic and SKU mix used in our test labs. The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on our weaker hardware SKUs with fewer cores. Second, a surge in load increased the likelihood for the networking gateways to encounter the contention. Within Microsoft 365, we maintain two independent and heterogenous network gateway systems designed for isolation and high availability. This allows many Microsoft 365 services to fail to/from them during outages or disruptions. Approximately fifteen hours before the outage, Teams failed over from the alternative networking gateway to the ‘primary’ gateway that also supports Exchange Online. This increased the load on the networking gateway and changed its traffic patterns. This type of failover is part of normal operations and is integral to standard Microsoft resiliency efforts. Third, at the same time, nearly 14% of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day. This server capacity was on track to be returned to active service through routine maintenance and health processes. The service had sufficient capacity headroom, so the reduction in server capacity had no impact on its own. Taken together, the increased load and the reduced server capacity, caused the network gateway to encounter the latent bug from July. This caused requests to build up until servers ultimately stopped responding. This occurred during the peak window of traffic in the United States. When servers lose connectivity, our traffic management systems automatically route users to other healthy endpoints. This happens often and is critical to avoiding any interruption due to hardware, software, or network defects and is normally what makes the service resilient. However, the redirected traffic increased the likelihood for the same problem to occur on the remaining endpoints. This caused a cascading failure from server to server as the same amount of load was spread across fewer servers. In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive. Normally, the network gateway is composed of thousands of ‘capacity units’ which represent shards of capacity, hardware, and networking infrastructure. These ‘capacity units’ are then organized into regions. In this case, due to a separate issue, the normal bulkheads between these different partitions did not stop the cascading failure. This is what allowed the impact to spread across our network gateway infrastructure globally. Several services like Teams, OneDrive, SharePoint, and Office apps failed out of the faulty network gateway infrastructure. That is why those services were restored earlier, with Teams chats and meetings functioning without issue for most of the outage. But not all services, like Exchange Online, can fail out of this network gateway. Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again. Our normally automated recovery processes were not designed to restore service from this broad unresponsive state. Additionally, since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change. After engineers confirmed the defect as the cause, we developed an update that returned the code path to the previously known good state. To effectively apply this fix, and to slowly bring the service online again, we took groups of servers out of service, applied the change, and restarted them before returning them to service. This process was carefully applied to large quantities of infrastructure at a time (e.g. 50%+ of a region). The final Post-Incident Review (PIR) will provide more details later this week.
Replying to @MSFT365Status
We have released the Preliminary Post-Incident Report of last week’s outage, which is accessible via the Admin center under MO1550821 and EX1550543.
61
159
37
1,089
153,915
‼️ UPDATE: The router behind Friday's nationwide US ground stop was a brand-new Juniper unit at a remote FAA radio site in Arkansas. US defence contractor L3Harris had just installed it. FAA officials believe a feature that broadcast data more widely than needed flooded the network. The FAA has switched that feature off on every router of that type. HPE, which owns Juniper, says the routers "did not fail."
264
422
140
3,021
252,835
Go big or go home.
Starship V4 will be ~70m long, which is roughly a 40% length increase and allows >60% more propellant load. Engine count will be raised to 9, with 3 more Raptor-Vacuum engines added, and these will be higher thrust V4 engines.
5
Why is @MarkJCarney letting her continously embarrass herself and Canada??
😬 With every interview on the Stellantis fiasco, Melanie Joly manages to find a new way to reveal her incompetency. 😡 She's a blunt instrument, a surgeon with a chainsaw. "I'm in charge! The PM is in charge!" 😳 She's confused, defensive & at times, pathetic. 😐 Even for her critics, this isn't fun anymore. 🙏 Carney, put her out of our misery. Show her mercy. Let her go. For the sake of our workers. Thx @cbcwatcher
1
7
Because it didn't happen Nov 13th. Lol.
Why are people predicting a massive blackout on November 17th @grok
21
Awesome
BREAKING: Samsung reportedly plans to make chips for SpaceX’s Starlink satellites in Austin, Texas. Samsung is investing $1.9 billion to expand its Austin chip factory, with the planned operations targeted to begin by 2030, according to Seoul Economic Daily. The chip, called STM32V8, is already used in Starlink’s mini laser system. SpaceX’s Starlink engineering team has praised the chip’s performance. The chips are currently manufactured in France. The reported Texas plan would add American production capacity for a key component behind Starlink’s satellite communications.
12
I am Ukrainian. In '22 I naturally wanted my country to win. In '22 I was a patriot. In '22, I volunteered to the AFU. Today, I couldn't care less. Ukraine is a shithole of bottomless corruption. The ideology is a death cult. I am not pro-Russian. I am pro-sanity. I do not care.
1,242
6,270
298
49,066
1,930,487
Wild Horizon retweeted
🚨 OMG. Treasury Sec. Scott Bessent just dropped the NAME of the WSJ editor who doxxed his house online, letting Iran geo-locate him, forcing him to move Q: Are there any direct Iranian attempts on your life? BESSENT: "We're not gonna talk about that...EMMA TUCKER at the WSJ AUTHORIZED a PHOTO of our HOUSE the DAY WE BOUGHT IT that had a geo-locate on it. You can go on the internet and see our home in DC because of RECKLESS JOURNALISM!" Well, not anymore — HE HAD TO MOVE The Journal ran an aerial view of his $12.5 MILLION Georgetown home back in January 2025, right after he bought it Now, Bessent is protected on a military base. This should be freaking illegal. Genuinely infuriating. Pray for Bessent and all of Trump's Cabinet 🙏🏻
656
5,713
196
22,764
578,479
The most stupid MP in Canada. So out of her league.
Rosie: Why would you now both threaten to sue Stellantis for its breach of contract and then say, but we're willing to help you if you're willing to stay in Canada? Joly: Because if they don't close the plant, then we're bringing back the jobs for a thousand people.
15
Wild Horizon retweeted
🚨 BREAKING: Trump U.S. Attorney Aaron Reitz just formally put Sen. Elizabeth Warren on LEGAL NOTICE, ordering her to preserve ALL records tied to her plans to investigate people close to President Trump FAFO! The letter cites 18 U.S.C. §§ 241 and 242, the federal laws against CONSPIRING to deprive Americans of their constitutional rights Reitz: "Investigations conducted solely for the personal aggrandizement of the investigators or to 'punish' those investigated are indefensible." "Warren is on the warpath to wield her committee gavel like a tomahawk, seeking the scalps of those who associate “too closely” with President Trump. Congressional investigations can’t be used to punish Americans for exercising their 1A rights." It covers Warren, her staff, consultants and even CAMPAIGN PERSONNEL She has until 6 PM on November 23 to confirm IN WRITING. Warren wanted to play prosecutor with no crimes committed, now SHE'S in hot water 😅
1,024
5,883
250
23,470
1,113,051
Awesome progress.
Replying to @DJSnM
Yes, the green flame is caused by burning copper from depleting fuel in the regenerative cooling circuit and going oxygen-rich (closer to stoichiometric temperature) in the main combustion chamber. Slight underperformance of engines on this flight is the reason. The booster is meant to land with tiny residuals, so the solution is to improve engine performance and performance consistency (underway with Raptor V3.2), as well as adjust flight software to do MECO1 about one second earlier to ensure enough propellant for boostback and landing (tiny change). These early ships and boosters are already outdated by improved designs in production or on the test stand, so this booster loss does not change our timeline to reach high volume, fully reusable Starship flights.
11
Wild Horizon retweeted
🚨 BREAKING: SHOCKING CLAIMS EMERGE FROM A CLOSED-DOOR DEMOCRATIC GOVERNORS’ MEETING OVER ELON MUSK AND SPACEX! Reported by multiple sources, Gina Hinojosa reportedly tells fellow Democratic governors that she is “not thrilled or overly excited” about Elon Musk’s latest SpaceX achievement. The report remarks go further: “Yes, what Elon Musk is doing is commendable. Nobody can deny the technological achievement. But that does not mean he gets to operate in Texas without answering questions about taxes, corporate obligations and what his companies owe the communities hosting them. If SpaceX wants to make Texas its home, then Texas must come first. There could be serious consequences if these obligations are not addressed.” But the most controversial part of this IN DOOR democratic meeting? Other Democratic governors reportedly back her position, raising questions about whether political opposition to Musk is becoming bigger than the policy disagreements themselves. Think about the timing. SpaceX is making extraordinary engineering progress, attracting international attention and putting American space exploration at the centre of global conversation. Yet the political conversation keeps returning to how Musk can be restricted, investigated or forced into submission. Nobody should be above the law, including Elon Musk. He should pay every tax legally owed. But neither should a successful American entrepreneur be treated as a political enemy simply because some politicians dislike him. The real question is whether America’s leaders want to compete with the rest of the world or spend their time fighting the people building its future. ONE FINAL QUESTION: IF ELON MUSK WERE BUILDING ALL THIS IN ANOTHER COUNTRY, WOULD THESE SAME POLITICIANS BE CELEBRATING HIM INSTEAD? 🇺🇸🚀
1,486
855
146
6,231
847,745
Wild Horizon retweeted
Inspecting ship 42 right now
5,868
16,558
1,197
173,753
7,124,352
Wild Horizon retweeted
Some phone videos I took from the tower
12,795
28,191
2,307
279,221
8,110,000
Wild Horizon retweeted
The next-generation 2027 Chevy Captiva is preparing to expand into additional international markets, bringing the updated crossover to more buyers. Here's where Chevrolet plans to introduce the new model as its global rollout continues. gmauthority.com/blog/2026/11… #chevy #chevycaptiva
1
4
355
Wild Horizon retweeted
Congratulations to an incredible team at SpaceX!!!
6,964
17,466
2,113
166,798
7,349,497
During your first town hall or information session held at the Recreation Center in St. Lawrence, you didn't leave the impression you were keen on Elon.
Life on Earth will never be the same again. This is my generation’s “Apollo Moment” if there ever was one. Thank you for leading the way for us SpaceX and @elonmusk. Boundless respect and congratulations from your fans and customers here at NordSpace in Canada!
1
3
734
RT @RCMPNL: 📢TRAFFIC ADVISORY (November 14, at 4:30 p.m.): Clarenville RCMP advises that the Trans-Canada Highway is closed in both directi…