Clarkezone retweeted
More details on the root cause can be found in the Microsoft 365 Admin center, and an abridged version can be found below.
We are deeply sorry for the impact to all our customers and recognize the impact that this has on their normal business & operations.
The incident on Thursday, November 12th, 2026, was caused by three main factors that together caused a global outage for the networking gateway that fronts many Microsoft 365 services.
First, a latent defect caused servers to become overwhelmed and unresponsive under certain load conditions. This was due to lock contention inside an internal process which caused the machine to become unavailable. This defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers; neither the underlying defect nor the fix applies to broader Windows deployments.
The initial change in behavior first started deployment in July of this year. The update went through safe deployment and saturated globally over several weeks, during which time no issues were observed. For more than three months, the change was deployed worldwide without impact.
Our normal load testing did not detect the latent issue because the different traffic and SKU mix used in our test labs. The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on our weaker hardware SKUs with fewer cores.
Second, a surge in load increased the likelihood for the networking gateways to encounter the contention. Within Microsoft 365, we maintain two independent and heterogenous network gateway systems designed for isolation and high availability. This allows many Microsoft 365 services to fail to/from them during outages or disruptions. Approximately fifteen hours before the outage, Teams failed over from the alternative networking gateway to the ‘primary’ gateway that also supports Exchange Online. This increased the load on the networking gateway and changed its traffic patterns. This type of failover is part of normal operations and is integral to standard Microsoft resiliency efforts.
Third, at the same time, nearly 14% of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day. This server capacity was on track to be returned to active service through routine maintenance and health processes. The service had sufficient capacity headroom, so the reduction in server capacity had no impact on its own.
Taken together, the increased load and the reduced server capacity, caused the network gateway to encounter the latent bug from July. This caused requests to build up until servers ultimately stopped responding. This occurred during the peak window of traffic in the United States.
When servers lose connectivity, our traffic management systems automatically route users to other healthy endpoints. This happens often and is critical to avoiding any interruption due to hardware, software, or network defects and is normally what makes the service resilient. However, the redirected traffic increased the likelihood for the same problem to occur on the remaining endpoints. This caused a cascading failure from server to server as the same amount of load was spread across fewer servers. In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive.
Normally, the network gateway is composed of thousands of ‘capacity units’ which represent shards of capacity, hardware, and networking infrastructure. These ‘capacity units’ are then organized into regions. In this case, due to a separate issue, the normal bulkheads between these different partitions did not stop the cascading failure. This is what allowed the impact to spread across our network gateway infrastructure globally.
Several services like Teams, OneDrive, SharePoint, and Office apps failed out of the faulty network gateway infrastructure. That is why those services were restored earlier, with Teams chats and meetings functioning without issue for most of the outage. But not all services, like Exchange Online, can fail out of this network gateway.
Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again. Our normally automated recovery processes were not designed to restore service from this broad unresponsive state. Additionally, since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change.
After engineers confirmed the defect as the cause, we developed an update that returned the code path to the previously known good state. To effectively apply this fix, and to slowly bring the service online again, we took groups of servers out of service, applied the change, and restarted them before returning them to service. This process was carefully applied to large quantities of infrastructure at a time (e.g. 50%+ of a region).
The final Post-Incident Review (PIR) will provide more details later this week.
Replying to @MSFT365Status
We have released the Preliminary Post-Incident Report of last week’s outage, which is accessible via the Admin center under MO1550821 and EX1550543.
Clarkezone retweeted
25 original XBOX games are rolling out today on PC, including KOTOR I & II, The Elder Scrolls III: Morrowind, and Blinx: The Time Sweeper. Select titles are also getting Achievements for the first time. Preserving our history means making sure people can still play the timeless games from the past.
Clarkezone retweeted
The largest drone show in Seattle. RIGHT NOW
Celebrating 25 years of XBOX! teapawt.net/i/broadcasts/2MErZNrmj…
Clarkezone retweeted
Back-to-back in my tech feed:
"The FAA announced on November 10 that it has expanded the use of SMART, an AI tool for advising air traffic controllers to manage congestion, across the US"
"FAA orders nationwide ground stop due to ‘extraordinary circumstances’"
Coincidence?
Clarkezone retweeted
Some great pix of the @space_needle lit up in @XBOX green and the largest drone show Seattle has ever seen to help cap off the Xbox 25th Anniversary FanFest event at @mopop
What a night!
Microsoft celebrating 25 years of XBOX tonight with #SpaceNeedle lit up in green and an amazing drone show! Not to mention a whole day without Outlook, the best celebration of all 😉 #Seattle #xbox25
Community meeting this Monday at 10 AM PT! Join us as Maxim presents: "Can the compiler prove it?", his status report from a research branch on contracts, inferred loop invariants, and an SMT solver for Mojo.
Come see what it could mean for the compiler to prove your code correct, ask Maxim your questions, and chat with other community members: luma.com/nov-modular
Clarkezone retweeted
For those who haven't been tracking it, Swift System provides type-safe, idiomatic Swift interfaces to native system calls on Windows, Linux, and macOS. It provides useful foundations for higher-level cross-platform libraries to build on. @ivatokar has the latest...
Swift System 1.9.0 is out, and this release is particularly relevant to low-level and server-side Swift.
It adds StatFS support for retrieving filesystem information and pollAdd support for Linux IORing. The release also makes Errno descriptions thread-safe and includes improvements to buffer handling.
These aren't headline-grabbing language features, but they matter when Swift code needs to work close to the operating system. Filesystem introspection, asynchronous I/O, and reliable error reporting are building blocks for infrastructure software.
For developers using Swift beyond UI frameworks, Swift System is worth watching because it helps expose system functionality through Swift-friendly APIs. The release requires Swift 6.2 or newer.
#SwiftLang #SwiftOnServer
forums.swift.org/t/swift-sys…
Clarkezone retweeted
Happy .NET 11 day! #dotnetconf starting in one hour… dotnetconf.net/
Replying to @SergioPedri
@SergioPedri is about to be live talking about CSWinRT3 which is a massive ground up rewrite he has lead that makes leaps forward in performance and efficiency of C# WinRT interop 🍿👀 #dotnetconf #dotnet11
.NET Conf 2026 - Day 2 is getting ready to kick off!
Join in live, ask questions, and get ready for more .NET 11 and Visual Studio announcements!
msft.it/6014ahvrQ
#dotnetconf #dotnet11
Clarkezone retweeted
Windows App Development CLI v0.8: WinUI DevTools Experimental Preview devblogs.microsoft.com/ifdef…
Clarkezone retweeted
virtualization only gets you part of the way. Each item also has to be cheap to render. You need persistent text state, increemntal updates, and scroll anchoring.
I've spent a good amount of time on this. here's a skill/prompt that should get you most of the way there @ClarkeZone, @kazzkiq, @arielsp11:
### Objective
Improve scrolling in this GPUI app without changing its behavior. Check the exact framework and component-library versions, trace the current rendering path, and profile a release build. Make the smallest changes the measurements support.
Preserve selection, scroll position, pagination, streaming, and normal input responsiveness, and keep anything that already works well. **Improve the implementation; don't add features or build a new rendering framework.**
### Areas to investigate
Address each where the evidence warrants it.
1. **Virtualization**
- Use variable-height virtualization for mixed messages, images, and diagrams.
- Render the visible area plus a modest buffer.
- Don't measure the whole history upfront or reset the list on every update.
2. **Stable identity and state**
- Key items by persistent identity, not index.
- Keep prepared text and other reusable state when items are inserted, updated, or moved.
- Make sure interactive elements have the identity GPUI needs to update hover and other interaction states immediately.
3. **Expensive preparation**
- Reuse parsed Markdown, syntax highlighting, diagram results, and decoded images.
- Move expensive preparation off the UI thread where supported.
- Key each cache on the inputs that actually determine its result. For example, a width change should invalidate affected text layout without reparsing unchanged Markdown.
4. **Update scope**
- Trace what scrolling, hovering, streaming text, and spinner ticks invalidate.
- Reuse unchanged content. A small animation shouldn't repeatedly rebuild expensive message bodies.
- Extracting a function or component doesn't by itself create that boundary. Check the installed GPUI version's caching and invalidation behavior before choosing one.
- Keep content-based measurement for variable-height rows.
5. **Changing heights**
- Reserve image and diagram space when dimensions are known.
- Remeasure affected items when content, width, or fonts change.
- When older content loads or content above changes, keep the reader's position, anchored to a stable item and offset where appropriate.
- Keep the existing rules for following new messages vs. reading older ones.
6. **Async loading**
- Fetch more history before the reader reaches the edge.
- Keep existing content scrollable while loading.
- No blocking network requests, disk reads, or expensive whole-history scans, copies, or sorts during rendering.
7. **Large items and memory**
- Virtualizing messages doesn't virtualize the contents of one huge message. Profile long code blocks, tables, and diagrams separately, and add finer virtualization only where measurements justify it.
- Check whether retained images and prepared content keep growing as more history is traversed.
8. **Measurement**
- Capture frame-time spikes and missed presentations alongside CPU, layout, and paint costs.
- Report the distribution and worst stalls, not just average FPS.
- At 120Hz there's about 8.3ms between refreshes, and that includes work outside the app's CPU rendering.
### Testing
Test in the **actual native app** with computer-use tools. Browser testing can help compare against a web version but doesn't verify GPUI rendering.
Take a baseline before changing code, then repeat the same interactions afterward with the same content, window size, and build configuration.
1. **Realistic history:** Use a long conversation with short messages, long Markdown, code blocks, and images, plus diagrams and charts if the app supports them. Confirm the test hits real pagination and async updates.
2. **Scrolling:** Scroll up through several pagination boundaries and back down. Try slow and fast scrolling, direction changes, scrollbar dragging, and trackpad momentum where available. Watch for blank areas, overlapping content, jumps, flicker, and delayed input.
3. **Updates while reading:** While in older history, let messages arrive and text stream. What you're reading should stay in place. Separately, check following updates at the bottom and deliberately returning to the latest message.
4. **Layout changes:** Resize the window while reading mid-conversation, and let images and other async content finish loading. Rows should stay correctly measured and the reading position sensible.
5. **Normal interaction:** Hover messages, select and copy text, type in the composer, and switch conversations and back, checking that reading position is restored as expected. Repeat relevant checks with a spinner running.
6. **Animation vs. content:** With unchanged content and a spinner running, measure whether expensive message rendering keeps running unnecessarily. Then test animation alongside real streaming.
7. **Cold and warm:** Test a first visit and a return visit. Check memory after traversing a lot of history and switching conversations. Explain which results depend on warm caches.
8. **Evidence:** Record hardware, OS, display refresh rate, framework versions, and release-build configuration. Use a profiler to find where time goes. Distinguish CPU render-to-paint timings from actual frame presentation.
### Verification and reporting
Mark each live check **PASS, FAIL, or NOT TESTED** with brief evidence. Distinguish manual from scripted input. If the tools can't reproduce trackpad momentum or measure real presentation timing, say so.
**Compilation, screenshots, and CPU render timings alone don't prove smooth scrolling or sustained 120fps.**
Keep verification focused: run relevant existing checks, use disposable instrumentation where helpful, and don't add a large test suite or permanent profiling machinery.
### Deliverables
1. **Measured bottlenecks:** relevant files, functions, and supporting measurements.
2. **Changes made:** what changed and how it fixes those bottlenecks.
3. **Before and after:** comparable measurements and live-check results.
4. **Remaining limitations:** unverified behavior, known limitations, and any temporary instrumentation removed.
If a suspected path turns out to be cheap already, leave it alone and report the evidence.
Clarkezone retweeted
The new Discord client using less RAM is just part of the story. Clicking around feels instant, I can't go back. Check out the boot time.
We've been early adopters of Rust since 2018 and a lot of our infra, voice tech, and even some of the current client is already Rust, the holy grail was getting the whole client there. That means Electron goes away, and so does React Native, mobile keeps native UI but shares app code with desktop via Rust. We hope to get it into testing with users soon™. (And it's not Tauri, it's GPUI, so good guess everyone.)
Alright, back to staring at the WoW Forever login screen.
DISCORD IS FINALLY DITCHING ELECTRON!!!!
We've been early adopters of Rust since 2018 and a lot of our infra, voice tech, and even some of the current client is already Rust, the holy grail was getting the whole client there. That means Electron goes away, and so does React Native, mobile keeps native UI but shares app code with desktop via Rust. We hope to get it into testing with users soon™. (And it's not Tauri, it's GPUI, so good guess everyone.)
Alright, back to staring at the WoW Forever login screen.
Omarchy has replaced Ubuntu as my distribution of choice for my Linux box. So good
Omarchy 4.5 is finally here! Traditional desktop mode, M1/M2/M3/Neo compatibility, 50 languages, 7-second installs, and a million other features, fixes, and improvements. omarchy.org/news/2026/11/oma…
1
3
258
Clarkezone retweeted
Ever wish you could practice a big talk in front of a real crowd before giving it?
Crowd Room on Apple Vision Pro lets you, and it coaches you on pace, filler words, and eye contact as you speak.
Clarkezone retweeted
Announcing XAML IntelliSense Support in the WinApp VS Code Extension: devblogs.microsoft.com/ifdef… #winui
































