Helge Hecht

Let's say you commit a change that makes an application 0.05% slower. No big deal, right? Well, at the scale of Meta, it is a big deal--a small slowdown for a large application can waste thousands of servers. It's such a big deal that Meta needs a way to catch these performance regressions and reliably figure out if an application is even a hundredth of a percent slower. The big problem with catching tiny performance regressions is that, well, they're tiny. It's easy for them to be lost in the noise. Any simple approach is going to return 99% false positives because most slowdowns are caused by transitory hardware issues, not actual code changes. Meta's solution to this problem has three high-level steps: 1. Look for performance regressions not in entire applications, but in individual subroutines. Applications have thousands of subroutines, so a small performance regression for an entire app is likely a large one for the affected subroutine. This way, you're looking for ~5% regressions, not ~0.05% regressions, which makes filtering out noise easier. 2. To make tracing at the coroutine level practical, do stack-trace sampling. Regularly collect stack traces of applications and use them to figure out in which subroutines applications spend their time. This is like conventional performance profiling, but done at massive scale. 3. When a regression is detected, do a root cause analysis to figure out if it's caused by transient issues or code changes. If it's caused by code changes, by analyzing the stack traces of the performance regression and comparing them to recent commits, it should be possible to automatically identify which changes caused the regression! What's the big takeaway? Performance really matters at scale, and small performance issues add up. The paper claims that this system saves millions of servers a year by detecting tiny performance regressions, which is a staggering number. If your code is used by billions of people, keep it optimized!

What happens if your CPU gets something wrong? If it wakes up one day and decides 2+2=5? Well, most of us will never have to worry about that. But if you work at a company the size of Google, you do, which is why this paper on "mercurial cores" is so fascinating. What the authors report--and supposedly this is common knowledge at the hyperscalers--is that a couple cores per several thousand machines are "mercurial." Due to subtle manufacturing defects or old age, they give wrong answers for certain instructions. These can cause all sorts of impossible-to-diagnose issues. Some rare problems at Google that were traced back to bad CPUs include: - Mutexes not working, causing application crashes - Silent data corruption - Garbage collectors targeting live memory, causing application crashes - Kernel state corruption causing kernel panics What makes CPUs go bad? It's very hard to tell. The authors posit that issues are becoming more frequent as CPUs get more complex, but there aren't solid numbers behind that. There are certainly strong relationships between frequency, temperature, voltage, and bad CPU behavior--most mercurial CPUs only cause problems under very specific conditions, but those conditions vary from CPU to CPU. Age is another source of problems, as older CPUs are more likely to exhibit problems. Bad CPUs are an especially serious problem because they're very hard to detect. If cosmic rays flip bits in storage or on the network, that can be detected through error coding. But there's no analogy for a CPU that allows cheap online verification of its correctness. Instead, the best detection techniques involve monitoring for symptoms. If a core exhibits exceptionally high rates of process crashes or kernel panics relative to its fellows, that's a strong indication something is wrong with it. For the most critical applications, the authors propose triple modular redundancy--redoing each of its computations on three cores and majority-voting a reliable result. More than anything, this paper is a call to action--letting everyone know that CPUs can fail. So now, if you ever find a bug you can't diagnose, you can blame the CPU! 🙂