Rob Adams 😷 🦋

Update on yesterday’s FAA ground stop. We now know what happened, and it’s worse than “old equipment.” It wasn’t a cyberattack and it wasn’t SMART. The FAA says there’s zero evidence of either. Here’s what happened. L3Harris crews were replacing an end-of-life copper line with fiber at one remote FAA site. A router involved in that work started flooding the network with more than 1,000 connection attempts every four to five seconds. Radar feeds and radio links to aircraft became unstable. After about 45 minutes of nobody knowing why, the FAA grounded the entire country. Once they isolated that one remote site, everything came back. So let me get this straight. One router at one site took down air traffic control for all 50 states. The FAA has now disabled the faulty function on roughly 60 routers of the same type nationwide. It still doesn’t know whether the router was defective or whether it was installed or configured wrong. Oh, and this has happened before. In 2009, a Harris technician misconfigured a router while moving the same network to fiber and disrupted 129 air traffic facilities. Same contractor, same migration, same failure mode, 17 years apart. This is exactly the point from my last post. Legacy modernization doesn’t fail because of the shiny new system. It fails at the seams, in the routine change nobody thought could take down the whole country. Good change management means one site can’t flood the network, isolation happens in minutes instead of hours, and nobody runs a copper-to-fiber cutover on critical infrastructure without a tested rollback and a hard limit on how far a failure can spread. None of that is new. We learned it in 2009. The question is why we had to learn it again. Tech is so fun...