Harry Ashton

It's 9:57 am. Been up all night. Postiz is back up! Been a very hard night, and I feel like I let everybody down. All the comments I got were very encouraging; nobody scolded me. Thank you so much for that! During the day, a major PHP vulnerability was discovered, affecting about half the web. Deployments on @railway stopped, and I assumed it was because of that, even though you do want deployments to upload the latest WordPress fix. Everybody pinged Railway on Discord, and there was no answer- crickets... and people started getting extremely worried, especially because of >> PHP. Then I got an email from Anthropic and Stripe saying my API keys were compromised. Still no answer from Railway, but I didn't take the risk; I rotated them and put Postiz down. We all, even in Discord, already understood there was a leak, but Railway reported it only one hour later. So the goal was to rotate all 140 keys (that's wild), because each key is duplicated across different containers (scattered). Some have the same variable with a different value (like different DB pool connections) I was overwhelmed; it was too much, and I froze. I didn't know what to do. I am lying on my Airbnb bed, but I know I need to get into sitting mode to be more serious, so I go to the kitchen and sit on a chair. I asked claude to vibe code a system that: - Will take all the keys and values from Railway (using their MCP) - Will have an easy way to update all of them or one by one. Based on containers. - Give me categories so I can be more organized. and prioritize. I also started rotating keys from 12 am to 7 am. All while Railway is still down and deployments are paused. Once I finished I used Railway MCP to update all the env variables. if AI didn't exists I don't know how I could have done it. Then, around 8:40am to 9:00am, we started deploying everything back to Railway to minimize downtime as much as possible. And now everything is up. I feel bad because DevRels at Railway sweet-talked me into staying after each incident and promised things wouldn't happen again; after two incidents, I should have migrated. I couldn't find a platform I enjoyed as much as Railway, so I didn't want to switch. Railway problems will happen again, and I will do everything I can to make sure they don't affect me like that anymore. The main problem was the lack of communication and 17 hours of downtime, which isn't realistic. In the next few days, we will come up with a solid plan to migrate from Railway to a different service, trust is gone. Also, Postiz grew from $300k to $303k MRR overnight. If you tried to cancel your subscription and couldn't, please contact [email protected], and they'll issue a full refund (super sorry about that). This week, will be security week for everybody!