The last few systems have been patched after significant delays from CloudLinux. But throughout the day from early morning, multiple possibility mitigations have been put in place. While not 100% effective, it limits exposure. Happy to report no customers faced issues.
We had roughly 5 minutes of downtime on nlsh06 due to a webserver crash that needed manual intervention. We are investigating the root cause.
We had a short alarm trigger on nlcp01 due to a connection flood which enabled our protection that served a Splashscreen / Proof-of-work to all visitors (including monitoring) until the connection count dropped. Sites continued to work during the 3.5 minutes alert.
We will do a quick maintenance of the hosting-panel.net control panel which requires we take it offline. We expect 1-2 minutes.
All affected servers in EU rebooted, downtime lasted anywhere between 2.5 minutes and 7.5 minutes. nash02 (North America) will be rebooted tomorrow morning EU time.
We will perform emergency patching later today in the evening of servers:
nlsh04,nlsh05,nlsh06,nlsh07,nlcp01,nlcp02,nlcp03
nash02 will be patched now.
Downtime per server is expected to be a couple of minutes, but can be slightly longer.
perfgrid.com will be unavailable for a bit, we're fixing some network issues. It does not have any direct customer impact. Support can be reached at [email protected] as usual, and billing.perfgrid.com works for any orders and panel access.
We're aware of the unavailability of our website perfgrid.com - this is due to a hypervisor issue which is being worked on as we speak. No customer sites/services are affected.