Today we experienced a server outage caused by a Denial Of Service (DOS) against one of our customers. In the spirit of openness we want to explain what happened.
Around 10:40 am EST the attack started. It's what's called a SYN flood. Basically the attackers opened a connection to the firewall but did not do what normal clients do and complete their part of the connection. This caused a much higher than normal number of concurrent connections to be opened. Our firewall can normally handle over a hundred thousand connections per second, which is normally more than enough. The attack consumed all of the firewall resources, consequentially blocking legitimate traffic.
We immediately started working with Rackpace's technicians to resolve the issue. Due to the overload on the firewall they were unable to connect to the firewall using normal techniques. Datacenter technicians had to enable a different method of access. Once technicians got access they were able to determine the cause of the issue and put measures in place to prevent it.
Around 11:40am EST the attack was successfully being blocked. We've been monitoring it since then and have seen no further issues.
We sincerely apologize for the downtime and inconvenience that it caused. We are constantly working on our infrastructure and monitoring it so that if something like this happens we are able to quickly remedy it.
There was no loss of data or any security breaches made. Only the firewall was targeted in this attack.
Thank you for your patience and your trust in our services.
If you have any questions or concenrs please do not hesitate to contact us.
