Service Disruption October 29th 2015

SUMMARY

As of October 29th, 15:00 GMT / 8AM PST we are working to resolve the following service incident:

We are receiving reports of performance issues. We are currently investigating this.

15:30 GMT / 8:30 AM PST:

We are seeing some improvement to the current issue impacting performance. More information to follow.

15:48 GMT / 8:48 AM PST:

We are happy to report that the issue impacting performance is now fully resolved. Post-mortem to follow below.

POST-MORTEM

This incident was a result of a DDoS (Distributed Denial of Service) attach against one of the Zendesk IPs, causing accessibility issues to one of our data centres. Actual DDoS attack flood time was between 14:37 - 14:40 UTC. Once the attack subsided, due to the time of day and volume of customers online, there was a reconnection volume which seemed to overwhelm the edge firewalls and caused routing protocol issues, which caused a second outage. 

Due to the short duration of the attack, our DDoS detection tools did not trigger actionable alerts. These thresholds have been adjusted, accordingly.

DDoS attack subsided on its own, network convergence and routing protocols instability settled once flood of legitimate reconnection activity normalized.