Service Incident on 14th May 2015

Summary

At around 8:00pm PDT on Thursday, May 14th,we are working to resolve the following service incident:

We started receiving reports of performance issues for accounts located on our West Coast data centre. Our team are actively investigating.

Timeline:

8:03PM PDT: \\\"We’re currently investigating reports of accounts not loading. More information to follow shortly.\\\"

8:39PM PDT: \\\"We are still working to resolve the service incident involving access to accounts on our West Coast Data center.\\\" 

8:51PM PDT: \\\"We are working to mitigate the issues affecting customers in our West Coast Data center. Thanks for your patience.\\\"

9:13PM PDT: \\\"We\\\'re continuing to work towards mitigating the issue affecting our West Coast Data center.\\\"

9:30PM PDT: \\\"We have mitigated the issues affecting customers in our West Coast Data Center. Post Mortem to follow\\\"

Investigation

The nature of this incident was a network abuse attack, or DDoS (Distributed Denial of Service) attack.

The activity began at 7:41 PM PDT (2:41 AM UTC) and the impact lasted until 9:20 PM PDT (4:20 AM UTC).

The Zendesk Operations, Network, and Security teams worked to identify the type of attack and deploy appropriate mitigating capability to bring services back online.

During the initial portion of this incident it was not clear that the source of the problem was an attack.  Instead, we were investigating what appeared to be significant performance issues with one of our West Coast Data Center routers. 

Approximately 35 minutes into the event, our Network Operations and Security Operations engineers identified the DDoS nature of the incident and immediately began to implement our DDoS mitigation capabilities. This brought service back online briefly before a second type of attack started against another resource in the network. Shortly thereafter another mitigation was put in place.

The decision was made to route traffic through our 3rd party DDoS mitigation service. In addition, we were able to re-route traffic to an alternative server. The issues were immediately remediated. We are consistently working to improve our DDoS remediation measures, including internal procedures and 3rd party services.