Service Disruption September 28th 2015

SUMMARY

As of the 28th of September 23:00 GMT / 15:00 PST, we are working to resolve the following service incident:

23:00 GMT / 15:00 PDT

We are currently investigating reports of issues accessing Zendesk for some customers. More info to follow as we dig deeper.

23:18 GMT / 15:18 PDT

We're continuing to investigate reports of issues with accessing Zendesk and latency for some customers. Apologies for the inconvenience.

23:45 GMT / 14:45 PDT

Investigation continues for access and latency issues, related to our North American data centers. 

00:15 GMT / 16:18 PDT

This issue has resolved. Impacted customers are reporting return to normal performance and accessibility.  Root cause is not yet determined, early indications suggest this may have been a network problem.  Summary of post-mortem investigation will be posted here.

POST-MORTEM

From 20:00UTC to 23:00UTC, on September 28th 2015, a major internet provider suffered an outage as a result of a misconfiguration. This outage affected two of our US-based datacenters and customers attempting to reach accounts hosted in these centers experienced slowness while connecting.

We actively test and check the health of our internet circuits and availability. We also rely on external monitors to provide path monitoring and upstream provider monitoring. However, the current tests in place failed to show us an issue with the internet provider to a test instance. Further, a notably large download that occurred at 13:00PDT at both data centers was thought to be the issue and time was spent troubleshooting that data transfer. Increasing the instances of tests and testing more paths/providers would increase the chance of detecting the issue sooner. Further, monitoring BGP changes at our edge using an external BGP route server may aid us in detecting path changes upstream.

This outage self recovered but the internet provider was removed from BGP peering at both data centers in the event of future issues/outages.

Please subscribe to this article for regular updates until the issue is resolved.  If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues.   We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to support@zendesk.com.