Service Incident on November 16th 2015

SUMMARY

As of 16:24 UTC/08:24 PST, we are working to resolve the following service incident:

We're currently investigating reports of latency and problems accessing Zendesk. Apologies for the inconvenience.

16:36 UTC/08:36 PST

The service incident affecting our customers has been resolved. Sorry for the disruption!

POST-MORTEM

This short outage was caused by a failure during testing of updates to our data centres infrastructure. Incorrect network configuration in the new procedure had caused public routes to our EU data centre to be disabled, leading to Zendesk becoming inaccessible to al customers whose accounts are located in the EU data centre for approximately 16 minutes. To mitigate the incident, we rolled back all changes performed during the testing, after which customers were able to get past our edge routers to access their accounts.

To prevent similar incidents going forward, we have reviewed our network testing protocols and well as worked to implement network monitoring improvements to improve detection times, help isolate issues, and reduce downtime.