Service Incident February 24th 2016

22:11 UTC | 14:11 PT
The performance issues with Voice and Help Center for some US data center customers have stabilized.

21:04 UTC | 13:04 PT
Mitigation efforts are underway for performance issues with Voice and Help Center impacting some US data center customers.

20:24 UTC | 12:24 PT
We are continuing to investigate Voice and Help Center performance issues for some of our US customers.

POST-MORTEM

Our Voice engineering team saw a spike in dropped calls reported through internal monitoring. Later, we discovered that Help Center was also being affected. Our investigation has shown that an API latency threshold was hit, to which both Voice and HC are sensitive. This also coincided with spikes in the proxy load and may be linked back to an increase in Voice throughput. Ultimately, we concluded that the issues we experienced were due to a lack of capacity in the proxies.

Since the incident, we've made some changes to how Voice traffic is routed. It will now bypass proxies, as well as fail over should errors occur. Since rolling out these changes, we've seen less errors hitting Voice.  

In addition, we continue our capacity expansion initiatives to fully prevent latency-related errors going forward. 

FOR MORE INFORMATION

Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to support@zendesk.com.