Service Disruption June 9th

12:32 GMT/ 04:32 PST:

We are currently experiencing performance issues and are investigating it. Thanks for your patience

13:02 GMT/ 05:02 PST:

We have identified the root cause of the performance issue impacting some of our customers. More to follow.

13:43 GMT/ 5:43 PST:

We are still working on resolving the performance issue impacting some of our customers. Apologies for the disruption.

14:34 GMT/ 6:34 PST:

We continue to work hard on the issue impacting some of our customers.

15:21 GMT/ 7:21 PST:

We are still working on the issue and are rolling out fixes. Thanks for your patience. 

16:28 GMT/ 8:28 PST:

We\\\'re finishing the deploy of a fix for the customer impacted by our current issue. We will update you shortly.

17:11 GMT/ 9:11 PST

The performance incident should now be fully resolved. Thank you for your patience. Post-mortem will be posted here.

POST-MORTEM

The root-cause of this issue was found to be a change that was released for optimizing the speed at which our services were running and processing data.  Overtime it was found that this optimization inadvertently caused a slow memory leak that eventually contributed to accounts becoming inaccessible to customers.  This issue was felt by customers across all our data centers as the code change was released across the entire Zendesk eco-system.  

Once this specific issue and cause was identified by our Operations team, our Developers immediately released a patch to fix the memory leak across all our data centers which both fixed the leak but also restarted the services that were affected, essentially cleaning up the latency issues that were existing. This patch allowed all services to return to normal and customer access issues to their accounts was resolved.

After the incident was resolved, it was found that there are areas of monitoring that could be improved to catch these memory leaks at an earlier state hopefully before customers are impacted.  This is currently being implemented.