Service Incident July 12th 2016 Macro issues in Pod 5

21:49 UTC | 14:49 PT
Issues impacting macro application are resolved, if you see issues please try refreshing.

21:35 UTC | 14:35 PT
We are seeing improvements in macro application and will continue to monitor the situation. Please refresh if you are still seeing issues.

POST-MORTEM

In anticipation of a framework upgrade, we deployed a single server on Pod 5 with the new framework for debugging. Soon afterwards some customers began reporting errors (specifically 403s) when applying macros to tickets. Once the issue was reported, we quickly rolled back the test server to put it in parity with the rest of the Pod. A refresh resolved the issues for customers.

Post-incident investigation revealed that when a customer's traffic was split between the two versions of the framework, customers received error screens. To remedy this, we are adding more monitoring to catch these errors before they affect customers.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.