Service Incident August 22nd 2016 Asynchronous jobs failing all Pods

22:48 UTC | 15:48 PT
The issue involving asynchronous jobs across Pods has been resolved.

22:42 UTC | 15:42 PT
In addition to recovery, any affected asynchronous jobs should now be able to complete successfully.

22:38 UTC | 15:38 PT
We're seeing recovery across Pods involving asynchronous ticket updates. Please contact us if you're seeing any new issues.

POST-MORTEM

A scheduled maintenance to upgrade production software encountered unforeseen issues. While working to resolve the unforeseen issues during this change, a command was issued which accidentally killed the process that runs DNS on all of the servers. To resolve the incident a query was issued to restart DNS first on Pod 5. When it was confirmed that this resolved the incident on Pod 5, the query was issued for the rest of the Pods. 

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.