Service Incident March 1st 2016

08:12 UTC | 00:12 PT
We’re happy to report that performance related issues for some customers are now resolved. Postmortem to follow:

07:25 UTC | 23:25 PT
We are continuing to work on the issues with attachments and csv exports. Next update in 30 minutes...

06:43 UTC | 22:43 PT
Work is still underway to resolve issues with attachments and csv exports. We will provide another update soon..

05:43 UTC | 21:43 PT
We have identified the root cause and are working on the solution. More to come...

05:20 UTC | 21:20 PT
We continue to investigate issues with attachments, more information to follow shortly.

POST-MORTEM

This incident impacted several features in Zendesk related to attachments in both Zendesk agent interface and Help Center, as well as email attachment uploads, ticket merging. Some issues with exports were also observed.  

The failures experienced were brought about by a configuration update, which contained a bug that was not detected in testing. A new version of the configs was applied but, due to the bug, was missing crucial information that specified bucket tags for attachments in each data center. Changes to refactored code had to be made on the fly, which fixed some issues in Zendesk but caused exceptions in Help Center. Once the configurations were updated correctly, Help Center issues were resolved.

We're working to improve our testing to detect code bugs such as this, as well as implementing automatic tagging so that the code does not need to specify attachment buckets going forward. 

FOR MORE INFORMATION

Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to support@zendesk.com.