19:28 UTC | 12:28 PT
Performance impacting issues affecting some customers access to their accounts has now been resolved. Postmortem to follow: zdsk.co/1SRSPmt
Customers were unable to access tickets and were presented with green screens (500 level errors). The issue was caused by a deployment that went out earlier in the day. Once the issues caused monitoring alerts to be fired, the deployment was rolled back; however, the bad data still persisted until we cleared the cache on the servers. Once the cache was cleared, services returned to normal.
To help mitigate this kind of issue, we are investigating more comprehensive rollback mechanisms, as well as creating better internal documentation on appropriate deploy processes.
FOR MORE INFORMATION
Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to firstname.lastname@example.org.