Service Incident March 3rd 2016

22:03 UTC | 14:03 PT
The user update issue should now be fixed. Please refresh your browser.

21:57 UTC | 13:57 PT
A fix is being deployed for the issue that is preventing user updates.

21:41 UTC | 13:41 PT
Work continues to investigate issues with updating user data via the interface. Updating via API still works.

POST-MORTEM

This incident was caused by a bug in our console code; the deploy containing this code was originally marked as failed, even though it pushed out new assets to some users. It was then deployed again in the deploy that started this issue. For the duration of the incident, any changes that users made in the agent interface were not being saved. 

Ideally, this scale of breakage would have been identified in browser tests. This particular incident would not have occurred if there was more manual testing done during development, despite any gaps in automation testing. 

Upon identifying the bug, we rolled back to an earlier version of the agent interface. Between the bad deploy and the first attempt at a rollback, a misconfiguration to proxy hosts had occurred, delaying things slightly.

As a result of this incident, we've expanded our browser test coverage to include user profile pages in Zendesk; covering agent/admin roles, as well as changing and saving user and customer data. This will allow us to catch a bug such as this in testing going forward. 

FOR MORE INFORMATION

Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to support@zendesk.com.