Service Incident Email Ticket Creation Delays May 18 2015


As of May 18, 2015 8:22am PDT, we are working to resolve the following service incident:

We are currently investigating reports of ticket creation delays for some customers. We determined that this was two separate issues - one regarding ticket creation via Twitter and Facebook, covered here, and this one, regarding ticket creation from emails, affecting a different set of customers.

11:27am PDT: We separated the two issues.

11:39am PDT: Ticket creation delays in our Facebook and Twitter channels should be resolved. We\\\'re still investigating a separate email delay issue.

12:12pm PDT: We are currently working to resolve the email ticket created delay affecting some customers. Full recovery may take a few hours.

4:14pm PDT: The ticket processing queues are getting down, but we still see some email delays.

5:00pm PDT: Delayed emails continue to clear, we\\\'re actively monitoring. We will update the status here again in 2 hours.

7:30pm PDT: We\\\'re continuing to monitor our backlog and can confirm that the email delay is continuing to clear. Thank you for your continued patience.

8:30pm PDT: The email backlog has been cleared and email is processing normally. Post-mortem to follow.


The root-cause of this issue was found to be an issue with bulk updates feature via our API.  

Once the issue was identified, our Engineers  released a patch to help alleviate strain on our API services during high traffic periods with the newer Bulk Update functionality.