Service Incident Twitter Facebook Ticket Creation Delays May 18 2015

SUMMARY

As of May 18, 2015 8:22am PDT, we are working to resolve the following service incident:

We are currently investigating reports of ticket creation delays for some customers. 

8:49am PDT: We are continuing to investigate the cause of ticket creation delays for some customers. 

9:12am PDT: We are working to resolve the ticket creation delay issue affecting some customers.

9:48am PDT:We are still working to resolve the ticket creation delay issue affecting some customers. Thanks again for your patience.

10:35am PDT: We are still working to resolve the ticket creation delay issue affecting some customers. Next update in one hour. 

11:39am PDT: Ticket creation delays in our Facebook and Twitter channels should be resolved. We\\\'re still investigating a separate email delay issue for a different set of customers.

POST-MORTEM

The root-cause of this issue was found to be a combination of three issues independent of each other but were overlapping in time and symptoms.  Dependent upon the datacenter, we had issues around ISP connectivity, Channel ticket creation traffic, or an issue with bulk updates via our API.  All 3 issues caused delays in ticket creation.

Once the issues were identified, our Engineers implemented the following actions to help resolve the issues.  For the ISP connectivity issues, we rerouted traffic through other avenues to alleviate this issue.  For the Channel ticket creation issue, a patch was rolled out to help make this process more efficient during times of higher traffic.  Lastly for the bulk update issue via the API, a patch was released to help alleviate strain on our API services during high traffic periods with the newer Bulk Update functionality.

There are no further steps to help prevent these issues in the future as the feature issues were resolved by a release of patches and the ISP issue was resolved in a timely manner once identified by a pre-existing process.