Service Incident February 1st 2016

 

01:02 UTC | 17:02 PT
The bulk update delay issue has been resolved. 

00:16 UTC | 16:16 PT
Mitigation efforts are underway to alleviate bulk ticket update delays in our European, West, and East coast data centers.

23:48 UTC | 15:48 PT
We're receiving increased reports of bulk updating ticket delays in EMEA/West coast USA. Our operations team is currently investigating.

POST-MORTEM

This incident was brought about when our bulk job processing queues increased dramatically, causing bulk ticket updates to be significantly delayed for data centers in Europe and US West Coast. The queue build up was caused by a ticket sharing loop on a number of customer accounts. The common denominator in this case was that all the accounts in question were using "Apply macro after update", a now-deprecated beta feature that was known to cause issues in the past. To remediate, we had to temporarily disable ticket sharing on the affected accounts.

While we do have a loop detection mechanism for sharing agreements, it is only designed to detect loops between two accounts; where more than two instances are involved, the loop will not be detected. We're working on changing this to catch larger loops. We're also working on implementing dynamic rerouting of queues to prevent queue buildup from affecting performance for our customers. 

 

FOR MORE INFORMATION