Service Incident December 26th 2015

As of December 26th at 20:09 GMT / 12:09 PST, we are working to resolve the following service incident:

Some customers are unable to bulk update tickets.

 20:27 GMT / 12:27 PST

We’re investigating reports of some customers unable to bulk update tickets. More information soon.

20:47 GMT / 12:47 PST 

Our engineers have identified issues affecting bulk updates and are working towards resolution.

21:08 GMT / 01:08 PST

We are still working on root cause identification and complete resolution.

21:28 GMT / 01:28 PST

All bulk update service has been restored.  Post-mortem to follow.

POST-MORTEM

This issue was caused by connectivity issues experienced by an external provider on our infrastructure. This affected multiple customers, blocking bulk updates to all accounts located in our US-East data centre. The root cause analysis from our provider points to a line card issue in their core network.

Since the issue happened at their core network degrading performance, and not a disruption at the customer premise layer, our routers couldn't automatically detect the failure and converge to different upstreams. This has been made manually by our Operations team, who migrated the traffic away from the affected network. Once the service migration was completed, the impacted traffic stabilized and stuck jobs began to process. 

FOR MORE INFORMATION