Service Incident Outbound Email Notification Delays May 22 2015

SUMMARY

As of May 22, 2015, 2:13pm PDT, we are working to resolve the following service incident:

Delays with outbound email notification. 

2:30pm PDT: We’re investigating a delay with email notifications for some accounts in our west coast data center. More details to come.

2:52pm PDT: We’re continuing to investigate a delay with email notifications for some accounts in our west coast data center. Thanks for your patience.

3:20pm PDT: We\\\'ve corrected the cause the email notifications delay in our west coast data center. Working now to decrease the delay time. Stay tuned...

3:47pm PDT: All email notification delays in our west coast data center are resolved. 

POST-MORTEM

This incident was found to be caused by a mail loop on a single ticket within a customer\\\'s account in the West Coast datacenter. The ticket was identified and soft-deleted, which fixed the issue. To prevent similar incidents going forward, we have planned improvements to mail loop detection in the application layer and possibly in the mail jobs; as well as investigate and improve the speed with which we process tickets with large numbers of CC.