Service Disruption Email Delays July 20th 2015

2:40am GMT+1/07:40am PST

We are currently investigating outbound email delays for some customers on our west coast data center. Apologies for any inconvenience!

4:20am GMT+1/09:20am PST 

We are still investigating the cause of outbound email delays affecting our West Coast data center.

 4:40am GMT+1/09:40am PST 

We are continuing to investigate the reports of delayed emails in our West Coast data center.

 4:56am GMT+1/09:56am PST 

We are still investigating the outbound email notifications affecting some customers. Will update again in 30 minutes or when new info is known.

 5:35am GMT+1/10:35am PST 

We’ve identified the cause of outbound email delays impacting some customers and are working to mitigate them.

 6:35am GMT+1/11:35am PST 

We are deploying a fix to help expedite the delivery of the delayed outbound emails from our west coast datacenter 

 7:13am GMT+1/12:14pm PST 

We’ve finished a deploy to help mitigate the delay of email delivery and are seeing improvements, but are actively monitoring the situation.

 7:51am GMT+1/12:51pm PST 

At this point all of the queued outbound notifications have been processed and are processing normally. Postmortem to follow.

POST-MORTEM

Mail servers in our West Coast data centre started processing less outbound email than was being created, leading to thousands of emails queuing up in. To clear the pending email queue, we worked to rearrange the number of servers dedicated to outbound email, including adding new services to the data centre.  

We\\\'re still investigating to understand the root cause of this incident fully; however, as short-term remediation actions, we\\\'re working on increasing operational capacity of the West Coast data centre to prevent this or similar incidents from recurring.