8:15 PM PDT:
We’re currently investigating delays on outbound email notifications affecting customers from our west coast data center
9:21 PM PDT:
We’re still investigating the delays on outbound email notifications affecting some customers
11:00 PM PDT:
We are still working to resolve the issue impacting some customer\\\'s email notification delays
1:45 AM PDT / 9:45 AM GMT:
We have found the cause of the issue impacting some customer email notifications and are currently working on a fix
3:25 AM PDT / 11:25 AM GMT
We have fixed the issues causing email notification delays.
The root-cause of this issue was found to be an email loop that was sending a large amount of automated emails at a faster rate than what our systems could process. This in-turn affected other customers outbound email that were located in that same datacenter.
Once the issue was identified, our Engineers applied aggressive measures to first stop the email loop from continuing. The next step they applied was to remove the emails sitting in the queue that were a direct result of this email loop as they were unintentional in nature. Once this step was completed, they were then able to add extra resources to process the labored email queues until the services were caught up.
A number of actions were deemed necessary once this issue was resolved to help prevent similar future incidents. The most important is to improve the email loop detection logic. Secondary to that is to investigate and improve the speed at which the email processes work in handling the email queues. The investigative steps are still in progress on these tasks.