Service Disruption Apr 27 2015 at 10 12 AM PDT

10:12 AM PDT:

We are currently investigating a delay in outbound email notifications for some customers.

10:37 AM PDT:

We’re still investigating the delays on outbound email notifications affecting some customers

10:56 AM PDT:

We’ve identified issue affecting outbound email notification delays and are working to resolve.

11:16 AM PDT:

We are still working to resolve the issue impacting some customer\\\'s email notification delays. Next update in 30 minutes

11:47 AM PDT:

We are still working to resolve the issue impacting some customer\\\'s email notification delays. 

12:23 PM PDT:

We are still working to resolve the issue impacting some customer\\\'s email notification delays.

12:57 PM PDT: 

We are working to resolve the backlog impacting some customer\\\'s email notification delays. Will update when we have a significant change

2:00 PM PDT: 

Our Devs are still looking at the issue

3:00 PM PDT: 

Our Devs are still looking at the issue

4:10 PM PDT: 

All-clear. There are no more email delays up to this point. 

POST-MORTEM

The root-cause of this issue was found to be an email loop that was sending a large amount of automated emails at a faster rate than what our systems could process.  This in-turn affected other customers outbound email that were located in that same datacenter.  The email loop itself was not identified as the cause of this issue until we had another observed email delay later in the day.  

For this specific outage, a resolution was found by applying extra resources to the email queues in order to process the email at a faster rate than the jobs were coming in.  While this process was occurring, our developers were continuing to investigate the increased load on the email processors as the initial belief was that the cause was from a product release earlier in the day.

To learn more about the email loop that was identified as a cause of this issue during a service incident later in the day, please go https://support.zendesk.com/hc/en-us/articles/205023218-Service-Incident-Apr-27-2015-at-8-00-PM-PDT.