Service Incident May 6th 2016 Inbound email delays

09:13 UTC | 02:13 PT
We are happy to report that the inbound email delays for some of our EU hosted customers have now been resolved!

08:52 UTC | 01:52 PT
We are seeing significant improvements as we continue to work on inbound emails being delayed for some of our customers.

08:17 UTC | 01:17 PT
Our Operations Team continues to work on resolving inbound email delays for some of our customers in our EU datacenter.

08:00 UTC | 01:00 PT
We continue to work on resolving inbound email delays affecting some customers.

POST-MORTEM

Customer communication via email on the inbound side was impacted in our Dublin datacenter. Email replies to tickets and new ticket creation via email were delayed by up to an hour or more. The issue was caused by a network congestion issue which led to packet loss. Mail was specifically impacted due to a dependency on the AWS S3 service for mail processing, although no emails were lost. The issue was resolved by rerouting traffic to avoid the congested AWS path.

Remediation action items include improving Network Operations detection, improving alerts for this type of issue, and adding logging metrics so we can more easily determine when an S3 slowdown is affecting the mail parsing queue.

FOR MORE INFORMATION

Please subscribe to this article for regular updates until the issue is resolved. If you aren't subscribed to our Twitter feed, we encourage you to do so in order to get the most current information about any service issues. We also record all site outages on our system status page where you can see the past 12 months of service uptime. If you have questions about this issue, please open a ticket with us by sending a note to support@zendesk.com.