Service Incident August 22nd 2016 Access issues in Pod 4

16:40 UTC | 09:40 PT
We have resolved issues affecting Pod 4. If you still experience anything, please let us know.

16:30 UTC | 09:30 PT
We're continuing to investigate the access issues for Pod 4 customers. Please let us know if you're experiencing these issues.

16:29 UTC | 09:29 PT
We're continuing to investigate access issues for a handful of Pod 4 customers. Please let us know if you're continuing to see issues here.

15:55 UTC | 08:55 PT
We are looking into a few Pod 4 customers who are unable to access their account. If you are seeing issues, please reach out.

15:54 UTC | 08:54 PT
We're continuing to investigate a few Pod 4 customers who are unable to access their account. If you are seeing issues, please reach out.

15:51 UTC | 08:51 PT
“We are continuing to investigate a few Pod 4 customers who are unable to access their account. If you are seeing issues, please reach out.”

15:15 UTC | 08:15 PT
Some customers in Pod 4 reporting green screens. We are working to confirm scope, and will tweet again with any specific developments.

POST-MORTEM

Customers in Pod 4 reported green screens and access issues with their accounts. Root cause investigation, including checking all network metrics, alarms, and dashboards, revealed no issue within our network to cause this incident. Based on traceroutes from one customer, it was discovered that they were traversing an ISP's network. Disabling that ISP circuit in the West Coast datacenter, which routed traffic through a different circuit, resolved the issue. A case was opened with the ISP to investigate; however they have so far reported no scheduled maintenance and no latency, loss, or over-utilization on their circuits during the timeframe of the event. We're investigating additional monitoring to help track down this kind of issue in the future.

FOR MORE INFORMATION

For current system status information about your Zendesk, check out our system status page. During an incident, you can also receive status updates by following @ZendeskOps on Twitter. The summary of our post-mortem investigation is usually posted here a few days after the incident has ended. If you have additional questions about this incident, please log a ticket with us.