Service Incident on July 25th 2015

SUMMARY

As of July 25th, 2:08pm PDT, we are working to resolve the following service incident:

2:08pm PDT: Network interruption at our US east coast data center.

2:59pm PDT: We are still looking at the issue.

3:42pm PDT: Some customers had Voice issues. Our team is investigating and monitoring.

4:24pm PDT: All known issues related to this incident have been resolved. Postmortem will be posted here once available. 

POST-MORTEM

During a scheduled and planned maintenance window to replace a failed network core chassis, we encountered a network disruption due to an unexpected routing loop. Faulty network elements are meant to be taken offline during planned maintenance, while this time, the chassis was left powered on. We turned the faulty chassis off as soon as we received first reports of connectivity issues. Such incidents are rare but we will be working to refine our internal procedures to ensure this kind of error doesn\\\'t recur going forward.