Service Incident on October 29th 2015

SUMMARY

As of October 29th, 2015, 2:10pm PDT, we are working to resolve the following service incident:

Performance and access issues for some of our customers.

2:15PM PDT: 

Service has been restored.

POST-MORTEM

This short outage occurred while bringing up routing adjacencies between production router in our US datacentres to a new VPN firewall, a routing export policy was applied that did not include the explicit "reject" statement at the end of the policy. Due to this change and missing statement, the default route from the VPN firewall was injected into production environments, causing a temporary black-hole of production traffic in the affected data centres. To recover the service, routing adjacency was shut down.

We will be ensuring that all policies have a reject statement to prevent any further recurrences of such incidents.