Service Incident November 4th 2015

As of 12:20am UTC / 16:20 PST we're investigating the following service incident:

We're currently investigating reports of an issue impacting ticket creation for inbound emails. More information to follow.

12:30am UTC /16:30 PST:

We have identified an issue with delays in ticket creation from inbound email & some Help Center errors. We are currently implementing a fix.

12:50am UTC /16:50 PST:

We are continuing to work on delays with ticket creation via inbound email and Help Centre errors, Next update in 30 minutes.

01:30am UTC /17:30 PST:

We are happy to report that the issue with the inbound email ticket creation delay and Help Center errors is now resolved. A Post Mortem will be available shortly.

POST-MORTEM

This incident was caused by an emergency change that was carried out to update some of our network certificates, which resulted in failures of some jobs (mainly inbound email processing), as well as runtime failures for Help Center theme edits.

After finding the root cause, we immediately reverted the emergency change. Once validated that this has resolved the errors, we put a recovery mechanism in place to allow the jobs to  process as they remained queued at this point. During recover, some users began reporting 500 errors when editing Help Center themes. As the change revert propagated on all mirrors, errors subsided and a full resolution was confirmed as inbound email queues returned to normal.

FOR MORE INFORMATION