Service Incident November 19th 2015

SUMMARY

As of 11:05 GMT / 03:05 PST we are working to resolve the following service incident:

We are investigating performance issues with Zopim. We will provide more information shortly.

11:24 GMT / 03:24 PST:

We are happy to report the performance issues affecting Zopim are now resolved. A Postmortem will be available here shortly.

POST-MORTEM

This incident began due to a database pile-up, which caused by an emergency backup job and resulted in performance and login issues across the Zopim application.

An emergency database backup routine was executed to generate formal point-in-time backup to repair MySQL read services. High parallel thread concurrency from a backup tool caused memory and encrypted disk starvation which resulted in a pile-up of SQL queries.

To resolve the problems, we killed the offending backup routine as soon as we identified the root cause, and restarted MySQL to ensure full resolution.