Reason For Outage
4th January 2017

Concerns regarding outages on 04/01/2017

This afternoon the servers were interrupted as connections to the database server "ECHO" could not be established.
This resulted in the CS:GO servers hosted on "FOXTROT" to cease operating normally.
Servers hosted on "ECHO" were completely shut down during the incident.
Such Symptoms Include:
Servers crashing.
Servers reverting to casual/competitive gameplay.
All users promoted to [CHAMPION] (Surf/Bhop)
Banned Users connecting
All users reverted to 0 points.
FastDL not operating.

The issue was traced down to broken & corrupt data within several MySQL tables. This data was replaced using backup data from our offshore location and the MySQL server proceeded to operate normally shortly afterwards.

To ensure stability, I/O tests, network logs, MySQL logs were reviewed to ensure health of the hardware and the processes in question, at which point it was decided to allow connections to the server once again.

Developers continued to monitor the resource usage until 8:00PM and an all-clear has been authorised.