Downtime 13 November 2013

Due to an error in rsyslog our logs filled our RAID to the last Bit (over 144GB of logs ). Normally the log daemon is supposed to split the log every day; however, it didn’t do that recently. In the result MySQL was not able to run as the disk was full.

We are very sorry that the site was down for ~21 hours, but we also want to enjoy our normal life outside the Internet; hence, we became aware of the problem just an hour ago.

For now, the RAID is back filled to 20% only, so even if rsyslog should run berserk again, it should take half a year or so for this error to affect site operations again.