euIRC network incident 2013-04-22
Here is a short explanation what happened yesterday.
The troulbe started at 19:47 GMT+2, and was caused by a kernel bug of the hub server. This bug caused all timers to stop and was leading to disconnects and reconnects of all servers. During this netjoins a similar bug caused a rare condition and was setting nearly all channelmodes on some of the older 32bit servers. Well, this wouldn't be a problem when not this servers were doing some more reconnects and started to push their broken channels back into the network.
To fix this issue, we had to restart all servers and import a backup of our network cloud storage (this contains all channelmodes, bans, badword lists etc.). We had all set up again, but one of the old servers wasn't restarted and poisoned the network again. Ok, another restart and backup import, but this time we checked every server carefully before linking it back to central. solved around 21:30 GMT+2
Sry to all german t-com customers, the backup was taken one day before t-com changed the hosts to dip0.t- ipconnect.de, so bans, invex and accesslists regarding this domain have to be set again.
Back to news overview
