It was a calm day at 4AM UTC time when the mymangaworld.com team wanted to setup a mail server there we’re 2 solutions we were considering mail cow and roundcube + postfix and dovecot. Mail cow runs in a docker container so we thought it might be better to use that as we use cloudpanel to handle all the web stuff on our server and that uausally binds to alot of ports non docker apps use such as 3306 the MySQL port.
So its set then just another docker install and all will be good to go right *EXTREMLEY LOUD INCORRRECT BUZZER* (play the sound file below to increase the immersion)
NOPE EVERYTHING WENT WRONG EVERYTHING
seconds after the install script runs the server freezes 2cookie opens up portanier to have a look and nothing laods… I then spam refresh the page hoping it was just javascript being javascript but no.. this was a server issue I open up cloudpanel and gateway timeout.
i open up SSH which was still working and run HTOP (A proformance monitor on linux similar to Task manager on windows) and CPU spikes to 100% and System D is using most of it I have no clue whats happening at this point
At first i thought it was a NodeJS app but no turns out it was mailcow all along 1 reboot later the cpu useage came back down but it fried the DB server that Cloudpanel uses somehow
Chapter 2 a last resort
WIth the DB server dead there was not much more we could do however luckiley before installing MailCow 2cookie installed OpenVPN access which went smoothly but since it was installed to the host 2cookie made a backup before doing so. The install went smoothly (OpenVPN is chill its mail cow that f*cked the server) luckiley we could restore to that backup.
Oh did i mention SSH was f*cked too
However there was a issue there was a small chance of the restore failing making us lose everything but then again the server was basicley f*cked at the moment
However with this in mind At 5AM UTC we made the hard choice to restore to that backup saving the server.
And we’re now back online! if theres one thing to take away from this its always have snapshots and be ready!