Complete Failure of Paul, Migration to New Servers

If you’re hosted on paul.cleverhost.ca, which is our oldest server in the fleet at four years old, then you may have noticed that websites and email have not been working normally since roughly 13:30 EDT on Wednesday, April 19. There was intermittent service that afternoon, however the server has been unable to perform its duties since roughly 20:00 on Wednesday.

The reason for this is that a customer account was attacked and breached on Wednesday afternoon. The attacker was able to gain administrator-level access, and then began to attempt a takeover of Paul. As soon as we saw what was happening, we took steps to attempt to regain control of the server. These attempts were successful, to a point.

In situations like this, it is not wise or safe to continue using the server. What we had hoped to do was to maintain enough control over the server to allow for a smooth transition to a new server. Unfortunately we weren’t able to do this as services like Apache, which serves your websites, Exim, which serves your email, and SSH, which allow us to connect to the server’s command line and perform commands, were all shut out Wednesday night.

We then turned our attention to migration, which is what we’ve been busy with ever since, along with doing our best to keep as many people as possible up-to-date on Twitter and Facebook.

While I cannot give you a specific date and time to when your account will be restored, I want to assure you we’re working as quickly as possible to bring this problem to an end for everyone. We have restored several accounts already, and are working through the night to get the work completed as soon as we can.

The good news is that we have your data thanks to backups and some files pulled off the server before it went offline. We have hopes that we’ll be able to bring the old Paul back online temporarily in order to make this transition a bit smoother, still. Our partners at the data centre are looking at the server now to see if we can do that.

In the meantime, if you are a heavy user of our email hosting services, you can expect to remain on Paul. If not, it is likely your account will be moved over to a third, new server called Alia.

We thank you for your patience and words of support you’ve been sending our way, and are eager to maintain and regain your trust.

Thank you,

Derek Silva