Amazon Instance crash over the weekend – Great recovery thanks to Flywheel & UpdraftPlus

 

Big call out and thanks to Flywheel and Updraft who saved our ass over weekend after the Startup88.com site got the speed wobbles and fell over.

We had been running on an decent sized Amazon EC2 instance for a year using a prepackaged Bitnami WordPress Install which I had been pretty happy with however as the site started to ramp up over the last 6 months and get more traffic spikes it became less reliable and we were seeing instance crashes a few times a week.

Saturday morning and we are updating articles on the site, the site has been through the second busiest time since launching and the MYSQL database on the Amazon Instance crashed and corrupted the MYSQL installation and wouldn’t recover. (apparently that’s what happens when you get 3x your normal daily traffic before 9am).

Even after rebooting a bunch of times which normally fixes the problem, it stayed broke. Even launching another instance just replicated the problem. The site itself was not so much the problem as the installation of MYSQL which was completely trashed.

I had been planning to move away from Amazon anyway as I was tired of supporting this myself or trying to get reliable affordable sysadmins, the site became wobbly anytime we got a surge of traffic and fortunately I had already setup an account at Flywheel.

None of this was Amazon or Bitnami’s fault, it was mine, I should have moved much sooner, you know you have to do the support yourself when you use Amazon EC2 but I didn’t think scaling was going to be much of a problem so soon and I knew that short of rebooting or loading a new instance, I had neither the time or the skills to be a sysadmin (Im still not sure why it crashed but then thats why Im not a sysadmin, suspect config and plugin issues).

But I figured it was going to be a drama to transfer because it was the weekend.

So I send Flywheel a message on Saturday morning thinking I would be struggling to get a reply, telling them I need to move now, or otherwise it will be another month before we get a chance.

Less than 10 minutes later I get a response (it was after hours on Friday night in their time zone) and I have a Google Chat session going with an engineer  (who turns out to be one of the co-founders Dusty Davidson).

Luckily we put in place a good backup solution called UpdraftPlus, which backs all the themes, plugins, images and the database up nightly to Google Drive (its a pain in the ass to work out how to do this manually on Amazon).

Flywheel offers free migration, normally during their working hours so I shared our backup with Dusty he had it loaded up, we switched the DNS over and although DNS took a few hours to propagate globally the site started serving traffic almost immediately.

So thanks again Flywheel and Updraft  for fantastic support.