The Adventures Of Judy2K

In search of the new, new thing

My Server Broke. So I Fixed It

I was recently forced to set up a new server, and I learned (or reinforced!) a few things along the way. Apologies for this being a bit rambling!

The recent scare around Shellshock prompted me to upgrade Bash on my server. Unfortunately, being the blasé soul that I am, I hadn’t installed the LTS version of Ubuntu when I set up my server, and updates are now no longer being provided for the version that I had. So, I ran ubuntu-release-upgrade or whatever the command was, and waited for it to complete. I didn’t backup first. I don’t know why I didn’t backup first - this is all hosted on EC2, so taking a snapshot of a server is a trivial thing to do. But I didn’t.

Eventually, the command completed, the server rebooted, and … nothing. It never came back. I can’t connect to it. I can attach the drive to another VM and look around, but there’s nothing being logged.

I decided this was a great opportunity to do something I’ve been meaning to do for ages. Have I told you that I love Ansible? If you’re vaguely technical, and you know me, I’m pretty certain I have told you that I love Ansible.

So, anyway, a few hours of work (and 1.5 weeks later) I have a new server set up. It was fantastically straightforward to get set up. I have the following workflow:

  1. Modify the Ansible scripts
  2. Run vagrant provision against a local VirtualBox VM
  3. Check it does what I want.
  4. Run Ansible against my EC2 instance.

I haven’t had a single problem so far. It’s the cleanest, best-documented server setup I’ve ever done (I consider the Ansible scripts to be documentation!). A random selection of things I’ve discovered along the way are:

Gitreceive

Gitreceive is a bash script that sets up a git user account on your server, and then will allow you to push repositories to it (a bit like Heroku). When the repo is pushed, the HEAD is tar-gzipped and this is piped to a script you specify.

You can do anything when a repo is pushed - if it’s a static site, you can just unzip it into your web root; if it’s something more complicated like a Django app, you can deploy it, run migrations, build static assets … you get the idea. I really like the simplicity of Heroku deployments, so I’m really looking forward to using this simple tool for automating future things that I’m planning.

If you use Docker, and like the sound of this, then you might want to have a look at Dokku, which is built on top of gitreceive, and deploys to Docker containers using Heroku’s own buildpacks.

Upstart

I am much more familiar with Supervisor as something that can keep me from directly writing initd scripts. But our DevOps guys at FanDuel use Upstart quite a bit, so I tried it, and it … wasn’t horrible. I’m not a convert yet, but I’m using it, and I’m happy.

uWSGI

uWSGI used to be a Python WSGI server, for hosting things like Django, Flask, etc. Now, it’s, well, according to the documentation: “The uWSGI project aims at developing a full stack for building hosting services.”

And it really seems to succeed, too! There are more configuration options than there are grains of sand in Dubai, but somehow it’s still reasonably simple to set up. It’s a great piece of software, and can host software built on a number of different platforms, not just Python.

Silver Linings

Looking on the bright side of what’s happened, I’m actually pretty happy. I’m happier with the new server, and I’ve learned a bunch of new stuff. Also, every time I go through this process (I recently set up a server for rebkwok) I get faster and better at it.

The feeling that I really get from this totally automated setup is of agility. I know that setting up a similar server means simply adapting the scripts I’ve crafted. I’ve also already changed some of my sites’ Nginx config and because it’s templated, I’ve been able to update them all in one go. Being able to modify and switch in and out and test things gives me much more confidence than the last server, which was configured by hand.

Maybe there’s something in this whole DevOps thing, after all.

Also, next time … I’m going to make that backup first.