Unwire your instance

When deploying on the cloud, you have to make sure you are not tied to your machine because at any point it might not be there anymore.

One solution to this would be to bundle an AMI (Amazon machine image), but then this ties you to the region your AMI is in (Us / Eu) and also to the architecture it’s been built for, so if you were gonna take that route, for high flexibility you’d have to bundle the following AMIs:

• 32bit US
• 32bit EU
• 64bit US
• 64bit EU

And even that doesn’t solve everything, because when a new distribution of your operating system comes out, you’d have to rebundle.

Also you have to take into consideration the ever-changing demands of clients and security patches applied to various software you’re using. These will all trigger a rebundle if you were gonna stay up to date.

It seems AMIs defeat the dynamic nature of the cloud.

We’ve been doing 100% of our hosting on EC2 for the past 2 years, and we’ve learned the best way to go is to run on vanilla instances that get customised on startup.

Running on vanilla instances has quite a few benefits over running from a bundled image: high availability (you’re not tied to a specific region), high flexibility (you can run either 32bit machines or 64bit, at any time) and the software you’re running is always up to date, just to name a few.

This does come at a cost though, it will take longer for your instances to start, as everything will have to be installed before they’re ready to use. On average, we’ve seen times between 5 and 10 minutes until an instance is ready, that includes startup time, software installation and software configuration.

When it comes to software configuration, if you’re using Amazon’s elastic block store (or persistent storage) to hold your applications, you could store your configuration there as well. This way when you start an instance and attach the EBS volume to it, all you have to do is symlink the configuration into place.

We used this way of configuring for quite a while, and our directory structure looked something like (relative to the EBS path):

	.config/
		apache2.conf
		php.ini
		eaccelerator.ini

We would then simply symlink `.config/apache2.conf` into `/etc/apache2/conf.d/`, but this meant that whenever we added new configuration files, we would have to change our startup script to symlink the new configuration.

We soon realised that even though this is an “ok” solution, it wasn’t really flexible.

As we’re currently in the process of moving part of our hosting onto Scalr, we also built “tool to be named here”, which handles software installation and configuration as well as application installation and configuration for us.

This time around, we’ve done configuration similar to how Debian packages get installed, map the file system root into a directory, then copy everything underneath into place.

An example configuration structure might look like this:

	config/
		etc/php5/
			apache2/conf.d/
				eaccelerator.ini
			conf.d/
				suhosin.ini

Once the instance is started and everything is installed, our setup tool then looks into the “config” directory (which is bundled with the setup tool as a Debian package, so our instances are always running the latest version), and then it copies everything into place.

Since the everything under the “config” directory has the same structure as “/”, we don’t need to worry about paths, everything goes where it should (we check this assumption before we package though).

Here’s the part of the setup tool that puts configuration into place:

1
2
3
4
5
6
7
8
9
10
11

	for config in `find $ROLE_PATH/$type/config -type f`; do
		TARGET_CONFIG=${config/#$ROLE_PATH\/$type\/config/}
		mkdir -p ${TARGET_CONFIG/`basename $TARGET_CONFIG`/}
 
		echo "Installing configuration: $TARGET_CONFIG"
		$config > $TARGET_CONFIG
		if [[ $? -ne 0 ]]; then
			echo "$config failed"
			exit 2
		fi
	done

“$ROLE_PATH/$type/config” would actually look something like: “/usr/local/3ev/server/roles/web/config”, we’ll talk more in depth about how this all works in a future blog post.

You will notice on line 6 that we actually call the configuration file as a script and its output goes into where the config file should be, instead of just copying it over.

That’s because the config file looks like this (eaccelerator.ini):

	#!/bin/bash
	cat <<-config
		zend_extension="`php -i | grep extension_dir | awk '{ print $3 }'`/eaccelerator.so"
		eaccelerator.shm_size="16"
		eaccelerator.cache_dir="/tmp/eaccelerator"
		eaccelerator.enable="1"
		eaccelerator.optimizer="1"
		eaccelerator.check_mtime="1"
		eaccelerator.debug="0"
		eaccelerator.filter=""
		eaccelerator.shm_max="0"
		eaccelerator.shm_ttl="0"
		eaccelerator.shm_prune_period="0"
		eaccelerator.shm_only="0"
		eaccelerator.compress="1"
		eaccelerator.compress_level="9"
	config

Initially the configuration files were in fact just plain text files, but then we realised that values will change on a per-instance basis, so we turned them into simple bash scripts that output the configuration.

Written by Andrei on the 30th April 2010