How to create an EC2 AMI

The easiest way to create an EC2 AMI (Amazon Machine Image) is to select one of the publicly available AMIs which suits your requirement. Let's say you selected an Ubuntu image, ami-0757b26e. If you don't have an Amazon Web Services (AWS) account, first create one. Also you will need to download the EC2 command line tools & then set them up. For instructions on how to setup the EC2 tools, read this.

Follw these step to create your AMI:

0. Generate a keypair if you have not already done so
e.g. ec2-add-keypair key1
The output will be something like the following:

KEYPAIR key1
-----BEGIN RSA PRIVATE KEY-----
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
-----END RSA PRIVATE KEY-----

Copy the string starting from -----BEGIN RSA PRIVATE KEY----- up to -----END RSA PRIVATE KEY----- and save it in your keys directory, say in the /home/azeez/.ec2/keys/id_key1 file. Make sure that only the owner can read & write to that file.
i.e. chmod 600 /home/azeez/.ec2/keys/id_key1

1. Launch an instance of ami-0757b26e, providing a key, say key1, which you generated in step 0
ec2-run-instances ami-0757b26e -k key1
or you could use the ElasticFox GUI to do the same thing
The output will be something similar to
---------------------------------------------------------------------------------------------------
RESERVATION r-d5825cbc 610968236798 default
INSTANCE i-5c7dd335 ami-0757b26e pending key1 0 m1.small 2008-10-20T03:25:27+0000 us-east-1b aki-a71cf9ce ari-a51cf9cc
---------------------------------------------------------------------------------------------------

2. Connect to that instance using SSH,
e.g. ssh -i /home/azeez/.ec2/keys/id_key1 root@ec2-67-202-60-248.compute-1.amazonaws.com

3. Make the necessary changes to that instance. For example, you may install some custom software on that instance.

4. Upload your Amazon Web Services (AWS) private key (PK) & certificate (CERT) files to that instance. You can use scp to do this.
scp -i /home/azeez/.ec2/keys/id_key1 pk-XXX.pem cert-xxx.pem root@ec2-75-101-215-95.compute-1.amazonaws.com:/mnt/

5. On that instance, create an image of the current setup.
ec2-bundle-vol -k /mnt/pk-xxx.pem -c /mnt/cert-xxx.pem -u [user-id] -d /mnt

pk-xxx.pem = the PK file you uploaded in step 4
cert-xxx.pem = the CERT file you uploaded in step 4.
user-id = Your AWS User ID

In this step, you may wish to exclude some directories from the new image. Use the -e option followed by the ABSOLUTE path of the directories to be excluded. By default, some directories, like /mnt/, are excluded during image creation.

6. Create a bucket in Amazon S3. You can use s3fox to do this using a GUI, or use the command line tooling.

7. Upload the newly created image to your S3 bucket which was created in step 6
ec2-upload-bundle -b [your-s3-bucket] -m /mnt/image.manifest.xml -a [aws-access-key-id] -s [aws-secret-access-key-id]
aws-access-key-id = your AWS access key
aws-secret-access-key-id = your AWS secret access key.

8. Register the image. On your local machine, run
ec2-register [your-s3-bucket]/image.manifest.xml
The AMI ID will be displayed if the image was successfully registered. Say this was ami-af34d0c6

9. Test your image. Launch an instance of your new image. On your local machine run, e.g.
ec2-run-instances ami-af34d0c6 -k key1
or you could use the ElasticFox GUI to do the same thing

10. Connect to your new instance using SSH. This is similar to step 2.

That's it. You have just created your own AMI.

If you would like to make your AMI public, do the following:
1. ec2-modify-image-attribute --launch-permission -a all
2. Check the launch permissions of an AMI
ec2-describe-image-attribute -l
ami_id= the ID of the AMI

Syncing contents between EC2 & S3 - s3sync

In my ongoing work on autoscaling Web services on EC2, when an Axis2 EC2 instance boots up, it needs to load all the configuration files & the service+module repository from Amazon S3. While searching for an appropriate tool to do this, I came across s3sync

---

Installation
1. Install ruby
On Ubuntu, it was apt-get install ruby ruby1.9
2. Install openssl
apt-get install openssl
3. Install openssl-ruby
apt-get install libopenssl-ruby

Before Running
Make sure that the following environment variables are exported.
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
In order to obtain the values for those, you need to register with Amazon Web Services (AWS).

Examples from the s3sync README
1. Put the local etc directory itself into S3
s3sync.rb -r /etc mybucket:pre
(This will yield S3 keys named pre/etc/...)

2. Put the contents of the local /etc dir into S3, rename dir:
s3sync.rb -r /etc/ mybucket:pre/etcbackup
(This will yield S3 keys named pre/etcbackup/...)

3. Put contents of S3 "directory" etc into local dir
s3sync.rb -r mybucket:pre/etc/ /root/etcrestore
(This will yield local files at /root/etcrestore/...)

4. Put the contents of S3 "directory" etc into a local dir named etc
s3sync.rb -r mybucket:pre/etc /root
(You need to first create the directory, /root/etc. This will yield local files at /root/etc/...)

5. Put S3 nodes under the key pre/etc/ to the local dir etcrestore
**and create local dirs even if S3 side lacks dir nodes**
s3sync.rb -r --make-dirs mybucket:pre/etc/ /root/etcrestore
(This will yield local files at /root/etcrestore/...)

---

Only the contents that have been changed in the S3 bucket will be loaded from S3. Now that I can sync between my EC2 instance & S3 bucket, I simply need to write an init script that loads the configuration & repository from a specified S3 bucket. This init script needs to be automatically run when the instance boots up. This can be done easily using chkconfig. chckconfig can be downloaded from here. Installation instructions are available here. Create the script, say syncs3, place it in /etc/init.d, and add it as follows:

chkconfig --add syncs3

Remember to rebundle your instance into a new image. Now whenever an instance of the new image starts up, it will try to load the configuration from the S3 bucket.

WKA Membership Scheme Fully Implemented

If you've been following my recent blog & Twitter posts, you already know that I have been trying to get Axis2 & WSO2 products to cluster on Amazon EC2. After a couple of weeks, I've been able to get the WKA (Well-Known Address) based membership management implementation to correctly work on EC2.

I've done all the testing and fine-tuning with WSAS
I have created an Ubuntu based AMI which contains two init scripts.

1. conf loader - This script loads the WSAS configuration files from Amazon S3
2. WSAS startup - This script fires up a WSAS instance when the AMI boots up

An Elastic IP has been assigned to one instance on this AMI. This instance is the one with the well-known address.

When new instances of this AMI are booted, the corresponding WSAS instances connect to the WKA member by sending a JOIN message. The WKA member responds with a MEMBER_LIST message, which contains the current group membership. Now, the new member sends a MEMBER_JOINED message to the group. See figure 1.

Figure 1: New Non-WKA Member(i) joins group

Also, the member who joins the group may be a WKA member. The relevant scenarios is, the WKA member crashing, and then coming back up. In such a case, the WKA member has lost all memory regarding the group membership. So, when others notice that a WKA member has joined, they will send a MEMBER_LIST message to this WKA member. Now the WKA member will be aware of the group membership. See figure 2.

Figure 2: WKA Member(i) joins group

Is this model correct?

1. Is it correct for the cluster to continue to operate when all WKA members are down? Actually the system will be correct, even though it cannot scale up anymore since new members cannot join the group. This is because when new members join the group, they try to connect to a WKA member and it is through the WKA member that it learns about the membership and then informs the group of its existence. When the WKA members are dead, this cannot happen. When a member dies, all others will automatically detect this since each node periodically pings all other members to see whether they are still alive. So even when all WKA members are dead, members leaving the group will be detected.

2. When a new non-WKA member joins the cluster, it gets the membership details from a WKA member. However, this WKA member may have just joined the group (after crashing). So the WKA member will inform the new non-WKA member that only both of them are in the group. If this happens, the system is in an incorrect state. To overcome this, before a non-WKA member sends a JOIN message to the WKA member, it will wait for a few seconds. This will give enough time for the other members in the group to detect that a WKA member has rejoined the group and to send it the MEMBER_LIST message. So the newly joining non-WKA member will get the correct member list.

Manage your files on Amazon S3 with S3Fox

S3Fox Organizer helps you organize/manage/store your files on Amazon S3. It provides a cool FTP client like interface for transferring files between S3 and your local file system.

This is comes as an easy to use Firefox extension which can be downloaded from here

S3 Bucket Explorer is another GUI based tool for managing S3. It comes as a stadalone application.