Autoscaling (Axis2) Web Services on Amazon EC2

Fault tolerance, high availability, & scalability are essential prerequisites for any enterprise application deployment. One of the major concerns of enterprise application architects is avoiding single points of failure. There is a high cost associated with achieving high availability & scalability. We will look at an economical approach towards automatically scaling Web service applications while maintaining the availability & scalability guarantees at an optimum economical cost. This approach, involving the Amazon EC2 cloud computing infrastructure, makes it unnecessary to invest in safety-net capacity & unnecessary redundancy.

The Web service application developer should only need to write the application once, and simply deploy it on the cloud. The scalability & availability guarantees should be provided automatically by the underlying infrastructure.

Autoscaling refers to the behavior where the system scales up when the load increases & scales down when the load decreases. Auto-healing refers to an approach where a specified minimum deployment configuration is maintained even in the event of failures. Such an approach is essential for cloud deployments such as Amazon EC2 where the charge is based on the actual computing power consumed. Ideally, from the clients' point of view, in an autoscaling system, the response time should be constant and the overall throughput of the system should increase.

We will describe in detail an economical approach towards building auto-scaling Apache Axis2 Web services on Amazon EC2. In the course of this thesis, we will introduce well-known address (WKA) based membership discovery for clustering deployments where multicast-based membership discovery is an impossibility.

We will also introduce an approach towards dynamic load balancing, where the load balancer itself uses group communication & group membership mechanisms to discover the domains across which the load is distributed. In a traditional setup, a single load balancer fronts a group of application nodes. In such a scenario, the load balancer can be a single point of failure. Traditionally, techniques such as Linux HA have been used to overcome this. However, such traditional schemes have quite a bit of overhead and also require the backup system to be in close proximity to the primary system. In case of catastrophic situations, this approach can result in complete failure of the system. We will introduce an auto-healing scheme in case of load balancer failure using Amazon Elastic IP addresses & a load balancer group, which can overcome these shortcomings.

More...

WSO2 WSAS 3.1.0 EC2 AMI Now Available!

WSO2 Web Services Application Server (WSAS) 3.1 Amazon EC2 image is now publicly available. Now you can try WSO2 WSAS on the cloud without downloading anything.

WSO2 WSAS AMI ID is : ami-b304e5da

Note: If you are new to Amazon instances, you may have a look at the "Getting Started with EC2" section given below

Running a WSAS AMI Instance

Run following command to run the instance

ec2-run-instances ami-b304e5da -k [gsg-keypair]

Output will be something like:
INSTANCE ami-b304e5da pending gsg-keypair

Instance-id will be something like i-10a64379

Get Public DNS Name for Insance

Run following command to get the public DNS name of the instance

ec2-describe-instances [instance-id]

Output will be something like :
RESERVATION r-fea54097 495219933132
INSTANCE i-10a64379 ami-633bda0a domU-12-34-31-00-00-05.usma1.compute.amazonaws.com running gsg-keypair

In this example, the public DNS name of this instance is domU-12-34-31-00-00-05.usma1.compute.amazonaws.com

Access WSO2 WSAS Management Console

Visit the following URL

https://[public]:9443/carbon/

Login using username admin and password admin

Getting Started with EC2

Prerequisites

Following link contains the prerequisites for Amazon EC2

http://docs.amazonwebservices.com/AmazonEC2/gsg/2006-06-26/prerequisites.html

Setting up an Amazon Account

Please see:
http://docs.amazonwebservices.com/AmazonEC2/gsg/2006-06-26/account.html

Setting up Amazon-API Tools

Please see:

http://afkham.org/2008/10/how-to-setup-ec2-tools.html
http://docs.amazonwebservices.com/AmazonEC2/gsg/2006-06-26/setting-up-your-tools.html

Creating a GSG-Keypair

Please see:
http://docs.amazonwebservices.com/AmazonEC2/gsg/2006-06-26/running-an-instance.html

You could use the ElasticFox Firefox plugin to easily acomplish most of the tasks described above.

ApacheCon Europe 2009

I will be making two presentations at the ApacheCon Europe 2009 at Amsterdam, Netherlands.

1. Autoscaling Web Services on Amazon EC2
This will be based on the work I've been doing on autoscaling Axis2 Web services.

2. OSGi as Framework for Building a Product Line: Experience and Best Practices
This will be based on our experience with WSO2 Carbon & will be presented by Ruwan Linton & I. Ruwan will also be talking about Achieving Scalability and High Availability for clustered Web Services using Apache Synapse

Paul Fremantle, CTO of WSO2, will also be making presentations on Open Source for Interoperability, Apache license as a business model: Challenges and opportunities & Event Driven Architectures with Apache Synapse

Say hi to us if you drop by at ApacheCon Europe 2009 :)

How to setup EC2 tools

0. Download EC2 tools

1. Extract ec2-api-tools.zip to some directory. I've installed them at $HOME/.ec2

2. Install a JDK (>= 1.5)

3. export/set the following environment variables; JAVA_HOME, EC2_HOME, EC2_PRIVATE_KEY, EC2_CERT, PATH.
You could simply add these to you /etc/profile, /etc/bash.bashrc or ~/.bashrc files. e.g. Include the following in your /etc/profile, /etc/bash.bashrc or ~/.bashrc files

JAVA_HOME=/usr/local/java

EC2_HOME=/home/azeez/.ec2
EC2_PRIVATE_KEY=/home/azeez/.ec2/pk-xxx.pem
EC2_CERT=/home/azeez/.ec2/cert-xxx.pem

PATH=$JAVA_HOME/bin:$EC2_HOME/bin:$PATH

export JAVA_HOME EC2_HOME PATH EC2_PRIVATE_KEY EC2_CERT

How to create an EC2 AMI

The easiest way to create an EC2 AMI (Amazon Machine Image) is to select one of the publicly available AMIs which suits your requirement. Let's say you selected an Ubuntu image, ami-0757b26e. If you don't have an Amazon Web Services (AWS) account, first create one. Also you will need to download the EC2 command line tools & then set them up. For instructions on how to setup the EC2 tools, read this.

Follw these step to create your AMI:

0. Generate a keypair if you have not already done so
e.g. ec2-add-keypair key1
The output will be something like the following:

KEYPAIR key1
-----BEGIN RSA PRIVATE KEY-----
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
-----END RSA PRIVATE KEY-----

Copy the string starting from -----BEGIN RSA PRIVATE KEY----- up to -----END RSA PRIVATE KEY----- and save it in your keys directory, say in the /home/azeez/.ec2/keys/id_key1 file. Make sure that only the owner can read & write to that file.
i.e. chmod 600 /home/azeez/.ec2/keys/id_key1

1. Launch an instance of ami-0757b26e, providing a key, say key1, which you generated in step 0
ec2-run-instances ami-0757b26e -k key1
or you could use the ElasticFox GUI to do the same thing
The output will be something similar to
---------------------------------------------------------------------------------------------------
RESERVATION r-d5825cbc 610968236798 default
INSTANCE i-5c7dd335 ami-0757b26e pending key1 0 m1.small 2008-10-20T03:25:27+0000 us-east-1b aki-a71cf9ce ari-a51cf9cc
---------------------------------------------------------------------------------------------------

2. Connect to that instance using SSH,
e.g. ssh -i /home/azeez/.ec2/keys/id_key1 root@ec2-67-202-60-248.compute-1.amazonaws.com

3. Make the necessary changes to that instance. For example, you may install some custom software on that instance.

4. Upload your Amazon Web Services (AWS) private key (PK) & certificate (CERT) files to that instance. You can use scp to do this.
scp -i /home/azeez/.ec2/keys/id_key1 pk-XXX.pem cert-xxx.pem root@ec2-75-101-215-95.compute-1.amazonaws.com:/mnt/

5. On that instance, create an image of the current setup.
ec2-bundle-vol -k /mnt/pk-xxx.pem -c /mnt/cert-xxx.pem -u [user-id] -d /mnt

pk-xxx.pem = the PK file you uploaded in step 4
cert-xxx.pem = the CERT file you uploaded in step 4.
user-id = Your AWS User ID

In this step, you may wish to exclude some directories from the new image. Use the -e option followed by the ABSOLUTE path of the directories to be excluded. By default, some directories, like /mnt/, are excluded during image creation.

6. Create a bucket in Amazon S3. You can use s3fox to do this using a GUI, or use the command line tooling.

7. Upload the newly created image to your S3 bucket which was created in step 6
ec2-upload-bundle -b [your-s3-bucket] -m /mnt/image.manifest.xml -a [aws-access-key-id] -s [aws-secret-access-key-id]
aws-access-key-id = your AWS access key
aws-secret-access-key-id = your AWS secret access key.

8. Register the image. On your local machine, run
ec2-register [your-s3-bucket]/image.manifest.xml
The AMI ID will be displayed if the image was successfully registered. Say this was ami-af34d0c6

9. Test your image. Launch an instance of your new image. On your local machine run, e.g.
ec2-run-instances ami-af34d0c6 -k key1
or you could use the ElasticFox GUI to do the same thing

10. Connect to your new instance using SSH. This is similar to step 2.

That's it. You have just created your own AMI.

If you would like to make your AMI public, do the following:
1. ec2-modify-image-attribute --launch-permission -a all
2. Check the launch permissions of an AMI
ec2-describe-image-attribute -l
ami_id= the ID of the AMI

Difference between EC2 AMI, AKI & ARI

AKI (Amazon Kernel Image)
Kernel loaded by the Amazon "boot loader"

ARI (Amazon Ramdisk Image)
"Disk" used by boot loader during kernel load

AMI (Amazon Machine Image)
Everything post-boot, including loadable kernel module. An AKI & ARI can be specified when starting an instance of an AMI

How does an EC2 instance find information about itself?

Do an HTTP GET from http://169.254.169.254/1.0/meta-data/

Using curl,
$ curl http://169.254.169.254/1.0/meta-data/
ami-id
ami-launch-index
ami-manifest-path
hostname
instance-id
local-ipv4
public-keys/
reservation-id
security-groups

e.g. an instance can get its instance ID by,
$ curl http://169.254.169.254/1.0/meta-data/instance-id/

Passing parameters to EC2 instances

In my work related to making Axis2 autoscale on EC2, I had to figure out how to pass parameters to each instance. Apparently, the userdata parameter set during new instance launch can be used for passing parameters. When or after the instance starts up, by sending an HTTP GET request to http://169.254.169.254/1.0/user-data, the values passed for the userdata field can be retrieved. This was very useful for me since I can pass the initial parameters as well as Amazon Web Service keys to the instance. The key files are necessary because when the system autoscales, the primary EC2 instances needs to start up new EC2 instance.

For more details, read 'Using Parameterized Launches to Customize Your AMIs'

Syncing contents between EC2 & S3 - s3sync

In my ongoing work on autoscaling Web services on EC2, when an Axis2 EC2 instance boots up, it needs to load all the configuration files & the service+module repository from Amazon S3. While searching for an appropriate tool to do this, I came across s3sync

---

Installation
1. Install ruby
On Ubuntu, it was apt-get install ruby ruby1.9
2. Install openssl
apt-get install openssl
3. Install openssl-ruby
apt-get install libopenssl-ruby

Before Running
Make sure that the following environment variables are exported.
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
In order to obtain the values for those, you need to register with Amazon Web Services (AWS).

Examples from the s3sync README
1. Put the local etc directory itself into S3
s3sync.rb -r /etc mybucket:pre
(This will yield S3 keys named pre/etc/...)

2. Put the contents of the local /etc dir into S3, rename dir:
s3sync.rb -r /etc/ mybucket:pre/etcbackup
(This will yield S3 keys named pre/etcbackup/...)

3. Put contents of S3 "directory" etc into local dir
s3sync.rb -r mybucket:pre/etc/ /root/etcrestore
(This will yield local files at /root/etcrestore/...)

4. Put the contents of S3 "directory" etc into a local dir named etc
s3sync.rb -r mybucket:pre/etc /root
(You need to first create the directory, /root/etc. This will yield local files at /root/etc/...)

5. Put S3 nodes under the key pre/etc/ to the local dir etcrestore
**and create local dirs even if S3 side lacks dir nodes**
s3sync.rb -r --make-dirs mybucket:pre/etc/ /root/etcrestore
(This will yield local files at /root/etcrestore/...)

---

Only the contents that have been changed in the S3 bucket will be loaded from S3. Now that I can sync between my EC2 instance & S3 bucket, I simply need to write an init script that loads the configuration & repository from a specified S3 bucket. This init script needs to be automatically run when the instance boots up. This can be done easily using chkconfig. chckconfig can be downloaded from here. Installation instructions are available here. Create the script, say syncs3, place it in /etc/init.d, and add it as follows:

chkconfig --add syncs3

Remember to rebundle your instance into a new image. Now whenever an instance of the new image starts up, it will try to load the configuration from the S3 bucket.

Fault Resilient Dynamic Load Balancing for High Availability

In a traditional load balancing configuration, a single load balancer fronts a group of application nodes. In such a setup, the load balancer can be a single point of failure. This will have a big impact on availability.

One way we can solve this problem is by having 2 or more load balancers in a load balancer group. The state will be replicated amongst the load balancer group members.

As shown in the diagram, the load balancer group will front a group of application members. These members will be running the applications. State replication amongst the application members will take place as usual.

To make the load balancing dynamic, we can make the load balancer members part of the application group as well. In other words, the group membership management service can span across the application group and load balancer group. Therefore, the load balancers will always be notified when application nodes join or leave the group. However, the application nodes are not aware of the load balancers.

The diagram shows a load balancer group with an active-passive load balancer configuration. This can be nicely combined with elastic IPs on Amazon EC2, it order to make the failure completely transparent to the client. In the case of active load balancer failure, the elastic IP (in the diagram, 75.101.131.185), can be dynamically mapped to the passive load balancer and then the passive member takes over.

I'm currently implementing this model with Apache Synapse as the load balancer and WSO2 WSAS as the application nodes.

WKA Membership Scheme Fully Implemented

If you've been following my recent blog & Twitter posts, you already know that I have been trying to get Axis2 & WSO2 products to cluster on Amazon EC2. After a couple of weeks, I've been able to get the WKA (Well-Known Address) based membership management implementation to correctly work on EC2.

I've done all the testing and fine-tuning with WSAS
I have created an Ubuntu based AMI which contains two init scripts.

1. conf loader - This script loads the WSAS configuration files from Amazon S3
2. WSAS startup - This script fires up a WSAS instance when the AMI boots up

An Elastic IP has been assigned to one instance on this AMI. This instance is the one with the well-known address.

When new instances of this AMI are booted, the corresponding WSAS instances connect to the WKA member by sending a JOIN message. The WKA member responds with a MEMBER_LIST message, which contains the current group membership. Now, the new member sends a MEMBER_JOINED message to the group. See figure 1.

Figure 1: New Non-WKA Member(i) joins group

Also, the member who joins the group may be a WKA member. The relevant scenarios is, the WKA member crashing, and then coming back up. In such a case, the WKA member has lost all memory regarding the group membership. So, when others notice that a WKA member has joined, they will send a MEMBER_LIST message to this WKA member. Now the WKA member will be aware of the group membership. See figure 2.

Figure 2: WKA Member(i) joins group

Is this model correct?

1. Is it correct for the cluster to continue to operate when all WKA members are down? Actually the system will be correct, even though it cannot scale up anymore since new members cannot join the group. This is because when new members join the group, they try to connect to a WKA member and it is through the WKA member that it learns about the membership and then informs the group of its existence. When the WKA members are dead, this cannot happen. When a member dies, all others will automatically detect this since each node periodically pings all other members to see whether they are still alive. So even when all WKA members are dead, members leaving the group will be detected.

2. When a new non-WKA member joins the cluster, it gets the membership details from a WKA member. However, this WKA member may have just joined the group (after crashing). So the WKA member will inform the new non-WKA member that only both of them are in the group. If this happens, the system is in an incorrect state. To overcome this, before a non-WKA member sends a JOIN message to the WKA member, it will wait for a few seconds. This will give enough time for the other members in the group to detect that a WKA member has rejoined the group and to send it the MEMBER_LIST message. So the newly joining non-WKA member will get the correct member list.

ElasticFox

Do not like the command line tooling provided by Amazon EC2? Do you prefer working with GUIs like me? Then ElasticFox is the answer. This is a Mozilla Firefox extension for interacting with Amazon EC2 (API version 2008-02-01)

You can read more about ElasticFox here

Here is the Firefox XPI
The only drawback is that this still does not support Firefox 3, so I've had to switch back to Firefox 2 :(

EC2 Instance Types Summarized

I've summarized the information on EC2 instance types

One EC2 Compute Unit (ECU) provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor. * Pricing is per instance-hour consumed for each instance type. A customer will select an instance type based on the computing power & storage required.

In addition to paying for the instance hours consumed, the customer will also have to pay for the bandwidth consumed.