In traditional open source projects, there is no formal QAing involved and it is the responsibility of the developers themselves to carry out this task. Due to the collaborative nature of open source software development & the participation of a wide and diverse community, the quality of open source projects tends to be higher than closed source, proprietary projects. However, as a company which provides support for open source SOA middleware products,
WSO2 has to ensure that all our product releases go through proper QA cycles before being shipped. The mindset required to test middleware products is different to the mindset of testing end-user products. This is because, the majority of the end users of middleware products are developers who write applications for other 'real' end users. The other end users include system deployers & administrators.
One of the biggest challenges we faced in early 2006 while trying to recruit Quality Assurance (QA) Engineers to test WSO2 middleware products was finding personnel who can write code to exercise all the features in our products. In reality, it is very difficult to find QA Engineers or testers who have the ability to write code which tests SOA middleware products. There are so many standards & specifications involved, hence these personnel have to first get a good understanding of these specifications & standards & then write tests which exercise the products. Traditionally, in most cases, I've seen testers having a spreadsheet with them, running the test cases using some product UIs and simply recording the status of the test; pass or fail. This process is automated later, and then there are stress & load tests. The testers, at most, write some test scripts using some kind of testing framework. In the case of middleware products, the UI is only a minor portion of the product. Middleware run the code developers write. Like I mentioned earlier, most of the end users of middleware products are developers who write applications for other 'real' end users. Hence the testers have to wear the developers' hat in order to effectively test middleware.
After searching for QA Engineers with the right attitude, we were able to put together a team of QA engineers. Over the past couple of years, this group of people has developed into an elite team ably led by
Charitha Kankanamge. His blog is used as a reference by thousands of developers who use our middleware products world-wide, and contains intricate technical details related to various technical topics. This team works very closely with the developers. If you look at some of the blogs of other WSO2 QA Engineers such as
Evanthika &
Krishantha, you will notice the level of technical competence & knowledge our QA engineers demonstrate. Some of these QA Engineers also happen to be Apache committers. Their contribution is one of a main reasons for the high quality exhibited by our products. We follow an agile & lightweight QA process at WSO2, which doesn't impose unnecessary overhead. I've seen some QA teams blindly trying to follow heavy weight processes and losing site of the real goal of QAing; ontime delivery of high quality products. Undoubtedly, this is the best team of QA engineers I've come across in my career. The facts that things always work out of the box for our users & no major issues have been reported by most of our users bears testimony to this.