There are a lot of activities that go on in a software testing environment. Consider a situation where you want a group of people to evaluate a set of options and choose one, single best option. For example, suppose you are creating a Web-based tool program for use on your company’s intranet. You have designed four quite different user interface prototypes, and you ask a group of people to evaluate each prototype and rank their preferences from best to worst. Now the question is just how can you make sense of this data? There is a large body of knowledge on various techniques to analyze group evaluations of a set of alternatives designed to select the best option. However, I’ve noticed that these techniques are almost totally unknown to most software testers. There are many fascinating effects in group analyses. Suppose you have four options (A,B,C,D) and 14 evaluators, and some data like this:

A > B > C > D according to 8 people

B > C > D > A according to 4 people

D > C > A > B according to 2 person

B > C > D > A according to 4 people

D > C > A > B according to 2 person

Suppose you decide to give 3 points for a top ranking, 2 points for a second place, 1 point for a third, and 0 points for a fourth place. Then option A has 26 points, option B has 28 points, option C has 20 points, and option D has 10 points. So option B is best. But then suppose someone observes that option D is such a loser it shouldn’t have even been there in the first place. You agree and toss out option D and now the data looks like:

A > B > C according to 8 people

B > C > A according to 4 people

C > A > B according to 2 person

B > C > A according to 4 people

C > A > B according to 2 person

But now if you re-compute you’ll see that option A has 18 points, B has 16, and C has 8, and so now option A becomes the winner instead of option B! Anyway, I am writing up a primer on collaboration techniques for the monthly Test Run column in MSDN Magazine. I also intend to submit a paper on this topic to the Pacific Northwest Software Quality Conference, and present it in October.