Mutation Testing

Mutation Testing is a technique that software engineers can use to measure the effectiveness of their overall testing effort. Suppose a team of testers has created many individual tests; let’s call the collection of tests the test suite. In mutation testing, the original system/program under test is mutated to create a faulty version called a mutant.  The mutant program is tested using the suite of individual test cases.  The test suite should produce new test case failures; if no new failures appear then that probably means the test suite does not exercise the code path containing the mutated code, which means the test suite does not fully test the system/program.  You can then create new test cases that do exercise the mutant code which may reveal bugs. Notice that the term "mutation testing" is somewhat misleading — in mutation testing you do not actually test the system under test, you measure the effectiveness of a collection of tests.
The screenshot below shows an example of mutation testing in action. I am analyzing a library named LibUnderTest.dll.  It is a demo library containing a single Max() method. In the first part of mutation analysis, the DLL under test is tested using an existing test suite. In this case the test suite is named TestHarness.exe. Again, the test suite is just a demo to keep things small and the ideas clear. The test suite has only 10 test cases for simplicity. Notice that all 10 test cases pass. After establishing baseline test results, the DLL under test is examined for potential mutations. I disassemble the DLL under test and examine the resulting .asm file for branching instructions.  In this example the DLL under test has three "ble" ("branch on less than or equal") branching instructions.  The mutation analysis then enters a main processing loop.  On each pass through the loop, the original DLL is mutated, creating a new mutant DLL.  The new mutant DLL is built and then the test suite is run against the mutant.  In this example, the first mutant did not produce any test failures.  This is not good and a message warning you that your test suite probably misses the code path containing the mutated code is printed.  The second and third mutants do produce test case failures (which is a good thing) and contribute to the "beta power" of the test suite.  The mutation analysis in the screenshot finishes by printing the relative mutation "alpha power" of 0.67 — which is just the proportion of mutations which produce test case failures.
This entry was posted in Software Test Automation. Bookmark the permalink.

2 Responses to Mutation Testing

  1. Oleg says:

    That\’s a cool technique indeed. But again – for a serious SUT number of potential mutations might be astronimical – how do you select best mutations then?

  2. James says:

    You are absolutely correct — for any real system under test the number of possible mutations is in the thousands or even millions. In situationms like this, the normal approach is to first scan through the IL and determine the number and location of potential mutations. Next, instead of mutating each possibility, you randomly select and mutate from the possibility-list. 

Comments are closed.