Mutation Testing is a technique that software engineers can use to measure the effectiveness of their overall testing effort. Suppose a team of testers has created many individual tests; let’s call the collection of tests the test suite. In mutation testing, the original system/program under test is mutated to create a faulty version called a mutant. The mutant program is tested using the suite of individual test cases. The test suite should produce new test case failures; if no new failures appear then that probably means the test suite does not exercise the code path containing the mutated code, which means the test suite does not fully test the system/program. You can then create new test cases that do exercise the mutant code which may reveal bugs. Notice that the term "mutation testing" is somewhat misleading — in mutation testing you do not actually test the system under test, you measure the effectiveness of a collection of tests.
The screenshot below shows an example of mutation testing in action. I am analyzing a library named LibUnderTest.dll. It is a demo library containing a single Max() method. In the first part of mutation analysis, the DLL under test is tested using an existing test suite. In this case the test suite is named TestHarness.exe. Again, the test suite is just a demo to keep things small and the ideas clear. The test suite has only 10 test cases for simplicity. Notice that all 10 test cases pass. After establishing baseline test results, the DLL under test is examined for potential mutations. I disassemble the DLL under test and examine the resulting .asm file for branching instructions. In this example the DLL under test has three "ble" ("branch on less than or equal") branching instructions. The mutation analysis then enters a main processing loop. On each pass through the loop, the original DLL is mutated, creating a new mutant DLL. The new mutant DLL is built and then the test suite is run against the mutant. In this example, the first mutant did not produce any test failures. This is not good and a message warning you that your test suite probably misses the code path containing the mutated code is printed. The second and third mutants do produce test case failures (which is a good thing) and contribute to the "beta power" of the test suite. The mutation analysis in the screenshot finishes by printing the relative mutation "alpha power" of 0.67 — which is just the proportion of mutations which produce test case failures.