Configuration Testing

Configuration testing is the process of testing a system under development on machines which have various combinations of hardware and software. In many situations the number of possible configurations is far too large to test. For example, suppose you are a member of a test team which working on some desktop user application. The number of combinations of operating system versions, memory sizes, hard drive types, CPUs alone could be enormous. If you target only 10 different operating system versions, 8 different memory sizes, 6 different hard drives, and 7 different CPUs, there are already 10 * 8 * 6 * 7 = 3,360 different hardware configurations. Then if you add various software components such as Web browser version, anti-virus software, earlier versions of the system under development, the number of possible configurations quickly becomes unmanageable. Note that not all software development efforts necessarily face an unmanageable number of configurations. For example, software developed for specific, non-PC hardware devices may have only a handful of hardware and software configurations.

Because the number of possible configurations to test is typically too large to effectively test, it is crucial that the planning effort for a software testing effort clearly identify which platforms will be supported. If, as is often the case, the number of combinations of hardware and software is simply too great to test thoroughly, then the test planning effort must carefully prioritize testing different configurations based on a number of factors such as the size of the user base, and the risk associated with an undiscovered bug in a particular configuration.

Hardware configuration testing is often performed in test labs which simply consist of a large number of machines with different types of hardware. When a testable build of the system under development is released, the build is installed onto each machine in the lab, and a suite of tests is run on each machine. Because there are usually a large number of machines in a test lab, in most situations a significant amount of automation is performed to install the system under test and execute test suites. Setting up a test lab can be expensive so in many situations hardware configuration testing is well suited to outsourcing to organizations which specialize in this type of testing.

Software configuration testing can be very time consuming because of the time required to install and uninstall software including operating system software. For this reason, software configuration testing is also often heavily automated. One configuration testing approach which is not generally possible for hardware testing, but is possible for software testing, is to test on virtual machines. A virtual machine consists of a single file, often called a VHD, or virtual hard drive, which when installed on a host machine can simulate a particular real software configuration. Multiple virtual machines, each with a different software configuration, can be installed and run on a single physical host machine, and tested simultaneously. A software configuration test effort can maintain a library of virtual machines in the form of a set of .vhd files which can be used as needed for testing. There are both commercial virtual machine systems as well as systems which are freely available. On Microsoft platforms, free virtual systems include Virtual PC and the very closely related Virtual Server (the differences between the two are very small and mostly technical in nature), and the 64-bit machine based Hyper-V system. There are several free or open source virtual machine software systems for Unix platforms. One such system is VirtualBox.


This entry was posted in Software Test Automation. Bookmark the permalink.