One software testing concept that beginning software test engineers often don’t fully understand is the difference between testing and analysis. A software test case is an entity (for lack of a better word) which has a pass or a fail result — nothing else. Software test analysis is an activity which results in information about the system under test. Here’s an example of what I mean. Suppose you are testing the performance of some application program which searches through a database store of book title information. You exercise the application under a set of fixed conditions with a particular input which represents part of a title (say, "football") and discover that results (say, the data for 3 different books) are returned in 1.50 seconds. This is analysis, not testing. A test case on the other hand would specify a certain time value that if results are returned in that time or less, then the test case result is a pass, but if results are returned in longer than the specified time value, then the test result is fail.
Analysis and testing are most often confused when looking at SUT performance, as in the example above. However, the principle applies to all types of testing, including functionality testing. For example, suppose you are testing a calculator program. You enter 3 into the calculator and then hit the Square Root key, and 1.73205 is displayed in the answer area. This is analysis. A test case might look like:
0001 3 SqRt 1.73204999
where the four fields represent a test case ID, two test case inputs, and the expected result. If you got 1.73205, then this would be a fail result because the actual result (1.73205) does not exactly match the expected result (1.7320499).
Now obviously, this is a question of semantics. But details are important in software testing. And analysis and testing go hand in hand. For example, especially with performance testing, you must sometimes do some preliminary performance analysis to determine the threshold values for your test cases.
If you enjoyed this blog entry, you might enjoy working at Microsoft. My parent company, Volt Information Sciences, is always looking for software engineers. Check the job listings at http://jobs.volt.com/.