Automatic Test Case Generation

One of my recent articles in Microsoft’s MSDN Magazine describes how to write powerful test automation for SQL stored procedures using LINQ technology ( see ). In that article I write, "Determining expected state is the most time-consuming task when testing stored procedures. You must manually determine the expected state of your test bed database after a stored procedure has been called." This weekend an MSDN reader sent me a question about this statement and one which I receive often. The reader’s question essentially asked, "Creating test case data is very painful. Is there any way to automate the creation of test case data?" I wrote back to the reader that he had essentially asked one of the big questions in software testing, and that the short answer is that there is currently no good way to automate the generation of test case data. Basically, any test case generation system needs to know what valid inputs are to a particular state, and what the corresponding outputs and expected states are. One approach is to use Model Driven/Based Development where you use a modeling language such as UML to define a software system, and then use tools to automatically generate source code and test case data. The problem with MDD/MBD is that it only works for relatively simple application programs (as opposed to system software) and costs significantly more than traditional techniques. Another approach to automatic test case generation is to formalize the specifications of your software system — defining inputs and outputs in a very rigorous way. Microsoft Research has been working on an extension to the C# language called Spec# which takes this approach. It is not currently usable in a production environment but I believe Spec# is a good approach in general and is the way in which automatic test case generation will eventually become a practical reality. A third approach to automatic test case generation is Model Driven Testing — you create a simplified model of the actual system being developed and then test the model instead of the actual system. This is truly a wacky idea that just doesn’t work in practice. So, for the next few years at least, test case generation is a time-consuming activity where a tester’s intuition and experience play a big role.
This entry was posted in Software Test Automation. Bookmark the permalink.