Integrating Custom Test Harnesses with Team Foundation Server

During the Q & A of a technical talk I delivered recently, Howard Dierking, the Editor-in-Chief of MSDN Magazine, (where I write a column on software testing called "Test Run") made a good observation. Namely, one of the disadvantages of custom, lightweight software test automation compared to test frameworks is manageability — because software test automation is now so much easier to write than it was just a few years ago (in particular automation written using PowerShell), you can easily become overwhelmed with test case data files, test harnesses, and test results files. Howard suggested the idea of using Team Foundation Server (TFS) to manage PowerShell-based lightweight custom test automation. So, I’ve been looking at this idea and there’s some good news and some bad news. You can think of Team Foundation Server as a large database that can store source code, perform version control, manage the build process, store bugs and other so-called work items, and store test cases and test results. Various flavors of Visual Studio essentially act as the client software for Team Foundation Server. The Visual Studio Team Test (VSTT) edition has built-in support for unit testing, some types of Web testing, a "generic test", and other test types. You can run tests on a host machine and then use VSTT to "publish" test results to the TFS databases. What I was hoping for was that there would be a simple, standard XML schema format for a VSTT results file. If so, then all you’d have to do to use TFS as a test results data store for custom PowerShell-based test automation would be to emit test results in the proper format and then (somehow) publish to TFS. It turns out that VSTT test results are stored as ".trx" ("Test Results XML") files which are just XML — however there is no published schema for these files — they are generated on the fly by VSTT in a fairly complex way. One way around this situation is to use the VSTT "Generic" test type which is a wrapper around a custom test harness. If you code your test harness to return a errorlevel value of 0 for pass or 1 for fail, you can execute the test from VSTT, which will produce a simple .trx file, which can be published to TFS. You can provide enhanced test results by coding your test automation to store results as XML following a simple SummaryResult.xsd schema (installed with VSTT), and then run your custom automation from a VSTT environment. VSTT will auto-magically grab your results.xml file, build up a results.trx file from the XML file, which you can then publish to TFS. There are some other approaches too. Running custom lightweight test automation from within VSTT has its pros and cons. By the way, because the backend TFS data warehouse is so complex, you really don’t want to write directly to it. Also, VSTT does support a fairly well-documented custom test type that allows you to create, just once, a test type that is essentially a custom PowerShell script. I have to look at this idea further. Finally, Visual Studio 2008 is likely to have a new, significantly streamlined way of publishing test results. (Thanks to Dominic Hopton of the VS team for some good information during a phone call the other day).
This entry was posted in Software Test Automation. Bookmark the permalink.