The Visual Studio Team System “off-road” code coverage experience



Off-road – eh?

The Code Coverage collection and analysis portions of Team System, which my team is responsible for, had a particular design philosophy in mind for this version. That philosophy was to have a seamless integrated experience when working with the unit testing framework in Team System.  That’s not to say you can’t use code coverage in whichever framework you deem necessary for your own purposes.  This post shows the fundamentals of collecting coverage information and analyzing it in the Beta 2 bits of Team System. 

 

Before going down this road, you should note that it would be a far better investment of your time is to integrate with the Team System testing framework.  It’s designed with extensibility in mind and you’ll get a host of “free” test features, such as test selection and reporting.

All Wheel Drive please!

In our code coverage implementation there are three primary steps:

 

1)      Instrumentation of the assemblies/binaries you want to collect code coverage from

2)      Collecting that information while exercising your code

3)      Analyzing that information afterwards

 

To instrument the exe/dll/assembly (I say that since .Net, mixed and native executable types are supported) we use the vsinstr.exe tool that comes with VSTS.  You’ll find this tool in the following directory:

Microsoft Visual Studio 8\Team Tools\Performance Tools

 

Instrumenting an exe/dll/assembly for coverage is as simple an operation as:

vsintr -coverage myassembly.exe

 

Now that assembly has been backed-up, and myassembly.exe has some instrumentation code in it to allow coverage information to be collected.  Before we go and exercise this code, we need to make sure the coverage monitor is running and ready to put results into a file.  We do this like so:

start vsperfmon -coverage -output:mytestrun.coverage

 

You’ll now see that you have a command shell running the monitor, basically sitting there doing nothing.  Now go ahead and exercise that code.  Give it a good run around.  Go for the burn.  When you feel that you’ve given it the once-over, you need to tell the monitor that you are done for the day:

vsperfcmd -shutdown

 

This command will wait until your process (the one exercising your code) has exited.

 

And there you have it - a lovely .coverage file has been produced to give you an idea of how well you exercised that code.  To load that file into VS for visualization, simply open the file mentioned in the “start vsperfmon” line above via the File->Open->File menu in VS.  The code coverage results window will appear giving you a breakdown of your coverage.  <insert favoured brass flourish>

Pimp my Ride!

I hear you cry, “JoC man, I need tighter control, I need all the gauges; give me the raw horsepower!”  Well, yes, command line control may not cut it for whatever integration or harness you want to tie this up to.  Let me talk a little about some of the assemblies that are shipped to add code coverage functionality that can be used to tie into your own mechanisms.  Please note that assemblies and interfaces are the Beta2 versions and may change before the final release of Visual Studio Team System.

 

First off, there is the controlling mechanism available in:

Microsoft.VisualStudio.Coverage.Monitor.dll

 

This assembly allows all of the mechanisms described in the above instrumentation and collection phases.  A small example piece of code that drives this like the example above would be:

using System;

using System.Collections.Generic;

using System.Text;

using System.Diagnostics;

using Microsoft.VisualStudio.CodeCoverage;

 

// You must add a reference to Microsoft.VisualStudio.Coverage.Monitor.dll

 

namespace CoverControl

{

       class CoverProgram

       {

              static void Main(string[] args)

              {

                     // Write something to call vsinstr.exe using

                     Process p = new Process();

                     StringBuilder sb = new StringBuilder("/COVERAGE ");

                     sb.Append("myassembly.exe");

                     p.StartInfo.FileName = "vsinstr.exe";

                     p.StartInfo.Arguments = sb.ToString();

                     p.Start();

                     p.WaitForExit();

                     // TODO: Look at return code – 0 for success

 

                     // A guid is used to keep track of the run

                     Guid myrunguid = Guid.NewGuid();

                     Monitor m = new Monitor();

                     m.StartRunCoverage(myrunguid, "mytestrun.coverage");

 

                     // TODO: Launch some tests or something

                     // that can exercise myassembly.exe

 

                     // Complete the run

                     m.FinishRunCoverage(myrunguid);

              }

       }

}

 

 

Remember that this is how the API looks in Beta 2.  If it changes in the final shipping version, I’ll announce it here so you have the heads-up, but sometimes we have to do that to fix bugs and issues.  Obviously, people can extend that to whatever mechanisms they want, including hooking it up to some GUI for control or a script/test harness that they rolled themselves.

 

Next up in the “raw horsepower” assemblies is:

Microsoft.VisualStudio.Coverage.Analysis.dll

 

This assembly provides the raw smarts under the code coverage visualization hood in VSTS.  Here’s how you can dump some XML for the covered lines in a coverage file collected above:

 

using System;

using System.Collections.Generic;

using System.Text;

using Microsoft.VisualStudio.CodeCoverage;

 

// You must add a reference to Microsoft.VisualStudio.Coverage.Analysis.dll

 

namespace CoverDump

{

       class DumpProgram

       {

              static void Main(string[] args)

              {

                     // Create a coverage info object from the file

                     CoverageInfo ci = CoverageInfoManager.CreateInfoFromFile("myfile.coverage");

 

                     // Ask for the DataSet.  The parameter must be null

                     CoverageDS data = ci.BuildDataSet(null);

 

                     // Write to XML

                     data.Lines.WriteXml("mylines.xml");

              }

       }

}

 

The XSD for the coverage dataset schema may also change slightly between now and the final shipped product.  You can now operate with the DataSets we provide programmatically to do your own filtering or roll-ups, or you can flush the XML and work with it through some XSLTs to present and roll-up as needed.  Team System will feature some aggregated reports for coverage, so you may find that any work you do here will be covered by what Team System does for you.

Destination reached

Hopefully, this has given people who want to go off-road with Code Coverage collection and analysis the guide to go do so, but I do want to re-emphasize what I said earlier in this post.  For most folks the right way to get what you want is to integrate into the test framework in Team System.  You get a lot more “free” features when doing this, such as reporting, run configurations, etc. To get more details on doing that, look at the white paper and samples included with the Team System extensibility SDK.

 

To quickly look at some code coverage on an ad-hoc basis, or where you just need to tie up to something else, off-roading can be fun however…

转载自http://blogs.msdn.com/b/ms_joc/archive/2005/04/08/406608.aspx

其它CodeCoverage相关参考Peter Huene's Blog http://blogs.msdn.com/b/phuene/