Failing Builds based on Code Coverage
The logical next step after you start unit testing your code is to analyse code coverage. You can do this easily in TFS by enabling Code Coverage in the test settings of the default build template. But what about failing builds (or checkins) based on low code coverage?
Out of the box, there’s nothing that can do that. You could write a checkin policy that inspects code coverage, but you’d have to make sure it’s deployed to all VS instances. Or, you could implement the logic in a build, and then make the build a rolling gated checkin build. That way if the build fails for low coverage, the checkin is discarded.
Modifying the Default Template
I’ve created a CodeActivity for your toolbox that will inspect the test coverage and return the coverage percentage. You can then easily implement some logic from there.
To get started, you’ll have to import the Custom assembly (link at the bottom of this post). Follow steps 1 to 3 of my previous custom build post to get going.
Now you can add the ColinsALMCorner.CustomBuildTasks in to the imports of your workflow:
Click on the Arguments tab and add an Int32 in argument called “CoverageFailureIfBelow”. Set the metadata as follows:
Now scroll down in the workflow until after the testing section, between “If CompilationStatus = Uknown” and “If TestingStatus = Uknown”. Add a Sequence between the two If activities called “Check Coverage”.
Click on Variables at the bottom of the workflow window and create an Int32 variable (scoped to Check Coverage, called coverageTotal). You can double-click the sequence to “zoom in”.
The “GetCoverageTotal” activity relies on the test attachments that the test engine uploads to TFS when the tests are completed (most notably the Code Coverage attachment). Since this is done asynchronously, I introduced a Delay activity. Add a new Argument to the workflow (if you don’t want to hard-code the delay) and set its default to 10 seconds.
Set the Timeout of the Delay activity to Timespan.FromSeconds(arg)
where arg is the name of your timeout argument.
Drop a “GetCoverageTotal” activity (from the custom assembly) into the sequence (NOTE: You may have to import the assembly into the Toolbox if you’ve never done that before). Set the properties as follows:
Once that’s done, you now have the coverage total, so you can do whatever you need to. Here’s what I did:
I put an If that evaluates if the total coverage is too low or not. If the coverage is too low, I set the BuildDetail.TestStatus to TestStatus.Failed. I also set the overall build status to PartiallySucceeded. I then have a WriteBuildWarning (“Failing build because coverage is too low”) or WriteBuildMessage (“Coverage is acceptable”) depending on the result.
Now I just queue up a build, and voila – I can now fail it for low coverage (or in this case, set the result to Partially Successful).
You can download the dll and the DefaultCoverageTemplate.11.1.xaml from my skydrive.
Happy coverage!
