In Visual Studio Team System 2010 we have made some huge steps forwards in software testing. Specifically we have invested in test management, manual testing, UI automation and lab management. If you would like some background information you can check out the following blogs:
My primary area of focus for this release has been the test management server and its public object model, so I plan to give some focus to that here. The OM we have built is the same layer that drives the new test management client that is shipping in VSTS 2010. So any of the data that you view and manipulate there, you have access to through our public object model.
There are a variety of different directions I could go when trying to give an overview of our API. I could put up some UML diagrams and talk about the major object and what they do, or I could just provide some targeted samples. In most cases people aren’t that interested in the design of the API or even an general overview. What most people want is samples on how to accomplish specific tasks. To that end I plan to provide an indeterminate number of posts that use the API to do specific things that I have been asked about.
How do I get my test results into TFS so I can report on them if I am not using TFS Build and MSTest?
That is one of the most common questions we have gotten sense version 1 of Team System with regards to test results and a public API. Unfortunately, in the first two release there was no good way to do this, your only option was to transform your test results file into our format and then publish with mstest.exe. While this was functional, it was difficult to do and error prone. With VSTS 2010 we have a nice .NET object model that will let you directly manipulate the runs and results on the server and hopefully give you a better shot at integrating our store into your build system.
Creating runs and results
The test management service is installed as part of any TFS 2010 server and generally works in the same way as other TFS services.
The first thing you will need to do is get a instance of our service and then get a reference to the TCM store for your team project
string collectionUrl = "http://chrispat-dev:8080/tfs/DefaultCollection";
string projectName = "chrispat.unit.1";
// get our team foundation connection using the executing user
TeamFoundationServer server = new TeamFoundationServer(collectionUrl);
// get the test management service and the store for our team project
ITestManagementService testManagement = server.GetService<ITestManagementService>();
ITestManagementTeamProject teamProject = testManagement.GetTeamProject(projectName);
Once you have that reference you will need to create a build to publish your results against. This can either be a build record that you simply use as a hook for reporting purposes or you can reference a build that you have actually executed through team build. I have included code in the attached sample on how to create a build definition and a build.
After you have a build you are ready to create a run and add test results to it.
// Create a run to hold our results
ITestRun testRun = teamProject.TestRuns.Create();
testRun.BuildFlavor = "Debug";
testRun.BuildPlatform = "Any CPU";
testRun.BuildNumber = build.BuildNumber;
testRun.BuildUri = build.Uri;
LoadTestResults(testRun,teamProject);
// read an NUnit results file and add the resuts to the run.
private static void LoadTestResults(ITestRun testRun, ITestManagementTeamProject teamProject)
{
XmlDocument results = new XmlDocument();
results.Load("TestResult.xml");
// set some run properties
XmlNode run = results.SelectSingleNode("/test-results");
testRun.Title = run.Attributes["name"].Value;
testRun.DateCompleted = DateTime.Parse(string.Format("{0} {1}", run.Attributes["date"].Value, run.Attributes["time"].Value));
testRun.IsAutomated = true;
testRun.IsBvt = true;
//go ahead and attach the log as well
testRun.Attachments.Add(testRun.CreateAttachment("TestResult.xml"));
XmlNodeList testCases = results.SelectNodes("/test-results/descendant::test-case[ancestor::*[parent::*][count(*)=1]]");
List<ITestCaseResult> tcmResults = new List<ITestCaseResult>();
foreach(XmlNode testCase in testCases)
{
// this is for our execution system, but should not be required, by beta you won't need this line
ITmiTestImplementation implementation = teamProject.CreateTmiTestImplementation(testCase.Attributes["name"].Value, "NUNIT", "", Guid.Empty);
// Add a test to our run, pull the key attributes from the NUnit xml result file
ITestCaseResult result = testRun.AddTest(0, testCase.Attributes["name"].Value,string.Empty, implementation);
result.Duration = TimeSpan.FromSeconds(Convert.ToDouble(testCase.Attributes["time"].Value));
result.Comment = testCase.OuterXml;
result.State = TestResultState.Completed;
// set the outcome on the result
if (testCase.Attributes["executed"].Value == "True")
{
if (testCase.Attributes["success"].Value == "True")
{
result.Outcome = TestOutcome.Passed;
}
else
{
result.Outcome = TestOutcome.Failed;
}
}
else
{
result.Outcome = TestOutcome.NotExecuted;
}
// add the result to our list so we can batch save at the end
tcmResults.Add(result);
}
// save the run
testRun.Save();
// batch save the results to the server
teamProject.TestResults.Save(tcmResults.ToArray(), false);
}
That is pretty much it, not a whole lot of code and fairly straight forward. I have attached a project to this post that contains a working sample that you can use in your own solution.
TCMSamples.zip