TestContext.TestResultsDirectory is the same for all tests

[avivanoff]

In MSTest V1 TestContext.TestResultsDirectory is different for every test. For instance:

TestMethod1 -> …\Deploy_1 2018-10-17 13_00_29\In\b3ce33a5-5c66-422e-960e-3cc97e6f7f13\MYMACHINE
TestMethod2 -> …\Deploy_1 2018-10-17 13_00_29\In\7f4f9ba7-9081-479b-9f49-7d1e020604ff\MYMACHINE

After convertsion to MSTest V2:

TestMethod1 -> …\Deploy_1 2018-10-17 13_00_29\In
TestMethod2 -> …\Deploy_1 2018-10-17 13_00_29\In

Is this a bug or should we update all our tests that rely on V1 behavior to account for the new one?

[vagisha-nidhi]

We have tried reproducing this issue, but weren't able to do so. We are getting only one TestContext.TestResultsDirectory for both MSTest V1 and MSTest V2 projects. Can you share sample repro projects for both MSTest V1 and V2 where you were able to get this behavior?

[avivanoff]

Add .testsettings file to MSTest V1 project.

[avivanoff]

This is what I found in DesktopTestContextImplementation.cs, line 171:

            // In MSTest, it is actually "In\697105f7-004f-42e8-bccf-eb024870d3e9\User1", but
            // we are setting it to "In" only because MSTest does not create this directory.

[vagisha-nidhi]

Hi @avivanoff
I tried with the following code in MSTest v1 adding testsettings file to the project. I am still getting only one folder. The behavior seems to be consistent for both MSTest v1 and v2.

        [TestMethod]
        [DeploymentItem("a.testsettings")]
        public void TestMethod1()
        {
        }

        [TestMethod]
        [DeploymentItem("b.testsettings")]
        public void TestMethod2()
        {
        }

Is there anything else to be added to this? Can you send a repro project?
Thanks.

[avivanoff]

Do not use .testsettings as a deployment item. Select them as Test Settings in Visual Studio when running tests, or as /Settings argument to vstest.console.exe.

[vagisha-nidhi]

I am still not able to repro this. I selected Test Settings file in Visual Studio when running tests. I am unable to get different folders in any case for MSTest V1. Can you please share a small repro project?

[avivanoff]

Sample project.
UnitTest2.zip

[parrainc]

@vagisha-nidhi I was able to replicate this issue as described by @avivanoff.
When running tests with Mstest v1, there's a different test run directory for each test (see example 1).
On the other hand, when running with MsTest v2, there's just one test run directory for every test (see example 2).

Notice that running with both versions (v1 and v2) without the .testsettings file, it produces the same test result directory: ending in \In (no custom directory for each test).

example 1 (with MSTest V1):

namespace LegacyMsTestV1 
{
    [TestClass]
    public class UnitTest1 
    {
        public TestContext TestContext { get; set; }

        [TestMethod]
        public void TestMethod1() 
        {
            // with .testsettings: this outputs: "c:\users\{userid}\...\TestResults\user_MachineName 2018-10-25 06_59_55\In\505149a4-92ae-4469-a0a3-b03948fc9539\MachineName"
            // without .testsettings: outputs same as with MSTest V2
            this.TestContext.WriteLine(this.TestContext.TestResultsDirectory);
        }

        [TestMethod]
	public void TestMethod2()
	{
            // with .testsettings: this outputs: "c:\users\{userid}\...\TestResults\user_MachineName 2018-10-25 06_59_55\In\c35ef1a0-0388-4a15-b093-a94933a21250\MachineName"
            // without .testsettings: outputs same as with MSTest V2
	    this.TestContext.WriteLine(this.TestContext.TestResultsDirectory);
	}
    }
}

example 2 (with MSTest V2):

namespace NewMsTestV2 
{
    [TestClass]
    public class UnitTest1
    {
        public TestContext TestContext { get; set; }

        [TestMethod]
        public void TestMethod1() 
        {
            this.TestContext.WriteLine(this.TestContext.TestResultsDirectory); // this outputs: "c:\users\{userid}\...\TestResults\Deploy_user 2018-10-25 07_07_05\In"
        }

        [TestMethod]
	public void TestMethod2()
	{
	   this.TestContext.WriteLine(this.TestContext.TestResultsDirectory); // this outputs: "c:\users\{userid}\...\TestResults\Deploy_user 2018-10-25 07_07_05\In"
	}
    }
}

[vagisha-nidhi]

Hi @avivanoff. I was able to get the repro here.
We don't have .testsettings file supported witth MSTest v2 and don't plan to do so, hence there can be behavior difference when using testsettings file. The recommended way to do is to use .runsettings file
Also, what is your scenario exactly for using these folder structures so that we can get you unblocked?

@parrainc MSTest v2 doesn't support testsettings so no tests are run in this case.

[parrainc]

@vagisha-nidhi Yup, i know. Just wanted to point out that in MsTest v1 without the .testsettings file it outputs the same results as in MsTest v2.

[avivanoff]

A lot of our tests rely on TestContext.TestResultsDirectory being unique to provide test isolation. And because most of our tests run in parallel, if TestContext.TestResultsDirectory is not unique, we have a lot of race conditions in terms of file access issues.

[vagisha-nidhi]

This is a valid ask but it's not a bug. The framework is designed in such a way to get only one TestContext.TestResultsDirectory for every test.
For running tests in parallel, your tests should be designed such so that no resource race conditions occur while file accesses in these directories.
Tagging @cltshivash and @PBoraMSFT to see if would like to take this up as enhancement.

[avivanoff]

The behavior between V1 and V2 changed. You should at least document it.

And thank you for all that extra work we now have to do.

[jayaranigarg]

@avivanoff : Apologies for the inconvenience but .testsettings are just not supported by MSTestV2. We have this documented here : https://github.com/Microsoft/testfx-docs/blob/master/docs/deltaWithMSTestV1.md

Tests should not even get discovered when you supply testsettings file with a MSTestv2 test project. How are you able to run your tests in parallel? Can you please attach a sample MSTestV2 based test project which you are using to hit this issue?