Changes in TestContext.Properties Flow for ClassInitialize has unintended side effects

[dv-sjun]

Describe the bug

We are using tests with [TestProperty(..)] to modify behaviour in a shared (as in base class) TestCleanup method. When updating to MsTest v4.3 our logic started breaking.

From my testing it seems because when the [ClassInitialize] method is called when executing the test with a [TestProperty(..)], then that value is used "forever" instead of only for that single test.

I think this broke because of #8386

Version used

From a reproduction csproj:

<PackageReference Include="MSTest" Version="4.3.2" />

Steps To Reproduce

I have this simple test class:

namespace Tests
{
    [TestClass]
    public sealed class Test1
    {
        public required TestContext TestContext { get; set; }

        [ClassInitialize]
        public static void ClassInit(TestContext testContext)
        {
            Console.WriteLine("Class Init - ID = " + testContext.GetHashCode() + " - Contains? " + testContext.Properties.ContainsKey("A"));
        }

        [TestMethod]
        [TestProperty("A", "A_Value")]
        public void TestMethod1()
        {
            var containsKey = TestContext.Properties.ContainsKey("A");
            Console.WriteLine("ID = " + TestContext.GetHashCode() + " - Contains? " + containsKey);
            Assert.IsTrue(containsKey);
        }

        [TestMethod]
        public void TestMethod2()
        {
            var containsKey = TestContext.Properties.ContainsKey("A");
            Console.WriteLine("ID = " + TestContext.GetHashCode() + " - Contains? " + containsKey);
            Assert.IsFalse(containsKey);
        }

        [TestMethod]
        public void TestMethod3()
        {
            var containsKey = TestContext.Properties.ContainsKey("A");
            Console.WriteLine("ID = " + TestContext.GetHashCode() + " - Contains? " + containsKey);
            Assert.IsFalse(containsKey);
        }
    }
}

Expected behavior

Every test always runs successfully.

Actual behavior

When using "Run until Failure" in Visual Studio, i get "Test run finished: 3 Tests (1 Passed, 2 Failed, 0 Skipped) run in 333 ms. Finished at iteration 3" with the following output:

 TestMethod1
   Source: Test1.cs line 16

Test has multiple result outcomes
   3 Passed

Results

    Iteration 1)   TestMethod1 
      Duration: 14 ms

      Standard Output: 
ID = 58366981 - Contains? True


    Iteration 2)   TestMethod1 
      Duration: 11 ms

      Standard Output: 
ID = 62476613 - Contains? True


    Iteration 3)   TestMethod1 
      Duration: 21 ms

      Standard Output: 
Class Init - ID = 62476613 - Contains? True
ID = 11404313 - Contains? True

 TestMethod2
   Source: Test1.cs line 25

Test has multiple result outcomes
   2 Passed
   1 Failed

Results

    Iteration 3)   TestMethod2 
      Duration: 53 ms

      Message: 
Assertion failed. Expected condition to be false.

actual: true

Assert.IsFalse(containsKey)

      Stack Trace: 
Test1.TestMethod2() line 30
MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

      Standard Output: 
ID = 39785641 - Contains? True


    Iteration 1)   TestMethod2 
      Duration: 14 ms

      Standard Output: 
Class Init - ID = 12547953 - Contains? False
ID = 11429296 - Contains? False


    Iteration 2)   TestMethod2 
      Duration: 11 ms

      Standard Output: 
Class Init - ID = 51781231 - Contains? False
ID = 65849037 - Contains? False

 TestMethod3
   Source: Test1.cs line 33

Test has multiple result outcomes
   2 Passed
   1 Failed

Results

    Iteration 3)   TestMethod3 
      Duration: 53 ms

      Message: 
Assertion failed. Expected condition to be false.

actual: true

Assert.IsFalse(containsKey)

      Stack Trace: 
Test1.TestMethod3() line 38
MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

      Standard Output: 
ID = 58366981 - Contains? True


    Iteration 1)   TestMethod3 
      Duration: 14 ms

      Standard Output: 
ID = 39785641 - Contains? False


    Iteration 2)   TestMethod3 
      Duration: 11 ms

      Standard Output: 
ID = 12547953 - Contains? False

It seems that as soon as the class init ist called when running TestMethod1, then it leaks into the other tests.

[Evangelink]

Hello @dv-sjun,

I can reproduce and that's indeed unexpected behavior. I'll investigate and fix that issue.