Migrating to NUnit

[Roemer]

Hi
Currently the project uses XUnit. Fair enough but I think it makes it more complex than it needs to be for this project. Especially when using the TestExplorer you cannot jup directly to a test, you only jump to the base "Fact". I know this is only the current implementation but to change this, all the unittest bases need to be rewritten and why not migrate to NUnit to make it easier in general?
There's also another issue with XUnit, I very often run into the problem with unloading appdomains:

System.CannotUnloadAppDomainException: Error while unloading appdomain. (Exception from HRESULT: 0x80131015)
   at System.AppDomain.Unload(AppDomain domain)
   at Xunit.ExecutorWrapper.Dispose()
   at Xunit.MultiAssemblyTestEnvironment.Unload(TestAssembly assembly)
   at Xunit.ConsoleClient.Program.RunProject(XunitProject project, Boolean teamcity, Boolean appveyor, Boolean silent)
   at Xunit.ConsoleClient.Program.Main(String[] args)

I'm currently unsure how to fix this (as far as I know, White does not keep any Threads open, which is usually the case for this).

So what's your oppinion on this? Is a migration to NUnit an option?

[dpisanu]

I would second the move to NUnit. Especially if there is going to be a change in maintainers.

[Roemer]

So it's 2:0 for NUnit for now. I'll already start with it and prepare some things.

[maxinfet]

Yeah that would make getting into this project a lot simpler. I would be really happy to see this change.

[Roemer]

I just did some tests with ButtonTest class, here's my proposal for it:

[TestFixture(WindowsFramework.WinForms)]
[TestFixture(WindowsFramework.Wpf)]
public class ButtonTests: WhiteUITestBase
{
    public ButtonTests(WindowsFramework framework)
        : base(framework) { }

    [Test]
    public void Click()
    {
        RunTestTakingFailingScreenshot(() =>
        {
            var button = MainWindow.Get<Button>("ButtonWithTooltip");
            button.Click();
            Assert.That(button.Text, Is.EqualTo("Button Clicked with Mouse"));
        });
    }

    [Test]
    public void ThrowsWhenNotFound()
    {
        RunTestTakingFailingScreenshot(() =>
        {
            using (CoreAppXmlConfiguration.Instance.ApplyTemporarySetting(c => c.FindWindowTimeout = 500))
            {
                Assert.That(() => { MainWindow.Get<Button>(SearchCriteria.ByAutomationId("foo")); },
                    Throws.TypeOf<AutomationException>().With.
                    Message.EqualTo("Failed to get (ControlType=button or ControlType=check box),AutomationId=foo"));

            }
        });
    }

    [Test]
    public void RaiseClickEvent()
    {
        RunTestTakingFailingScreenshot(() =>
        {
            var button = MainWindow.Get<Button>("ButtonWithTooltip");
            button.RaiseClickEvent();
            Assert.That(button.Text, Is.EqualTo("Clicked"));
        });
    }
}

The abstract base class (WhiteUITestBase) is quite similar to the current one, it opens the app on OneTimeSetup and Disposes it at the end. The RunTestTakingFailingScreenshot can be used to collect a screenshot when the test-action fails.
The Arguments in the TestFixture allow to run all tests for different frameworks.
This also looks good in Visual Studio Test Explorer, see:

Any feedbach on this?

[maxinfet]

I think that looks pretty good, particularly the resulting changes to the contents of the test explorer. I don't like the duplication on the RunTestTakingFailingScreenshot but that's pretty minor.

[dpisanu]

Any chance to make a seperate branch so we can work on parallel on it?

[Roemer]

Any chance to make a seperate branch so we can work on parallel on it?

That is the plan yes, but I'd like to have some feedback from the "old" crew first.

[Roemer]

I don't like the duplication on the RunTestTakingFailingScreenshot but that's pretty minor.

We can get rid of that :) By using the TestContext and the TearDown attribute.
Looks like this:

[TearDown]
public void TestTeardown()
{
    if (TestContext.CurrentContext.Result.Outcome.Status == TestStatus.Failed)
    {
        TakeScreenshot(TestContext.CurrentContext.Test.FullName);
    }
}

Now the only thing unsecured is if the test runs into an unexpected exception, but I think that should be handled manually by the writer of the test.

[maxinfet]

@Roemer Yeah that looks good to me. I agree with the manual handling by the author.

[Roemer]

I also modified the ControlTypeProperty.cs DisplayValue from LocalizedControlType to ProgrammaticName.Split(new[] { '.' }, 2).Last(). This makes the error texts always in english and therefore easier to test in other languages. The downside is, that this is a breaking change (button is now Button, check box is now CheckBox). As alternative, we could dynamically use the strings from for example ControlType.Button.LocalizedControlType. Not sure what's better.

[Roemer]

I guess I go with the 2nd approach. Tests then should look like this:

Assert.That(() => { MainWindow.Get<Button>(SearchCriteria.ByAutomationId("foo")); },
    Throws.TypeOf<AutomationException>().With.
    Message.EqualTo(String.Format("Failed to get (ControlType={0} or ControlType={1}),AutomationId=foo",
        ControlType.Button.LocalizedControlType, ControlType.CheckBox.LocalizedControlType)));

[Roemer]

Just to let you know, I just startet a new nunit branch: https://github.com/TestStack/White/tree/nunit
The TestStack.White.UnitTests are fairly straight forward to migrate, for the TestStack.White.UITests, I created the WhiteUITestBase and migrated the ControlTests/ButtonTests.cs. Feel free to continue migration!

[maxinfet]

Awesome, I will be taking a look at that as soon as I get home

[Roemer]

I hope we can get this thru quickly, then we can enable the appveyor build and have a fully functional CI :)

[Roemer]

I just migrated some more. This should give you a start to continue. I'll also continue later.