Bug: Could not find test executor with URI 'executor://mstestadapter/v2' when using vstest with code coverage

[test-in-prod]

Description

When using /Collect:"Code Coverage" arg with vstest.console.exe on a build server (i.e. TestAgent or Community), we get a weird error (exactly):

Could not find test executor with URI 'executor://mstestadapter/v2'. Make sure that the test executor is installed and supports .net runtime version .

If we omit /Collect:"Code Coverage" the tests are discovered and ran, but we do not get a code coverage report. The unit test project is a .NET Core targeting win7-x64, Framework 4.6. We tried downgrading MSTest.TestAdapter / MSTest.TestFramework versions (from 1.2 to 1.1.17/14/whatever) to no avail.

Running with /Diag produced this interesting entry in the host.log:

TpTrace Warning: 0 : 2908, 7, 2017/12/20, 15:24:04.736, 272393501515, testhost.exe, TestPluginDiscoverer: Failed to load extensions from file 'C:\Jenkins\auth-services\ws@unit-testing-47\src\SPC.Services.Auth.Roles.Tests.Unit\bin\Debug\net462\win7-x64\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.dll'.  Skipping test extension scan for this file.  Error: System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter' or one of its dependencies. The system cannot find the file specified.
File name: 'Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter'
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.InternalLoadAssemblyName(AssemblyName assemblyRef, Evidence assemblySecurity, RuntimeAssembly reqAssembly, StackCrawlMark& stackMark, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginDiscoverer.GetTestExtensionsFromFiles[TPluginInfo,TExtension](String[] files, Dictionary`2 pluginInfos)

This DLL is 100% present in the directory claimed at that path, so what gives??

Steps to reproduce

1. Create a unit test project
2. csproj should like like so:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net462</TargetFramework>
    <RuntimeIdentifier>win7-x64</RuntimeIdentifier>
    <IsPackable>false</IsPackable>
	<DebugType>Full</DebugType>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.3.0" />
    <PackageReference Include="Microsoft.VisualStudio.TestPlatform.ObjectModel" Version="14.0.0" />
    <PackageReference Include="MSTest.TestAdapter" Version="1.1.14" />
    <PackageReference Include="MSTest.TestFramework" Version="1.1.14" />
	<PackageReference Include="Microsoft.CodeCoverage" Version="1.0.3" />
  </ItemGroup>
</Project>

3. Run vstest.console.exe mytest.dll /Logger:trx /Platform:x64 /inIsolation (successful)
4. Run `vstest.console.exe mytest.dll /Logger:trx /Platform:x64 /inIsolation /Collect:"Code Coverage" (fail)

Could not find test executor with URI 'executor://mstestadapter/v2'.  Make sure that the test executor is installed and supports .net runtime version  .
No test is available in C:\Jenkins\auth-services\ws@unit-testing-47\src\SPC.Services.Auth.Roles.Tests.Unit\bin\Debug\net462\win7-x64\SPC.Services.Auth.Roles.Tests.Unit.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try aga
in.

Expected behavior

Unit tests run and a TRX + .coverage files produced

Actual behavior

Unit tests are not discovered/ran, no coverage is produced.

[jayaranigarg]

@CryptonZylog : Can you try updating "Microsoft.NET.Test.Sdk" nuget package to latest and see if it works for you?
Also, can you tell us which VS version are you using? You can find out the version in VS IDE by going to Help->About Microsoft Visual Studio. (Top Left- 2nd line should have a version)

[test-in-prod]

I have updated the package, however the problem remains the same.

We're on v15.5.2.

Thanks.

[jayaranigarg]

@CryptonZylog : Can you please collect fusion logs for the CLI command and share them with us?
This is how you can collect those assembly binding logs.

[test-in-prod]

VsTestDebug.zip

Please see the logs attached.

Thanks!

[jayaranigarg]

@CryptonZylog : Thank you for sharing logs with us.
We see that you are trying to generate code coverage using VS Community, however code coverage is an enterprise feature and is not available in Community/Professional SKUs.
Can you please try the above scenario on a VS enterprise machine and see if it works for you?

[test-in-prod]

I see. We have Community installed on this build server and aren't really keen on installing Enterprise. Is there another way around this problem? In the end, it may be helpful to also have a more meaningful error message.

Thanks!

[jayaranigarg]

@CryptonZylog : There is no workaround. You will simply need VS enterprise for code coverage. We will consider printing a more meaningful error message for such cases. Thank you for your feedback!

[rajeshgupta723]

following these steps helped resolved this error for me:
• Run Visual Studio as Administrator.
• Make sure Service fabric project was published as X64 based architecture.
• Make sure Test Settings is pointing to X64 Architecture, and correct .runsettings file was selected.