[wasm] runtime tests timing out on `Build Tests` step for rolling builds

[radical]

Runtime tests job has been timing out on rolling builds, specifically on the Build Tests step. The last passing rolling build was https://dev.azure.com/dnceng-public/public/_build/results?buildId=428248 , and the first failing one https://dev.azure.com/dnceng-public/public/_build/results?buildId=429084 .

The changes between them 52aca8c...9dfbb61 .

There were 3 PRs with changes in src/tests:

• [Mono] Disabled failing tests on Mono Android and interpreter lanes to keep CI green #92728
• Merge tests under tracing / Reflection / ReadyToRun / Regressions #92029
• Fix conflicting test assembly name in classloader tests #93012

cc @fanyang-mono @trylek @akoeplinger

Known Issue Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Runtime tests job has been timing out on rolling builds, specifically on the Build Tests step. The last passing rolling build was https://dev.azure.com/dnceng-public/public/_build/results?buildId=428248 , and the first failing one https://dev.azure.com/dnceng-public/public/_build/results?buildId=429084 .

The changes between them 52aca8c...9dfbb61 .

There were 3 PRs with changes in src/tests:

• [Mono] Disabled failing tests on Mono Android and interpreter lanes to keep CI green #92728
• Merge tests under tracing / Reflection / ReadyToRun / Regressions #92029
• Fix conflicting test assembly name in classloader tests #93012

cc @fanyang-mono @trylek @akoeplinger

Author:	radical
Assignees:	-
Labels:	arch-wasm, blocking-clean-ci
Milestone:	-

[radical]

@trylek The runtime tests job on your PR #92029 timed out too, but the PR was merged before the job was completed. Could you please take a look at this?

[akoeplinger]

An important thing to notice here is this message which gets shown in the root log:

##[warning]Agent  was purged, cancelling the pipeline.
,##[warning]Received request to deprovision: The request was cancelled by the remote provider.

As far as I know this means the AzDO agent was killed somehow. Maybe we're hitting OOM with the AOT compiler due to the increased number of tests in the merged tests setup? But it's weird given this is a monointerpreter job so I assume it should do little AOTing.

[radical]

@lewing we might be hitting container limits for runtime tests again. These have been failing since Oct 5, so we are not running any runtime tests.

[radical]

An important thing to notice here is this message which gets shown in the root log:

##[warning]Agent  was purged, cancelling the pipeline.
,##[warning]Received request to deprovision: The request was cancelled by the remote provider.

As far as I know this means the AzDO agent was killed somehow. Maybe we're hitting OOM with the AOT compiler due to the increased number of tests in the merged tests setup? But it's weird given this is a monointerpreter job so I assume it should do little AOTing.

Actually, if it was hitting container limits then it should have failed much sooner, but the job failed/got-cancelled after 4h 1m 9s!

[radical]

@akoeplinger is there any way to get the console log from that, even if it might be truncated near the end?

[akoeplinger]

@radical I don't think so, it looks like AzDO doesn't preserve the log in this case...

but the job failed/got-cancelled after 4h 1m 9s!

It is wildly different between runs, I've also seen it after 1h50m.

[radical]

@trylek Could you please take a look at this? I think #92029 might be the one causing this. This is happening consistently since Oct 5.
cc @lewing

Meanwhile, I'm trying a revert - #93362 .