Reduce Apple mobile CI matrix by kotlarmilos · Pull Request #125441 · dotnet/runtime

kotlarmilos · 2026-03-11T13:52:24Z

Description

Reduce the Apple mobile CI test matrix to mitigate macOS queue congestion and align with the default scenarios.

What runs where

`runtime.yml` — ios + maccatalyst arm64 smoke (6 jobs)

Platform	Runtime	Test type	Config
ios_arm64	Mono	Library smoke	Release
ios_arm64	NativeAOT	Library smoke	Release
ios_arm64	CoreCLR	Library smoke	Release
maccatalyst_arm64	Mono	Library smoke	Release
maccatalyst_arm64	NativeAOT	Library smoke	Release
maccatalyst_arm64	CoreCLR	Library smoke	Release

`runtime-extra-platforms-ioslike.yml` — tvos full + Debug smoke (6 jobs)

Platform	Runtime	Test type	Config
tvos_arm64	Mono	Library tests	Release
tvos_arm64	Mono	Runtime tests	Release
tvos_arm64	NativeAOT	Library smoke	Release
tvos_arm64	CoreCLR	Library tests	Release
tvos_arm64	CoreCLR	Runtime tests	Release
tvos_arm64	CoreCLR	Library smoke	Debug

`runtime-extra-platforms-ioslikesimulator.yml` — iossimulator Release + Debug smoke (9 jobs)

Platform	Runtime	Test type	Config
iossimulator_x64, iossimulator_arm64	Mono	Library tests	Release
iossimulator_arm64	Mono	Runtime tests	Release
iossimulator_x64, iossimulator_arm64	NativeAOT	Library smoke	Release
iossimulator_x64, iossimulator_arm64	CoreCLR	Library tests	Release
iossimulator_arm64	CoreCLR	Runtime tests	Release
iossimulator_arm64	CoreCLR	Library smoke	Debug

`runtime-extra-platforms-maccatalyst.yml` — maccatalyst full + Debug smoke (12 jobs)

Platform	Runtime	Test type	Config
maccatalyst_x64, maccatalyst_arm64	Mono	Library tests	Release
maccatalyst_arm64	Mono	Runtime tests	Release
maccatalyst_arm64	Mono	Library smoke (AppSandbox)	Release
maccatalyst_x64, maccatalyst_arm64	NativeAOT	Library smoke	Release
maccatalyst_arm64	NativeAOT	Library smoke (AppSandbox)	Release
maccatalyst_x64, maccatalyst_arm64	CoreCLR	Library tests	Release
maccatalyst_arm64	CoreCLR	Runtime tests	Release
maccatalyst_arm64	CoreCLR	Library smoke (AppSandbox)	Release
maccatalyst_arm64	CoreCLR	Library smoke	Debug

Other changes

Re-enabled 3 tvOS test suites that were previously excluded due to mysterious crashes (System.IO.Hashing.Tests, System.IO.MemoryMappedFiles.Tests, System.Runtime.Numerics.Tests)
Entries reordered consistently: Mono → NativeAOT → CoreCLR, library → runtime within each group

Copilot

Pull request overview

Reduces macOS CI load by shrinking the Apple mobile test matrix (iOS/tvOS simulators/devices and MacCatalyst), aiming to keep coverage focused on common developer/user scenarios while removing redundant jobs.

Changes:

runtime.yml: drop tvOS device from mobile smoke jobs and remove the MacCatalyst Mono smoke entry.
runtime-extra-platforms-ioslike.yml: shift iOS-like device coverage to tvOS-only.
runtime-extra-platforms-ioslikesimulator.yml: focus simulator coverage on iOS simulator only, switch to Debug, and remove NativeAOT simulator runtime tests.
runtime-extra-platforms-maccatalyst.yml: remove MacCatalyst x64 coverage (arm64 only).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.

File	Description
eng/pipelines/runtime.yml	Removes tvOS device and MacCatalyst smoke jobs from the default runtime pipeline’s Apple-mobile matrix.
eng/pipelines/extra-platforms/runtime-extra-platforms-ioslike.yml	Removes iOS device runs so iOS-like device testing is tvOS-only in extra-platforms.
eng/pipelines/extra-platforms/runtime-extra-platforms-ioslikesimulator.yml	Narrows simulator jobs to iOS simulator, switches to Debug, and drops NativeAOT simulator runtime tests.
eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml	Drops MacCatalyst x64 from extra-platforms, leaving arm64 only.

You can also share your feedback on Copilot code review. Take the survey.

eng/pipelines/extra-platforms/runtime-extra-platforms-ioslikesimulator.yml

eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml

eng/pipelines/extra-platforms/runtime-extra-platforms-ioslike.yml

eng/pipelines/runtime.yml

vitek-karas · 2026-03-11T14:04:28Z

I didn't look at the existing coverage, so maybe this is already covered:

I think we should have at least some basic testing of Release builds running on simulators (simulator is different enough platform from iOS, such that it warrants at least basic coverage)
Similarly we should have at least basic coverage for running Debug builds on real devices (while not as common, it IS a scenario for users) - this should include both iOS and tvOS devices.
macCatalyst - we should have some coverage for both Debug and Release (since both are scenarios for that target)

Reduce the Apple mobile test matrix from 47 to 24 jobs (-23 per run), restructured around real user scenarios: - runtime.yml: iOS-only smoke tests (8 -> 3 jobs) Remove tvos_arm64 from smoke tests, remove maccatalyst Mono_Smoke - ioslike.yml: tvOS devices only in Release (12 -> 6 jobs) Remove ios_arm64 (covered by runtime.yml smoke) - ioslikesimulator.yml: iossimulator only in Debug (15 -> 8 jobs) Remove tvossimulator, change Release to Debug, drop NativeAOT - maccatalyst.yml: arm64 only (12 -> 7 jobs) Remove maccatalyst_x64 (Intel Macs EOL) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

eng/pipelines/extra-platforms/runtime-extra-platforms-ioslike.yml

eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml

eng/pipelines/runtime.yml

…urations for clarity

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated no new comments.

You can also share your feedback on Copilot code review. Take the survey.

… exclusions from tests

…ous crashes on tvOS

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

You can also share your feedback on Copilot code review. Take the survey.

eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml

src/libraries/tests.proj

- Restore MacCatalyst Mono + CoreCLR + NativeAOT smoke in runtime.yml - Add NativeAOT library smoke to ioslikesimulator and maccatalyst - Replace NativeAOT runtime tests with library tests in maccatalyst - Add CoreCLR AppSandbox library tests for maccatalyst - Change maccatalyst CoreCLR x64+arm64 from smoke to full library tests - Use x64 only for library tests in ioslikesimulator - Reorder entries: Mono > NativeAOT > CoreCLR, library > runtime Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…uce-apple-ci-matrix # Conflicts: # eng/pipelines/extra-platforms/runtime-extra-platforms-ioslike.yml # eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml

kotlarmilos · 2026-03-25T09:18:29Z

I agree that it overlaps with devices but I considered iOS simulator a different platform (more different than Android devices and emulators) so they deserve to be tested.

Ok, sounds good. Reverted to test Release.

One interesting question could be: historically, has either of the configurations found issues that didn't show in the other one?

Yes, sometimes a debug assert is hit. For that reason, it introduces just one job for the default config per platform to run in Debug to catch such errors.

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

src/libraries/System.IO.MemoryMappedFiles/tests/MemoryMappedFile.CreateFromFile.Tests.cs

Add maccatalyst_arm64 smoke tests for all three runtimes to the default pipeline to catch PR breakages early. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

kotlarmilos · 2026-03-25T10:08:33Z

Impact of this PR:

Complete testing matrix (CoreCLR, Native AOT, Mono)
Number of jobs in PR pipeline didn't change
Number of jobs in extra-platforms reduced by 12 per run
Reduced test coverage on x64 simulators and x64 MacCatalyst to library tests only
Removed test coverage on tvossimulators - considered redundant with iossimulators

dotnet-policy-service · 2026-03-25T10:09:52Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

matouskozak

LGTM! Thanks a lot. Let's monitor for a few days the times of the iOS device runs from runtime pipeline and see if they are able to handle the load well.

matouskozak · 2026-03-25T14:17:32Z

eng/pipelines/extra-platforms/runtime-extra-platforms-ioslike.yml

      testGroup: innerloop
      nameSuffix: AllSubsets_NativeAOT_Smoke
      buildArgs: --cross -s clr.alljits+clr.tools+clr.nativeaotruntime+clr.nativeaotlibs+libs+libs.tests -c $(_BuildConfig) /p:ArchiveTests=true /p:RunSmokeTestsOnly=true /p:DevTeamProvisioning=- /p:BuildTestsOnHelix=true /p:UseNativeAOTRuntime=true /p:RunAOTCompilation=false /p:ContinuousIntegrationBuild=true
      timeoutInMinutes: 180


The NativeAOT jobs outside of runtime.yml do not set useNativeAOTRuntime: true. Just a nit I don't think it's doing anything special.

steveisok

I'm supportive. I think devices are still going to have a hard time when they get into a state of needing to be rebooted.

kotlarmilos · 2026-03-27T13:03:26Z

/azp run runtime-ioslike,runtime-ioslikesimulator,runtime-maccatalyst

azure-pipelines · 2026-03-27T13:03:54Z

Azure Pipelines successfully started running 3 pipeline(s).

Copilot AI review requested due to automatic review settings March 11, 2026 13:52

dotnet-policy-service bot assigned kotlarmilos Mar 11, 2026

github-actions bot added the area-Infrastructure-mono label Mar 11, 2026

Copilot started reviewing on behalf of kotlarmilos March 11, 2026 13:53 View session

kotlarmilos requested review from akoeplinger, matouskozak and vitek-karas March 11, 2026 13:54

kotlarmilos changed the title ~~Reduce Apple mobile CI matrix to alleviate macOS queue congestion~~ Reduce Apple mobile CI matrix Mar 11, 2026

Copilot AI reviewed Mar 11, 2026

View reviewed changes

kotlarmilos force-pushed the reduce-apple-ci-matrix branch from 88bd5b0 to ce99125 Compare March 11, 2026 14:48

This was referenced Mar 11, 2026

Cannot find 'arm64-v8a' device dotnet/dnceng#2284

Open

[wasm] OpenQA.Selenium.WebDriverTimeoutException: timeout: Timed out receiving message from renderer #117486

Open

XHarness DevToolsActivePort file doesn't exist #124438

Closed

matouskozak reviewed Mar 12, 2026

View reviewed changes

eng/pipelines/extra-platforms/runtime-extra-platforms-ioslike.yml Show resolved Hide resolved

eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml Show resolved Hide resolved

eng/pipelines/runtime.yml Show resolved Hide resolved

Rename build parameters for iOS/tvOS simulator and MacCatalyst config…

25affcc

…urations for clarity

Copilot AI review requested due to automatic review settings March 12, 2026 10:01

Copilot started reviewing on behalf of kotlarmilos March 12, 2026 10:03 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

kotlarmilos added 2 commits March 12, 2026 11:15

Refactor MacCatalyst CI configurations and remove unnecessary project…

f0439a4

… exclusions from tests

Remove project exclusion for System.Formats.Cbor.Tests due to mysteri…

0bfee88

…ous crashes on tvOS

Copilot AI review requested due to automatic review settings March 12, 2026 10:16

Copilot started reviewing on behalf of kotlarmilos March 12, 2026 10:17 View session

Copilot AI reviewed Mar 12, 2026

View reviewed changes

eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml Outdated Show resolved Hide resolved

src/libraries/tests.proj Show resolved Hide resolved

kotlarmilos and others added 4 commits March 12, 2026 11:24

Merge branch 'main' into reduce-apple-ci-matrix

8607492

Add maccatalyst_x64 platform to MacCatalyst CI jobs

31bac4d

Merge remote-tracking branch 'origin/reduce-apple-ci-matrix' into red…

bedaf5d

…uce-apple-ci-matrix # Conflicts: # eng/pipelines/extra-platforms/runtime-extra-platforms-ioslike.yml # eng/pipelines/extra-platforms/runtime-extra-platforms-maccatalyst.yml

Copilot AI review requested due to automatic review settings March 12, 2026 11:07

Copilot AI review requested due to automatic review settings March 25, 2026 09:13

Copilot started reviewing on behalf of kotlarmilos March 25, 2026 09:15 View session

Copilot AI reviewed Mar 25, 2026

View reviewed changes

src/libraries/System.IO.MemoryMappedFiles/tests/MemoryMappedFile.CreateFromFile.Tests.cs Show resolved Hide resolved

kotlarmilos and others added 2 commits March 25, 2026 10:26

runtime.yml: Add maccatalyst arm64 Mono+NativeAOT+CoreCLR smoke

955f471

Add maccatalyst_arm64 smoke tests for all three runtimes to the default pipeline to catch PR breakages early. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Restore descriptive comments above jobs in extra-platforms yml

a9deec1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings March 25, 2026 09:33

kotlarmilos marked this pull request as ready for review March 25, 2026 09:33

Copilot started reviewing on behalf of kotlarmilos March 25, 2026 09:34 View session

runtime.yml: Remove tvOS device jobs

ebb69cb

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI reviewed Mar 25, 2026

View reviewed changes

kotlarmilos requested a review from matouskozak March 25, 2026 10:08

kotlarmilos added area-Infrastructure os-ios Apple iOS and removed area-Infrastructure-mono labels Mar 25, 2026

kotlarmilos added this to the 10.0.x milestone Mar 25, 2026

github-project-automation bot added this to Runtime Infra Mar 25, 2026

kotlarmilos modified the milestones: 10.0.x, 11.0.0 Mar 25, 2026

matouskozak approved these changes Mar 25, 2026

View reviewed changes

steveisok approved these changes Mar 25, 2026

View reviewed changes

Merge branch 'main' into reduce-apple-ci-matrix

b128067

Conversation

kotlarmilos commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

What runs where

runtime.yml — ios + maccatalyst arm64 smoke (6 jobs)

runtime-extra-platforms-ioslike.yml — tvos full + Debug smoke (6 jobs)

runtime-extra-platforms-ioslikesimulator.yml — iossimulator Release + Debug smoke (9 jobs)

runtime-extra-platforms-maccatalyst.yml — maccatalyst full + Debug smoke (12 jobs)

Other changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vitek-karas commented Mar 11, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

kotlarmilos commented Mar 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

kotlarmilos commented Mar 25, 2026

Uh oh!

dotnet-policy-service bot commented Mar 25, 2026

Uh oh!

matouskozak left a comment

Choose a reason for hiding this comment

Uh oh!

matouskozak Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

steveisok left a comment

Choose a reason for hiding this comment

Uh oh!

kotlarmilos commented Mar 27, 2026

Uh oh!

azure-pipelines bot commented Mar 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

kotlarmilos commented Mar 11, 2026 •

edited

Loading

`runtime.yml` — ios + maccatalyst arm64 smoke (6 jobs)

`runtime-extra-platforms-ioslike.yml` — tvos full + Debug smoke (6 jobs)

`runtime-extra-platforms-ioslikesimulator.yml` — iossimulator Release + Debug smoke (9 jobs)

`runtime-extra-platforms-maccatalyst.yml` — maccatalyst full + Debug smoke (12 jobs)