[release/10.0] JIT: Use r9,r10 for GS cookie check in Swift reverse pinvokes

[jakobbotsch]

Customer Impact

• Customer reported
• Found internally

The JIT corrupts return values from x64 Swift reverse pinvokes when 4 registers are used to return in if the method simultaneously uses unsafe value types that induce a GS cookie check.

Regression

• Yes
• No

Introduced by #119864.

Testing

Internal stress testing (that forces GS cookie check) caught this in our existing Swift tests.

Risk

Low. Change only affects Swift reverse pinvokes with GS cookie checks.

[Copilot]

Pull Request Overview

This PR fixes a regression in the JIT compiler where return values from Swift reverse pinvokes were corrupted when 4 registers were used for return values and the method required a GS (Guard Stack) cookie check. The fix changes the registers used for GS cookie checks in Swift calling convention from R8/R9 to R9/R10, avoiding conflicts with Swift return value registers.

• Updates register allocation for GS cookie checks in Swift calling convention
• Prevents corruption of Swift reverse pinvoke return values when using unsafe code

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[LoopedBard3]

Closing in favor of: dotnet/performance#5024 Sorry, updated incorrect PR.

[PranavSenthilnathan]

Since this PR is targeting release/10.0, this is a reminder that code complete is Monday November 17. If you want the changes in the PR in the release, please merge it on time.

[jakobbotsch]

Won't push this for servicing given Swift reverse pinvoke + GS cookie check + macOS x64 requirements, and no customer reports.