fix(extF80): rem and sqrt mishandle pseudo-infinity inputs

[johnwbyrd]

Summary

The extFloat80 format stores the integer bit (J, bit 63 of the significand) explicitly. Every combination of exponent and significand is a valid encoding with a defined mathematical value. The original Intel 8087 hardware defined and correctly processed all such encodings -- unnormals, pseudo-denormals, pseudo-infinities, and pseudo-NaNs -- as described in Intel's US Patent 4,325,120 and in every subsequent Intel architecture manual covering the x87 FPU.

In extFloat80, infinity can be encoded two ways: the canonical form {exp=0x7FFF, sig=0x8000000000000000} with J=1, and the pseudo-infinity {exp=0x7FFF, sig=0x0000000000000000} with J=0. Both represent the same value (infinity).

Wrong results demonstrated

Expression	SoftFloat returns	Correct result
rem(1.0, {exp=0x7FFF, sig=0})	NaN with invalid flag	1.0 with no flags (rem(finite, Inf) = finite)
sqrt({exp=0x7FFF, sig=0})	{0x7FFF, 0x0000..0} (non-canonical)	{0x7FFF, 0x8000..0} (canonical +Inf)

A standalone program demonstrating both: https://gist.github.com/johnwbyrd/cb75a1844661677fe614bc39f171fe39 (Bugs 7 and the sqrt case)

Root causes and fixes

extF80_rem.c: When the divisor B is pseudo-infinity, the infinity handler doubles expB (to force an early return of A unchanged), but leaves sigB=0. The subsequent normalization guard sees sigB=0 and jumps to the invalid label, returning NaN. The fix sets sigB to the canonical J=1 value in the infinity handler so the normalization guard passes through. The actual value of sigB doesn't matter... the huge expDiff causes goto copyA before sigB is used in any computation.

extF80_sqrt.c: When the operand is pseudo-infinity, the positive-infinity path does return a, passing the non-canonical encoding through as the result unchanged. The fix constructs and returns the canonical infinity encoding.

Related

• TestFloat generator fix: extF80 test generator: cover all valid J-bit encodings berkeley-testfloat-3#21
• SoftFloat add/sub fix: fix(extF80): correct add/sub for valid inputs with J-bit clear #38
• SoftFloat mul fix: fix(extF80): extF80_mul returns infinity for Inf times zero #39

Suggested test plan

• Build SoftFloat with this patch, rebuild TestFloat with extF80 test generator: cover all valid J-bit encodings berkeley-testfloat-3#21, and run testsoftfloat -level 1 extF80_rem and testsoftfloat -level 1 extF80_sqrt -- both should pass across all rounding modes
• Run testsoftfloat -level 1 extF80_div as a regression check
• Run testsoftfloat -level 1 f64_add to confirm non-extF80 operations are unaffected

[johnwbyrd]

Closing this PR. The fix here is superseded by a later, more comprehensive PR that addresses non-canonical extFloat80 encodings across all extF80 operations.