Fix MAG L1C gap-filling to follow spec section 7.3.4 step 3; pass T013-T016, T024

[sapols]

Summary (From Claude)

Addresses IMAP-Science-Operations-Center#1993 — MAG L1C validation tests T015 and T016 were failing.

Root cause: The existing L1C gap-fill code did not implement the algorithm specified in section 7.3.4 step 3 of IMAP-MAG-SW-009-01B (MAG Baseline Science Algorithm Doc, issue 5 rev 1). The spec requires: find the nearest BM timestamp (tC) to the gap start (tA), decimate BM samples around tC, shift the decimated timestamps by (tA − tC), and trim to the open interval (tA, tB). The old code instead built a fixed-cadence scaffold timeline and tried to fill it from burst data after the fact — producing wrong timestamps, wrong row counts, and wrong interpolation results for any test case with non-2Hz rates or config-mode transitions.

All 114 MAG tests now pass, including all 10 L1C validation tests (T013–T016, T024 × mago/magi).

Why this PR changes so much code

The short answer: the old gap-fill approach was structurally incompatible with the spec, not just parametrically wrong. Fixing it required replacing the control flow, not just tweaking values.

The old code used a scaffold-first approach:

1. Build a complete regular-cadence timeline for the entire gap
2. Try to fill each scaffold slot by finding the nearest burst sample
3. Mark unfilled slots as MISSING

This fails because:

• The scaffold timestamps don't match what the spec's decimate-shift-trim algorithm produces
• After CIC filtering, adjusted timestamps may not land on scaffold positions
• Gaps beyond BM coverage get MISSING rows that shouldn't exist (the spec says only produce timestamps derivable from BM data)

The new code uses a plan-first approach that mirrors the spec directly:

1. Detect gaps using rate-aware segment analysis
2. For each gap, compute a GapFillPlan (frozen dataclass) with the exact synthetic epochs and their BM source indices — BEFORE any interpolation
3. Insert plan-derived epochs into the NM timeline
4. Interpolate using the plan's pre-computed BM windows

This separation of "what timestamps to produce" from "how to interpolate them" is what makes the new code correct — and it's why so many lines changed. The actual algorithm in each new function is straightforward; the bulk of the diff is the new function structure and docstrings.

What's preserved unchanged: mag_l1c(), select_datasets(), generate_empty_norm_array(), vectors_per_second_from_string(), remove_missing_data(), and the overall process_mag_l1c() flow. The existing L1A, L1B, L1D, and L2 tests are completely unaffected.

Bugs fixed

1. Gap-fill algorithm did not follow spec section 7.3.4 step 3

The spec (pages 29–32) prescribes: find tC (nearest BM sample to tA), decimate around tC at burst_rate / norm_rate step, shift decimated timestamps to NM alignment via tA + period_offsets × nm_spacing, trim to (tA, tB). The old code skipped all of this — it generated a regular grid at fixed cadence and searched for the nearest burst sample per grid point. This produced correct-looking results only when the NM rate happened to be 2 Hz and BM coverage was complete.

2. Nanosecond float round-trip precision loss

Timestamp values (int64 nanoseconds, ~10^18) were being passed through float64 arithmetic, which has only ~15.9 significant digits. This caused silent ±1–100 ns drift in epoch values. Fixed by adding _to_int64_ns() / _to_int64_ns_scalar() helpers that keep all timestamp math in int64 and only use np.rint() when forced to convert from float.

3. Hardcoded 0.5s gap cadence ignored NM data rate

generate_missing_timestamps() always used 0.5s spacing regardless of the actual NM rate. The spec (section 4.4.1, page 9) says cadence depends on the NM data rate before the gap. T016 proves this: MAGo uses rate 4 → 0.25s spacing, MAGi uses rate 1 → 1.0s spacing. The old hardcoded 0.5s only appeared correct because T013–T015 all use N2_2 mode (rate 2 = 0.5s). Fixed: spacing is now int(1e9 / rate).

4. Spurious micro-gaps at Config mode transitions

When MAG switches science modes (e.g., N2_2 → B64_8), the rate change creates timestamp spacings that find_gaps() misidentified as data gaps. The old code had a fragile post-hoc filter deleting gaps under 1.5s. Fixed: _find_rate_segments() walks backward from configured transition points to find where observed cadence matches the new rate, then find_all_gaps() applies each segment's rate when checking for gaps.

5. CIC filter edge loss from missing extrapolate=True

linear_filtered() called remove_invalid_output_timestamps() which trimmed output timestamps to the input range — but the CIC filter's delay truncation had already shortened the input, causing valid output timestamps to be discarded. The spec's reference code (page 32) uses IUS (InterpolatedUnivariateSpline) which extrapolates by default. Fixed: pass extrapolate=True to linear().

6. Wrong 2D array indexing in fill_normal_data()

Used 1D boolean mask on 2D vectors array, causing shape mismatch. Fixed: apply mask to first axis only (vectors[mask, :]).

7. Off-by-one in burst data slicing

interpolate_gaps() used burst_gap_end exclusively, dropping the last sample needed for interpolation. Fixed: include the boundary sample.

New functions (in mag_l1c.py)

Function	Purpose
_to_int64_ns() / _to_int64_ns_scalar()	Prevent float round-trip loss in ns timestamps
GapFillPlan (dataclass)	Encapsulates gap metadata: tA, tB, rates, synthetic epochs, source indices, BM window
_find_rate_segments()	Detect observed-cadence segments with backward walk from config transitions
get_vecsec_dict()	Extract rate mapping from dataset with sensible default
find_nearest_epoch_index()	Nearest-timestamp lookup with earlier-on-tie determinism
build_decimated_indices()	Decimate around anchor at burst/norm ratio step
build_synthetic_epochs()	Convert decimated BM indices to zero-jitter NM timestamps
_find_segment_for_time()	Resolve a timestamp to its rate segment
build_gap_fill_plan()	Core spec step 3: decimate → shift → trim → plan
build_gap_fill_plans()	Wrapper: compute BM segments once, plan all gaps
build_timeline_from_gap_plans()	Insert synthetic epochs into NM timeline
_build_fallback_gap_fill_plan()	No-NM fallback (T024): scaffold-based plan with BM coverage trim
build_default_gap_fill_plans()	Wrapper for fallback path

Modified functions

Function	What changed
process_mag_l1c()	Normal+burst path now uses build_gap_fill_plans() → build_timeline_from_gap_plans() → interpolate_gaps()
fill_normal_data()	Fixed 2D indexing bug
interpolate_gaps()	Uses GapFillPlan objects; source tracking via dict lookup + set (was np.searchsorted); no-NM fallback delegates to build_default_gap_fill_plans()
find_all_gaps()	Rate-aware gap detection using _find_rate_segments() instead of single global rate
generate_missing_timestamps()	Uses gap's rate column instead of hardcoded 0.5s
generate_timeline()	Uses gap rate for timestamp generation

New tests

• test_find_nearest_epoch_index_prefers_earlier_on_tie — deterministic tie-breaking
• test_find_segment_for_time_uses_precomputed_segments — segment resolution
• test_build_decimated_indices_includes_anchor — anchor always present
• test_generate_missing_timestamps_uses_gap_rate — rate-aware cadence
• test_build_gap_fill_plan_uses_shifted_burst_timestamps — spec step 3 shift
• test_build_gap_fill_plan_regularizes_burst_jitter — zero-jitter output from jittery input
• test_build_gap_fill_plan_uses_observed_burst_transition_boundary — real T015-scale timestamps
• test_build_gap_fill_plans_match_step_three_shifted_timeline — end-to-end timeline check
• test_find_all_gaps_uses_observed_transition_boundary — rate transition detection
• test_generate_timeline_rate_gap — rate-aware gap fill in legacy path
• test_find_rate_segments — backward-walk segment detection
• test_build_gap_fill_plan — core plan correctness

Validation test changes

• Row count: exact match (assert len(expected_output.index) == len(l1c["epoch"].data)) — was ±1 tolerance
• Timestamp tolerance: 1 ms (was 500 ms)
• Added checks for range, compression, compression_width, and interp columns when present in expected output

Spec references

• IMAP-MAG-SW-009-01B (MAG Baseline Science Algorithm Doc, issue 5 rev 1):
  • Section 7.3.4, pages 29–32: L1B→L1C steps 1–6 (the primary algorithm)
  • Section 4.4.1, page 9: data cadence depends on science mode rate
  • Page 32: CIC filter reference code using IUS (extrapolates by default)
• IMAP-OPS-TP-ICL-001 (SDC Data Validation, issue 1 rev 0, 13 Mar 2025):
  • Section 4.3.2, pages 11–12: T013–T016, T024 test descriptions

Recommended pre-merge validation

Per @alastairtree: run on early MAG commissioning data (where modes changed several times) and send the MAG team a produced L1C CDF for review.