fix(mlxcel-core): make audio-path MLX ops fallible at the FFI boundary

[inureyes]

Summary

The audio synthesis and transcription forward paths build MLX graphs through mlxcel-core cxx ops declared as -> UniquePtr<MlxArray> (not Result). When the underlying MLX C++ throws (shape mismatch caught eagerly at graph build, allocation failure at eval), cxx hard-aborts the process via std::terminate. That is not a Rust panic, so the catch_unwind backstop in src/server/audio_worker.rs (run_guarded) cannot intercept it regardless of the release panic setting. This implements option (a) from the issue: add Result-returning variants of the MLX ops that dominate the audio forward path and thread them through the Kokoro/Whisper code, so a thrown MLX exception becomes a recoverable per-request Err instead of a process abort.

What changed

• src/lib/mlxcel-core/src/lib.rs, cpp/mlx_cxx_bridge.h, cpp/mlx_cxx_bridge.cpp: add two fallible FFI variants. try_matmul(a, b) -> Result<UniquePtr<MlxArray>> mirrors matmul; because it is declared -> Result, cxx wraps the call in try/catch and converts a thrown MLX exception into a Rust Err. try_array_to_raw_bytes(arr) -> Result<Vec<u8>> mirrors array_to_raw_bytes (contiguous + eval + copy-out) inside one cxx try/catch boundary. The non-fallible siblings are unchanged, so no other model path is affected.
• src/models/kokoro/ops.rs: add a fallible try_matmul wrapper, and rewrite to_vec_f32 (the single point where the lazy Kokoro graph is forced) to materialize through try_array_to_raw_bytes. This replaces the previous try_eval followed by a second, non-fallible eval inside array_to_raw_bytes, closing a real abort hole (an allocation failure making the contiguous copy of a large tensor was uncaught) and removing a redundant eval.
• src/models/kokoro/mod.rs: route the two alignment-expansion matmuls (en = d_t @ aln, asr = t_en @ aln) through ops::try_matmul. Their inner dimension is the frame count T, derived from the runtime duration prediction, so these are the data-dependent construction ops on the synthesis path.
• src/lib/mlxcel-core/src/ffi_tests.rs: regression tests for the new fallible ops.

Ops made fallible and why this set

MLX is lazy for the heavy ops, so the entire forward computation (every conv, the LSTM, the matmuls) and the allocation of the large T-dependent tensors are deferred to a single eval. Both audio paths force that graph at one materialization point: Kokoro at to_vec_f32, Whisper at argmax_with_masks. Routing the materialization through a fallible boundary therefore catches the eval-time fault set (allocation failure on the large data-dependent tensors, and any deferred op error) across the whole graph. MLX additionally validates some shapes eagerly at op construction, so the data-dependent alignment matmuls (whose inner dim comes from runtime data) get the fallible try_matmul variant to catch a graph-build throw too. This is the "minimize the number of new fallible variants" path the issue suggests: two new variants cover the realistic fault set on the synthesis path. Construction throws on the remaining weight-fixed ops require a checkpoint/config mismatch and are not reachable on the guarded forward path (input capped to 4096 chars, vocab-restricted g2p, 510-token truncation, finite/positive speed, per-token frame counts clamped to [1,100], whitelisted voices).

Transcription (Whisper) already forces its entire encoder and decoder graph at the single argmax_with_masks point through the pre-existing try_eval, so its eval-time faults are already recoverable. Its construction shapes are fixed by the 30 s mel window and the checkpoint config, and its matmuls live in shared mlxcel_core::layers used by every model, so they are intentionally left unchanged to keep this scoped to the audio path and avoid cross-cutting churn.

Test plan

• cargo check -p mlxcel-core --lib (runs build.rs, compiles the C++ shim, validates the cxx bridge): clean.
• cargo check -p mlxcel --lib --tests (server-side threading): clean.
• cargo test -p mlxcel-core --lib ffi_tests::try_: 3 passed. try_matmul_returns_err_on_shape_mismatch asserts a deliberate (2,3) x (4,5) mismatch returns Err (graph-build throw caught, not a process abort); try_matmul_ok_matches_matmul asserts the happy path; try_array_to_raw_bytes_round_trips_f32 asserts the fallible readback round-trips.
• cargo clippy -p mlxcel-core --lib --tests -- -D warnings and cargo clippy -p mlxcel --lib -- -D warnings: clean.
• cargo fmt --check: clean.
• No measurable throughput regression: the fallible variants add no eval/sync (try_matmul builds a lazy node; to_vec_f32 now does one fewer eval), so no per-op hot-path cost is expected. A full release build and the audio throughput bench were not run under the implementation watchdog; deferred to the release CI / maintainer.

Closes #382

[inureyes]

Implementation Review Summary

Intent

Make the audio synthesis/transcription forward path robust to MLX FFI C++ exceptions by adding Result-returning try_* cxx variants, so a C++ throw becomes a recoverable per-request Err instead of a std::terminate process abort (issue #382, option a).

Findings Addressed

None requiring a fix. No CRITICAL or HIGH findings.

Verification

• All stated requirements implemented
• No placeholder/mock code remaining
• Integrated into project code flow
• Project conventions followed
• Existing modules reused where applicable
• No unintended structural changes
• Tests pass

Detail

1. cxx Result mapping is correct. try_matmul and try_array_to_raw_bytes are declared -> Result in the bridge and implemented as non-noexcept C++ functions, so cxx wraps the call in its catch (const std::exception&) shim. MLX raises std::exception-derived types (std::invalid_argument on eager matmul shape validation, allocation failures at eval), which the shim converts to a Rust Err. try_matmul_returns_err_on_shape_mismatch (a deliberate (2,3)x(4,5) mismatch) is a genuine regression test that the throw is caught, not aborted.

2. Error propagation is clean: ops::try_matmul(...).map_err(|e| anyhow!(e))? and to_vec_f32(...).map_err(|e| anyhow!(e))? on the real forward path (kokoro/mod.rs:230,234,254), then KokoroEngine::synthesize maps the anyhow error to AudioModelError::Inference (kokoro_tts.rs:166), with run_guarded catch_unwind as defense-in-depth. No .unwrap()/.expect() on the fallible calls in production (only in test code).

3. Whisper exclusion is justified. Eval-time faults are already routed through the pre-existing try_eval at the single argmax_with_masks materialization point (decoding.rs:105), which forces the whole encoder+decoder graph. Construction-time shapes are fixed: input is always normalized to the 30s window (pad_audio_to_window_multiple + padded_window -> [1, N_FRAMES, n_mels]), token-embedding gather indices are bounded by argmax over n_vocab, and the decoder positional slice is bounded by offset + seq <= n_text_ctx (loop guard + max_new = n_text_ctx/2). The only residual construction throws require a checkpoint/config mismatch, which is the same load-time exclusion the issue grants the synthesis path. Unlike Kokoro, the Whisper frame dimension is not data-dependent, so there is no equivalent data-dependent construction op to wrap.

4. No new hot-path overhead. try_matmul builds a lazy node identically to matmul. to_vec_f32 now materializes through one fallible try_array_to_raw_bytes (contiguous + eval + copy) instead of try_eval followed by a second eval inside array_to_raw_bytes, removing a redundant eval. The C++ contiguous(arr.inner) default matches the prior contiguous(f, false) (allow_col_major=false), so the readback bytes are byte-identical. No per-op sync/copy added.

5. Conventions: tests in ffi_tests.rs, non-fallible siblings left intact for all other models, Closes #382 present, no AI attribution, no em dashes.

Remaining Items

• Acceptance criterion 3 (no measurable throughput regression) is empirically deferred to release CI per the PR test plan; the static argument (no added eval/sync, one fewer eval) supports it but no audio throughput bench was run.

[inureyes]

PR Finalization

Tests: No new tests added. The three tests in src/lib/mlxcel-core/src/ffi_tests.rs (error on shape mismatch, Ok-matches-matmul, readback round-trip) cover the FFI behavior completely. The Kokoro ops wrappers (ops::try_matmul, ops::to_vec_f32) are one-liner format wrappers with no branching logic worth a separate test.

Documentation: Two files updated (commit 46a482912):

• docs/adr/0003-release-panic-unwind-with-core-thread-abort.md: the Consequences bullet that described issue fix(server): make audio synthesis robust to MLX FFI exceptions (cxx std::terminate bypasses the panic backstop) #382 as an open residual now records the resolution and its scope (Kokoro alignment matmuls + PCM readback covered; Whisper unaffected; non-std::exception throws still terminate). References line updated to include PR fix(mlxcel-core): make audio-path MLX ops fallible at the FFI boundary #384.
• docs/audio-api.md: the open-residual sentence replaced with the current containment status and scope limits.

Lint/Format: cargo fmt --check -p mlxcel-core clean; cargo clippy -p mlxcel-core --lib --tests -- -D warnings clean; cargo clippy -p mlxcel --lib -- -D warnings clean.