Skip to content

fix(server): make audio synthesis panic-safe in release builds (panic="abort" defeats the AudioWorker catch_unwind backstop) #375

Description

@inureyes

Problem / Background

PR #374 added the Kokoro text-to-speech provider, which runs the StyleTTS2 + iSTFTNet forward pass on the shared single AudioWorker thread (src/server/audio_worker.rs) behind POST /v1/audio/speech. To keep one bad request from bricking all audio requests, PR #374 added a catch_unwind boundary around each engine call so a synthesis panic becomes a per-request error and the worker thread survives.

The problem is that Cargo.toml sets [profile.release] panic = "abort", and the release workflow and documented production build use cargo build --release. With panic = "abort", a panic does not unwind, so catch_unwind cannot intercept it: in a release/production build a synthesis panic aborts the entire server process, not just the offending request. The audio forward path contains multiple expect/unwrap sites. The existing input guards (4096-char input cap, Kokoro-vocab-restricted g2p output, 510-token truncation, finite/positive speed, per-token frame counts clamped to [1,100], whitelisted voice names) make these largely unreachable today, but the backstop is ineffective in release builds, so any future regression that introduces a reachable panic would crash the whole server.

Proposed Solution

Pick one of the following remediations, to be decided in this issue:

  • Option A: reconsider panic = "abort" for the server release build (for example, use panic = "unwind" for the binary, weighing the binary-size and performance tradeoff that the setting was chosen for) so the per-request catch_unwind backstop works in production.
  • Option B: keep panic = "abort" and instead make the audio synthesis forward path panic-free by converting the attacker-reachable expect/unwrap sites in src/models/kokoro/* and src/server/kokoro_tts.rs to recoverable Result errors, so no input can panic regardless of unwind policy.

Acceptance Criteria

  • A synthesis-path panic in a release build results in a per-request error response and does NOT abort the server process.
  • A regression test (or documented manual verification) demonstrates this behavior in release mode.
  • The chosen approach does not regress the input guards or the debug-build behavior already covered by the audio_worker panic-survival test.

Reference

Surfaced in PR #374; see the security-review comment on that PR for the two documented remediation options.

Metadata

Metadata

Assignees

Labels

area:inferenceGeneration, sampling, decoding (incl. speculative, DRY)area:modelsModel architectures, weights, loading, metadatapriority:mediumMedium prioritystatus:doneCompletedtype:securitySecurity vulnerability or fix

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions