feat(voice): use natural Pocket TTS segmentation - #3588
Merged
tlongwell-block merged 3 commits intoJul 31, 2026
Merged
Conversation
johnmatthewtennant
force-pushed
the
jtennant/pocket-tts-natural-segmentation
branch
from
July 29, 2026 18:37
4f0ea44 to
83239d2
Compare
johnmatthewtennant
changed the title
Signed-off-by: John Tennant <jtennant@squareup.com>
johnmatthewtennant
force-pushed
the
jtennant/pocket-tts-natural-segmentation
branch
from
July 29, 2026 19:17
83239d2 to
4e6d2df
Compare
johnmatthewtennant
marked this pull request as ready for review
July 29, 2026 19:32
johnmatthewtennant
added
triage-ready
Each boundary scan in split_at_natural_boundaries tokenized every word and clause boundary through end-of-text, skipping candidates that already exceeded the limit instead of stopping at them. Because token_count re-encodes the prefix text[start..end], tokenizer input grew superlinearly in prompt length, and that work is paid before the first chunk reaches synthesis -- taxing the time-to-first-audio this segmentation is meant to improve. A 1.9 KB prompt at the 2000-char MAX_TTS_TEXT_LEN cap tokenized 2.36 MB, and doubling a prompt multiplied tokenizer input by ~5.5x. Prepared token counts are monotonic in prefix length, so the first candidate that overflows proves no longer candidate can fit. Break out of the scan there. Chunk boundaries are unchanged: verified output-identical against the previous behavior across 5040 cases (21 corpora x max_tokens 1..=60 x both split policies x two tokenizer shapes), while saving 1,073,280 tokenizer calls and 568 MB of tokenized input. Co-authored-by: Tyler Longwell <tlongwell@squareup.com> Signed-off-by: Tyler Longwell <tlongwell@squareup.com>
Advances this PR's base from 881ce50 to the pushed #3257 head 254eed8. Clean auto-merge, no manual resolution: three files auto-merged (crates/buzz-voice/src/pocket.rs, desktop/src-tauri/Cargo.toml, desktop/src-tauri/Cargo.lock) with zero conflicts and zero index stage entries. Resulting tree 4a821be matches the expectation registered independently before this merge existed. The O(n^3) segmentation fix on 15667de survives byte-identically: crates/buzz-voice/src/pocket_april.rs is 37ef88e in both this tree and the pre-merge head, and the base never touched that file. Derived with rerere disabled so no cached resolution could be replayed. Co-authored-by: Tyler Longwell <tlongwell@squareup.com> Signed-off-by: Tyler Longwell <tlongwell@squareup.com>
tlongwell-block
merged commit Jul 31, 2026
4d18f5b
into
jtennant/pocket-tts-streaming
32 checks passed
johnmatthewtennant
added a commit
that referenced
this pull request
Jul 31, 2026
## Context Pocket TTS decoder streaming begins playback while each model-valid chunk is still being decoded. Segmentation should preserve a fast first-sentence boundary, enforce the model's exact 50-token safety limit, and use natural prosody boundaries for oversized sentences and the remainder. ## Summary Move Pocket TTS segmentation into the shared tokenizer-aware layer. A fitting first sentence stays separate for low time-to-first-audio. Oversized sentences use clause, word, then UTF-8 scalar fallback, while later sentences pack into the largest natural unit of at most 50 prepared model tokens. Returned chunks are contiguous and reconstruct the prepared prompt exactly. ## Changes - Replace the shared word-only splitter with distinct playback and model-safety policies using actual SentencePiece counts. - Keep a fitting first sentence as the initial playback unit, then pack later sentences into the largest natural model-valid units. - Split oversized sentences at clause, word, then UTF-8 scalar boundaries. - Preserve punctuation, whitespace, Unicode, text order, and exact reconstruction across chunks. - Remove the Desktop 200-character splitter while preserving its low-latency first-sentence behavior in the shared layer. - Route Desktop playback directly through the shared splitter without changing streaming, cancellation, queue, fade, or voice-switch behavior. - Remove the unused Desktop sentence splitter and its tests. ## Related issue None found. ## Testing Parent versus child medians after one warmup and three measured runs: | Response | Parent TTFA | Child TTFA | Delta | Parent RTF | Child RTF | | --- | ---: | ---: | ---: | ---: | ---: | | Short | 335.328 ms | 317.890 ms | -17.439 ms | 0.231865 | 0.220702 | | Medium | 472.118 ms | 383.934 ms | -88.184 ms | 0.214413 | 0.188397 | | Long | 922.557 ms | 682.583 ms | -239.974 ms | 0.186114 | 0.185482 | In this measured run, preserving the first-sentence boundary improved median TTFA for all three response lengths while the natural remainder policy kept median RTF lower for all three. Cancellation after callback 2 returned `Interrupted` in 0.0195 ms. Boundary cancellation prevented chunk 2 from starting. Every final callback matched returned PCM byte-for-byte, callback lengths were monotonic, splitting was deterministic, and text reconstruction was exact. ## Reviewer-reproducible examples Run model-independent boundary properties: ```sh cargo test --manifest-path crates/buzz-voice/Cargo.toml --lib natural_split cargo test --manifest-path crates/buzz-voice/Cargo.toml --lib oversized ``` Run the cached-model cases: ```sh BUZZ_POCKET_TEST_MODEL_DIR=/path/to/april-int8-bundle \ cargo test --manifest-path crates/buzz-voice/Cargo.toml --lib -- --ignored --nocapture ``` The 90-token Gary sentence reconstructs exactly as three natural chunks of 32, 42, and 15 prepared tokens. Automated transcription found complete expected endings in all six clips, with no clipping, late onset, or meaningful DC offset. Manual listening in the combined daily-driver build also completed successfully. ## Screenshots N/A, nonvisual audio behavior. --------- Signed-off-by: John Tennant <jtennant@squareup.com> Signed-off-by: Tyler Longwell <tlongwell@squareup.com> Co-authored-by: npub1jmc9dt2lyvzu3h0kxlwxt5zg4fxp9476awyxw6gwxn72g6cw7exqs64whm <96f056ad5f2305c8ddf637dc65d048aa4c12d7daeb8867690e34fca46b0ef64c@buzz.block.builderlab.xyz> Co-authored-by: Tyler Longwell <tlongwell@squareup.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Pocket TTS decoder streaming begins playback while each model-valid chunk is still being decoded. Segmentation should preserve a fast first-sentence boundary, enforce the model's exact 50-token safety limit, and use natural prosody boundaries for oversized sentences and the remainder.
Summary
Move Pocket TTS segmentation into the shared tokenizer-aware layer. A fitting first sentence stays separate for low time-to-first-audio. Oversized sentences use clause, word, then UTF-8 scalar fallback, while later sentences pack into the largest natural unit of at most 50 prepared model tokens. Returned chunks are contiguous and reconstruct the prepared prompt exactly.
Changes
Related issue
None found.
Testing
Parent versus child medians after one warmup and three measured runs:
In this measured run, preserving the first-sentence boundary improved median TTFA for all three response lengths while the natural remainder policy kept median RTF lower for all three.
Cancellation after callback 2 returned
Interruptedin 0.0195 ms. Boundary cancellation prevented chunk 2 from starting. Every final callback matched returned PCM byte-for-byte, callback lengths were monotonic, splitting was deterministic, and text reconstruction was exact.Reviewer-reproducible examples
Run model-independent boundary properties:
Run the cached-model cases:
BUZZ_POCKET_TEST_MODEL_DIR=/path/to/april-int8-bundle \ cargo test --manifest-path crates/buzz-voice/Cargo.toml --lib -- --ignored --nocaptureThe 90-token Gary sentence reconstructs exactly as three natural chunks of 32, 42, and 15 prepared tokens.
Automated transcription found complete expected endings in all six clips, with no clipping, late onset, or meaningful DC offset. Manual listening in the combined daily-driver build also completed successfully.
Screenshots
N/A, nonvisual audio behavior.