Skip to content

feat: add reliable macOS audio capture streams - #264

Open
marcusschiesser wants to merge 2 commits into
vercel-labs:mainfrom
marcusschiesser:codex/reliable-audio-stream
Open

feat: add reliable macOS audio capture streams#264
marcusschiesser wants to merge 2 commits into
vercel-labs:mainfrom
marcusschiesser:codex/reliable-audio-stream

Conversation

@marcusschiesser

Copy link
Copy Markdown
Contributor

Closes #262.

Supersedes #257 with the reliable, pull-based stream design discussed in the replacement issue. File/container writing remains out of scope and is tracked separately in #263.

Summary

  • add macOS 15+ system-audio, microphone-only, and combined capture using ScreenCaptureKit and AVFoundation
  • expose separate, timestamp-aligned signed 16-bit PCM source bytes through reliable Zig effects and TypeScript commands
  • keep accepted frames in a bounded runtime ring; stop with consumer_too_slow instead of overwriting when the app does not drain it
  • enumerate microphone devices, report invalidation changes, and provide explicit non-prompting status/request permission operations
  • journal and replay lifecycle/read deliveries, including the PCM bytes consumed by the app

Stream contract

  • one active capture per app
  • 20 ms paired blocks, reads bounded to 20–100 ms
  • default 5 second ring, configurable from 1–30 seconds
  • disabled sources return an empty byte slice
  • timestamp gaps are zero-filled and counted independently from real acoustic silence
  • stop seals producers but preserves buffered frames for draining; discard releases immediately
  • no file output, mixing, encoders, gains, or mode option

Platform and packaging

  • system and combined capture use ScreenCaptureKit on macOS 15+
  • microphone-only capture uses AVFoundation without requiring Screen Recording access
  • older macOS, Chromium, Windows, Linux, and mobile hosts report the features unsupported
  • add system_audio manifest permission and microphone/system-audio privacy strings, generating the relevant Info.plist purpose keys
  • leave App Sandbox entitlements app-owned and document com.apple.security.device.audio-input

Verification

  • zig build test
  • zig build validate
  • npm run gate -- --skip-bench in packages/core
  • pnpm --dir docs check
  • Zig and TypeScript audio-capture example tests
  • macOS audio-capture example build/link against AppKit, AVFoundation, ScreenCaptureKit, AudioToolbox, and CoreMedia
  • managed and standalone example checks, including the post-rebase gpu-components target
  • mobile layout/package checks
  • Objective-C syntax/availability build for the audio backend

Local scripts/gate.sh fast has three environment/baseline limitations worth calling out:

  • the aggregate native-example shard stalls in Zig while the terminal example resolves/downloads its pinned Ghostty dependency; the audio-relevant standalone and registered example targets pass independently
  • bench-render misses the existing toggle-big-view 9 ms budget at 10.18 ms; the same command on origin/main fails at 10.22 ms
  • the CEF link check skips because this checkout has no prepared CEF distribution; the Chromium host change is a signature-matched unsupported stub

Manual TCC prompt/restart testing remains a draft-PR follow-up because it requires interactive Screen Recording and microphone grants on a test machine.

@vercel

vercel Bot commented Aug 2, 2026

Copy link
Copy Markdown

@marcusschiesser is attempting to deploy a commit to the Vercel Labs Team on Vercel.

A member of the Team first needs to authorize it.

@marcusschiesser
marcusschiesser marked this pull request as ready for review August 3, 2026 14:14
@marcusschiesser

marcusschiesser commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Design: expose reliable paired system and microphone audio streams

1 participant