fix: switch scanJsonlFile and parseSessionFile to readSessionLines to prevent OOM#132
Merged
iamtoruk merged 1 commit intogetagentseal:mainfrom Apr 22, 2026
Conversation
… prevent OOM
readViaStream (used for files ≥8 MB) reconstructs the full file as a
single string via chunks.join('\n'), giving the same peak allocation as
readFile. Callers then call content.split('\n'), creating a second copy.
With FILE_READ_CONCURRENCY=16 and files up to 128 MB this can exhaust
the V8 heap (~6 GB theoretical peak).
readSessionLines already exists as a proper async generator that yields
one line at a time. Switch both hot-path callers to iterate it directly
so the full file string is never held in memory.
Adds two tests: a spy test confirming readSessionLines is called (not
readSessionFile), and a 500-entry correctness test.
Fixes getagentseal#131
webrulon
pushed a commit
to webrulon/codeburn
that referenced
this pull request
Apr 22, 2026
OOM streaming fix (getagentseal#132), compact menubar mode (getagentseal#133), keychain credential fix + App Nap hardening (getagentseal#134).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #131
Problem
readViaStream(the code path for files ≥ 8 MB introduced in #67) reassembles the entire file into a single string viachunks.join('\n'), giving the same peak allocation as a plainreadFile. Callers then docontent.split('\n'), creating a second full copy. WithFILE_READ_CONCURRENCY = 16and files up to 128 MB, theoretical peak heap usage is ~6 GB — right where the crash lands:Fix
readSessionLinesalready exists insrc/fs-utils.tsas a proper async generator that yields one line at a time and never holds the full file in memory. This PR switches the two hot-path callers to iterate it directly:scanJsonlFileinsrc/optimize.tsparseSessionFileinsrc/parser.tsNo concurrency change needed — with true line-by-line streaming, 16 concurrent files each hold only one line at a time.
Tests
Two new tests added to
tests/optimize-fs.test.ts:readSessionLinesis called andreadSessionFileis not called byscanJsonlFileChecklist
npm run buildpassesnpm testpasses (323 tests, 0 failures)