Skip to content

Chunked JSON transport for large stat payloads (Phase 1 + 2)#108

Merged
steve-the-edwards merged 2 commits intomainfrom
sedwards/chunked-stat-transport
Apr 16, 2026
Merged

Chunked JSON transport for large stat payloads (Phase 1 + 2)#108
steve-the-edwards merged 2 commits intomainfrom
sedwards/chunked-stat-transport

Conversation

@steve-the-edwards
Copy link
Copy Markdown
Collaborator

@steve-the-edwards steve-the-edwards commented Apr 16, 2026 •
edited
Loading

Summary

Large stat payloads in hosted Invert reports cause browser timeouts. This PR adds a chunked JSON transport that splits oversized stats into parallel-fetchable chunks and teaches the runtime to load them via fetch() instead of <script> tag injection.

Phase 1: Producer-Side Chunking

When a stat file's serialized JSON exceeds 2 MB, InvertJsReportWriter emits:

  • js/stat_<key>.manifest.json — chunk manifest
  • js/stat_<key>.chunk.N.json — individual chunks (valid StatJsReportModel JSON each)
  • js/stat_<key>.js — legacy single-file JS (always emitted for backward compat)

Each chunk contains the full statInfo and a subset of statsByModule entries, split on sorted module-key boundaries targeting ~2 MB per chunk.

Phase 2: Runtime Chunked Loading

  • invert.js: New externalLoadChunkedJsonFile(key, callback) — fetches manifest, loads all chunks in parallel via fetch(), merges statsByModule, passes merged JSON to callback. Falls back to legacy externalLoadJavaScriptFile if manifest 404s.
  • RemoteJsLoadingProgress: Routes stat_* keys through chunked loader. Bumps timeouts from 10s/30s to 15s/120s. Replaces window.alert() on timeout with bounded state cleanup.
  • External.kt: New externalLoadChunkedJsonFile external declaration.

Why This Is Faster

Step Legacy Chunked
Download Single 100MB+ JS file Parallel fetch() of ~2MB chunks
Parse Browser JS engine parses/executes entire file fetch().json() — no JS execution
Bridge JSON.stringify() 100MB object back to string Data arrives as JSON, stringified once
Timeout risk 30s hard limit 120s safety net, parallel fetches complete faster

Backward Compatibility

Report Artifact Runtime Behavior
Legacy (no manifest) Old runtime Works (unchanged)
Legacy (no manifest) New runtime Works (manifest 404 → fallback)
Chunked + legacy JS Old runtime Works (ignores manifest, loads legacy)
Chunked + legacy JS New runtime Works (loads via chunks)

Files Changed

File Change
ChunkManifest.kt New model in invert-models
InvertJsReportWriter.kt writeChunkedStatIfNeeded() + buildChunks()
InvertJsReportWriterTest.kt 7 new tests
invert.js externalLoadChunkedJsonFile()
External.kt New external declaration
RemoteJsLoadingProgress.kt Stat routing + timeout improvements

@steve-the-edwards @ampcode-com
Emit chunked JSON manifests for large stat payloads (Phase 1)
026791d 
When a stat file's serialized JSON exceeds 2 MB, InvertJsReportWriter
now emits a chunk manifest (stat_<key>.manifest.json) and multiple
chunk files (stat_<key>.chunk.N.json) alongside the legacy single-file
JS. Each chunk is a valid StatJsReportModel with the full statInfo and
a subset of statsByModule entries, split on sorted module-key
boundaries targeting ~2 MB per chunk.

This is producer-side only — no runtime/browser changes yet. The
legacy js/stat_<key>.js file is always emitted for backward
compatibility.

Adds ChunkManifest model to invert-models and 7 new tests covering:
- Small stats produce no chunks
- Large stats produce manifest + chunk files
- Chunks are valid StatJsReportModel JSON
- Merged chunks equal the original model
- No duplicate module keys across chunks
- buildChunks preserves all modules
- Single oversized module produces one chunk

Amp-Thread-ID: https://ampcode.com/threads/T-019d96d7-0e27-7579-b264-f93da72fbaa4
Co-authored-by: Amp <amp@ampcode.com>
@steve-the-edwards steve-the-edwards marked this pull request as ready for review April 16, 2026 16:15
@steve-the-edwards @ampcode-com
Runtime chunked JSON loading for stat keys (Phase 2)
7f15c23 
Add externalLoadChunkedJsonFile to invert.js: for stat_ keys, tries
fetching a chunk manifest, then loads all chunks in parallel via
fetch(), merges statsByModule across chunks, and passes the merged
JSON to the Kotlin/JS callback. Falls back to the legacy script-tag
loader if no manifest is found (404).

Update RemoteJsLoadingProgress to route stat_ keys through the
chunked loader. Bump timeouts from 10s/30s to 15s/120s. Replace
window.alert() on timeout with a console log and bounded state
cleanup.

Add externalLoadChunkedJsonFile external declaration to External.kt.

Amp-Thread-ID: https://ampcode.com/threads/T-019d96d7-0e27-7579-b264-f93da72fbaa4
Co-authored-by: Amp <amp@ampcode.com>
@steve-the-edwards steve-the-edwards changed the title Emit chunked JSON manifests for large stat payloads Chunked JSON transport for large stat payloads (Phase 1 + 2) Apr 16, 2026
@steve-the-edwards steve-the-edwards merged commit a8d0c4a into main Apr 16, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zz-squareup zz-squareup zz-squareup approved these changes

@handstandsam handstandsam Awaiting requested review from handstandsam

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@steve-the-edwards @zz-squareup