fix(detection): stop framework detection OOM on large repos - #970
Draft
posthog[bot] wants to merge 1 commit into
Draft
fix(detection): stop framework detection OOM on large repos#970posthog[bot] wants to merge 1 commit into
posthog[bot] wants to merge 1 commit into
Conversation
Framework detection ran recursive `**` globs before any agent work, and the Python/Swift/Cloudflare detectors only ignored venv/build-artifact dirs — not node_modules or .git. On large JS/TS repos fast-glob buffered the whole node_modules tree (100K+ entries) into memory just to check whether the project was Python, crashing `npx @posthog/wizard` with a JS heap OOM during detection, before any setup could happen. - Add a shared DETECTION_IGNORE_PATTERNS (node_modules, .git, dist, build, venv/.venv/env/.env, __pycache__) and apply it to the Python, Swift, and Cloudflare detectors, matching the discipline the Django/Flask/FastAPI detectors already follow. - Wire the detection timeout in framework.ts to an AbortController and stream the heavy globs via globWithAbort, so a runaway walk actually stops reading the filesystem instead of leaking memory after Promise.race resolves. Generated-By: PostHog Code Task-Id: 61aad81a-0f26-44b6-b42b-62b4cc64597b
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Results will be posted here when complete. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Running
npx @posthog/wizard@latestcould crash with a fatal JavaScript heap out-of-memory error during framework detection — before any setup work started, so affected users had no way to install PostHog and no workaround. Because the process died mid-detection it never self-reported, showing up as silent onboarding drop-off rather than a captured error.The root cause: on every run, detection loops over all registered frameworks and calls each
detect()predicate unconditionally. The Python, Swift, and Cloudflare detectors ran recursive**globs whoseignorelists covered only venv/build-artifact dirs — notnode_modulesor.git. On a large JS/TS repo, fast-glob buffered the entirenode_modulestree (often 100K+ entries) into memory just to check whether the project was Python. ThePromise.racetimeout inframework.tsdidn't abort the walk either — it only stopped awaiting it, so memory kept climbing in the background until the heap died. Scoped to projects with big filesystem trees, so a subset of runs rather than all.Why: The wizard is PostHog's primary onboarding entry point, and this was a hard crash of the core flow for users with large repos.
Changes
DETECTION_IGNORE_PATTERNS(node_modules,.git,dist,build,venv/.venv/env/.env,__pycache__) and apply it to the Python, Swift, and Cloudflare detectors — matching the ignore discipline the Django/Flask/FastAPI detectors already follow. This is the definitive fix for the OOM.framework.tsto anAbortControllerand route the heavy recursive globs through a newglobWithAborthelper that streams the walk anddestroy()s it on abort, so a runaway detection actually stops reading the filesystem instead of leaking memory after the race resolves.Test plan
src/lib/detection/__tests__/glob.test.ts: covers the ignore constant,globWithAbort(pass-through, ignore list, pre-aborted, abort mid-walk), and regression tests that Python/Swift detection no longer walks intonode_modules.tscadds no new type errors, prettier + eslint clean on changed files.Created with PostHog Code from this inbox report.