Skip to content

Add typecheck budget guard to catch type-instantiation regressions#555

Merged
sroussey merged 3 commits into
mainfrom
claude/youthful-lovelace-14qjr5
Jun 9, 2026
Merged

Add typecheck budget guard to catch type-instantiation regressions#555
sroussey merged 3 commits into
mainfrom
claude/youthful-lovelace-14qjr5

Conversation

@sroussey

@sroussey sroussey commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a CI guardrail (scripts/typecheck-budget.ts) that gates per-package TypeScript instantiation counts to catch type-level performance regressions at PR time. Instantiation counts are deterministic and machine-independent, making them reliable for detecting the kind of schema composition or generic explosion that can silently degrade tsc performance.

Changes

  • scripts/typecheck-budget.ts — New script that:

    • Discovers all composite packages (packages/*/tsconfig.json, providers/*/tsconfig.json)
    • Warm-builds the full dependency graph with tsc -b to emit .d.ts files
    • Type-checks each package in isolation with tsc --extendedDiagnostics and extracts the Instantiations count
    • Compares against committed per-package budgets with configurable tolerance (default +15%)
    • Supports --update to re-baseline, --json to emit raw measurements, --no-build to skip warm build, and --tolerance to override
    • Exits 1 on regressions, reports improvements and new packages

  • scripts/typecheck-budget.json — Initial budget baseline with instantiation counts for all 34 packages (tolerance 15%, floor 50k instantiations)

  • docs/technical/typecheck-performance-investigation.md — Detailed investigation documenting:

    • Timeline of typecheck slowdown (11s → 40s over ~4 months)
    • Two culprit commits isolated by git bisect:
      • 16a94b54 (MCP allOf spreading): packages/tasks 289k → 6.77M instantiations (23×, already fixed)
      • 888da0e1 (task-runner refactor): packages/ai 0.9s → 3.4s check time (2×, durable)
    • Root-cause analysis and four prototyped fixes (all measured to be ineffective)
    • Rationale for the guardrail approach

  • .github/workflows/test.yml — New typecheck-budget CI job that runs before build, fails the PR on regressions

  • package.json — Added typecheck:budget script

Implementation notes

  • Uses Bun.Glob to discover packages (fast, no external deps)
  • Extracts Instantiations and Check time from tsc --extendedDiagnostics output via regex
  • Removes tsconfig.tsbuildinfo before each isolated check to force full re-check (deps already built)
  • Tolerance is fractional (0.15 = +15%); floor filters out trivially small packages to reduce noise
  • Budget file is JSON with per-package instantiation counts, making it easy to review diffs in PRs
  • Both culprit commits documented in the investigation would have failed the guard

This guardrail is stable across CI runners (instantiations are deterministic) and catches exactly the class of regression that caused the 3.6× slowdown documented in the investigation.

https://claude.ai/code/session_018SckHLTLmkxKCSWyb74Lke

@claude
ci: add typecheck instantiation-budget guard + perf investigation wri…
3112b58 
…teup

Adds scripts/typecheck-budget.ts, which type-checks each composite package in
isolation and gates its (deterministic, machine-independent) instantiation count
against committed per-package budgets in scripts/typecheck-budget.json. Wired
into CI as the `typecheck-budget` job and exposed as `bun run typecheck:budget`.

Also documents the typecheck slowdown investigation (timeline, bisected culprit
commits, root-cause analysis, and prototyped fixes) in
docs/technical/typecheck-performance-investigation.md.

https://claude.ai/code/session_018SckHLTLmkxKCSWyb74Lke
@sroussey sroussey mentioned this pull request Jun 9, 2026
Closed
3 tasks
@github-actions

github-actions Bot commented Jun 9, 2026 •
edited
Loading

Copy link
Copy Markdown

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 61.99% 24860 / 40102
🔵 Statements 61.83% 25716 / 41586
🔵 Functions 62.94% 4699 / 7465
🔵 Branches 50.71% 12198 / 24053
File CoverageNo changed files found.
Generated in workflow #2530 for commit eaae9e1 by the Vitest Coverage Report Action

claude added 2 commits June 9, 2026 03:29
@claude
perf(ai): replace FromSchema-derived types with concrete interfaces (…
03bb8cf 
…AI Text slice)

Proof-slice for the typecheck-cost work: the dominant `packages/ai` check cost is
the `FromSchema<typeof Schema>` conditional-type machinery (json-schema-to-ts),
not the Workflow/CreateWorkflow generics. Converting the plain
`type X = FromSchema<typeof Schema>` aliases to concrete types removes that
per-use instantiation cost.

This slice converts the AI Text task family + the foundational model types
(ModelConfig/ModelRecord). Each concrete type is the type checker's exact
resolution of the original FromSchema type (verified by a clean full `tsc -b`),
so it is equivalent to the prior derived type; the schema constants remain the
runtime contract. Budget baseline ratcheted to match.

Measured (pinned tsc, deps prebuilt, isolated `tsc -p`):
- ai: 564,545 -> 520,877 instantiations (this 7-file slice)
- full-ai conversion projects to ~320k instantiations / -39% check time,
  with a ~-14% ripple on packages/test (tracked in #556)

Tradeoff: a hot-path `Equal<X, FromSchema<...>>` drift guard is intentionally
omitted because it would recompute FromSchema and defeat the win; the schemas
stay the runtime source of truth.

Refs #556.

https://claude.ai/code/session_018SckHLTLmkxKCSWyb74Lke
@claude
perf(ai): convert remaining FromSchema-derived task types to concrete…
eaae9e1 
… types

Completes the FromSchema->concrete migration started in the AI Text slice.
Converts the remaining plain and composite (Omit<...> / WithImageValuePorts<...>)
type aliases across packages/ai to concrete types, removing the json-schema-to-ts
conditional-type instantiation cost from the AI task input/output types.

Each concrete type is the type checker's exact resolution of the original
FromSchema type (verified by a clean full `tsc -b`); the schema constants remain
the runtime contract. ModelConfig/ModelRecord and ChatMessage are referenced by
name rather than inlined for readability. Four vector/embedding outputs that
involve branded typed-array types (TextEmbedding, ImageEmbedding,
VectorSimilarity) are intentionally left on FromSchema since they do not print
to a clean concrete form.

Measured (pinned tsc, deps prebuilt, isolated `tsc -p`):
- ai:   564,545 -> 92,750 instantiations (-84%), 4.29s -> ~2.0s check (-52%)
- test (ripple): 1,427,138 -> 947,215 instantiations (-34%), 12.4s -> ~9.2s

Budget baselines ratcheted to match.

Closes #556.

https://claude.ai/code/session_018SckHLTLmkxKCSWyb74Lke
@sroussey sroussey linked an issue Jun 9, 2026 that may be closed by this pull request
Typecheck slowdown: durable packages/ai regression from 888da0e1 (core task/Workflow generics) #556
Closed
3 tasks
@sroussey sroussey merged commit 3b8075b into main Jun 9, 2026
14 checks passed
@sroussey sroussey deleted the claude/youthful-lovelace-14qjr5 branch June 9, 2026 04:18
@sroussey sroussey mentioned this pull request Jun 10, 2026
Closed
4 tasks
sroussey pushed a commit that referenced this pull request Jun 10, 2026
@claude
fix(ai): revert prompt-type ContentBlock substitutions; rely on drift…
4ca9121 
…-guard test

The H-1 substitutions in PR #558 caused two CI failures:

1. `build` — MessageConversion.ts:52,174 stopped compiling because the
   new `readonly` modifier on `prompt` broke `Array.isArray()` narrowing
   (TS doesn't narrow readonly arrays via Array.isArray).

2. `typecheck-budget` — packages/ai went 92,750 → 152,539 instantiations
   (+64%), defeating PR #555's perf goal. ContentBlock plus FromSchema
   references in types.test-d.ts together blew the budget.

PR #555 verified the inline literals are byte-equal to FromSchema's
resolution, so the substitution was gratuitous tightening, not a safety
win. The schema-vs-type drift risk is still real but addressed by the
H-3 nightly drift guard.

Changes:
- Restore original inline `prompt` literal in AiChatTask.ts (ContentBlock
  import stays — used elsewhere).
- Restore original inline `prompt` literal in ToolCallingTask.ts; drop
  unused ContentBlock import.
- Simplify types.test-d.ts to assert
  expectTypeOf<X["prompt"]>().toEqualTypeOf<FromSchema<typeof S>["prompt"]>()
  for AiChat + ToolCalling (now true). Keep .not.toEqualTypeOf for
  ChunkRetrieval (intentional if/then/else divergence).
- Exclude src/**/*.test-d.ts from packages/ai/tsconfig.json so
  typecheck:budget doesn't measure FromSchema cost in the test file.
  Nightly workflow's --typecheck engine is unaffected.

MessageConversion.ts is unchanged — narrowing works again with the
restored mutable arrays.
sroussey added a commit that referenced this pull request Jun 12, 2026
@sroussey
chore: release 0.3.14 for 36 packages
036cb19 
## @workglow/browser-control

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/task-graph

### Features

- add bugs URL to package.json files across all packages and providers

### Bug Fixes

#### task-graph,storage

- cache restart-resume + SharedInMemory sync barrier (#552)

### Documentation

#### task-graph

- fix TaskOutputTabularRepository README examples for new constructor signature

## @workglow/javascript

### Features

- add bugs URL to package.json files across all packages and providers

### Bug Fixes

- tsgo issue

## @workglow/ai

### Features

- add typecheck budget guard to catch type-instantiation regressions (#555)
- add bugs URL to package.json files across all packages and providers

### Bug Fixes

#### ai

- export ChunkRetrievalInputSchema + nightly schema-vs-type drift guard (#565)

## @workglow/knowledge-base

### Features

- add bugs URL to package.json files across all packages and providers

## workglow

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/storage

### Features

- add bugs URL to package.json files across all packages and providers

### Bug Fixes

#### task-graph,storage

- cache restart-resume + SharedInMemory sync barrier (#552)

## @workglow/mcp

### Features

- add bugs URL to package.json files across all packages and providers

### Bug Fixes

#### mcp

- thread run-scoped registry through discoverSchemas (#577)
- resolve auth credentials through the run-scoped registry

## @workglow/util

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/test

### Features

- add bugs URL to package.json files across all packages and providers

#### supabase

- add Supabase vector storage with pgvector support (#578)

### Bug Fixes

#### mcp

- thread run-scoped registry through discoverSchemas (#577)
- resolve auth credentials through the run-scoped registry

#### task-graph,storage

- cache restart-resume + SharedInMemory sync barrier (#552)

### Tests

- pin HF router provider for tool-calling conformance tests (#564)

### Chores

- update deps

### Updated Dependencies

- `@aws-sdk/client-sqs`: ^3.1068.0
- `@cloudflare/workers-types`: ^4.20260612.1
- `@types/dom-chromium-ai`: ^0.0.17
- `miniflare`: ^4.20260611.0

## @workglow/tasks

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/job-queue

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/indexeddb

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/openai

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/llamacpp-server

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/mlx

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/electron

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/ollama

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/node-llama-cpp

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/aws

### Features

- add bugs URL to package.json files across all packages and providers

### Chores

- update deps

### Updated Dependencies

- `@aws-sdk/client-sqs`: ^3.1068.0

## @workglow/anthropic

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/google-gemini

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/postgres

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/stable-diffusion-server

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/supabase

### Features

- add bugs URL to package.json files across all packages and providers

#### supabase

- add Supabase vector storage with pgvector support (#578)

## @workglow/playwright

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/sqlite

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/cloudflare

### Features

- add bugs URL to package.json files across all packages and providers

### Chores

- update deps

### Updated Dependencies

- `@cloudflare/workers-types`: ^4.20260612.1

## @workglow/huggingface-transformers

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/tf-mediapipe

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/chrome-ai

### Features

- add bugs URL to package.json files across all packages and providers

### Chores

- update deps

### Updated Dependencies

- `@types/dom-chromium-ai`: ^0.0.17

## @workglow/huggingface-inference

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/cactus

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/bun-webview

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/cli

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/web

### Features

- add bugs URL to package.json files across all packages and providers

### Chores

- update deps

### Updated Dependencies

- `@tailwindcss/vite`: ^4.3.1
- `tailwindcss`: ^4.3.1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Typecheck slowdown: durable packages/ai regression from 888da0e1 (core task/Workflow generics)

2 participants

@sroussey @claude