Skip to content

feat(registry): add experimental HTML-in-canvas launch blocks#611

Open
miguel-heygen wants to merge 7 commits intomainfrom
feat/html-in-canvas-launch
Open

feat(registry): add experimental HTML-in-canvas launch blocks#611
miguel-heygen wants to merge 7 commits intomainfrom
feat/html-in-canvas-launch

Conversation

@miguel-heygen
Copy link
Copy Markdown
Collaborator

@miguel-heygen miguel-heygen commented May 3, 2026

Problem

Chrome/Brave now expose the experimental CanvasDrawElement / HTML-in-canvas API, but HyperFrames did not have a launch-ready catalog story for it. The raw capability is easy to dismiss as a tech demo unless creators can install a block that turns one editable product UI into useful launch/tutorial shots.

hyperframes validate also needed to use the same browser args as the renderer so WebGL and CanvasDrawElement compositions are validated against the right environment.

What this fixes

  • Adds native drawElementImage() capture support for @hyperframes/shader-transitions, with the existing fallback path preserved.
  • Enables CanvasDrawElement in renderer Chrome args and reuses renderer Chrome args from hyperframes validate.
  • Adds registry stability: "experimental" metadata, schema/type/catalog/CLI surfacing, and generated docs warnings with agent setup instructions.
  • Marks canvas-dom-refractor as experimental.
  • Adds experimental-screen-callout, the simple workflow proof for one DOM screen -> zoom lens, spotlight, cursor trail, annotations, and crop strip.
  • Adds experimental-product-launch-portal, the flagship creator block: one editable product UI becomes shadered portal warps, device panels, crop callouts, overload glitch beats, and a final launch CTA from the same HTML-in-canvas source.

Root cause

The project had shader and catalog infrastructure, but not a native browser capture path or a creator-facing component that made the API useful. CanvasDrawElement needs a layout-subtree canvas, requestPaint()/paint lifecycle, and drawElementImage() working together; the catalog blocks now keep that native path gated and preserve DOM fallbacks for browsers without the experimental API.

Verification

Local checks

  • bun run --filter @hyperframes/shader-transitions test
  • bun run --filter @hyperframes/shader-transitions typecheck
  • bun run --filter @hyperframes/shader-transitions build
  • bun run --filter @hyperframes/engine test src/services/browserManager.test.ts
  • bun run --filter @hyperframes/engine typecheck
  • bun run --filter @hyperframes/core test src/registry/types.test.ts
  • bun run --filter @hyperframes/core typecheck
  • bun run --filter @hyperframes/cli typecheck
  • bunx oxlint packages/cli/src/commands/catalog.ts packages/core/src/registry/types.ts packages/core/src/registry/index.ts packages/core/src/index.ts packages/core/src/registry/types.test.ts scripts/generate-catalog-pages.ts
  • bun run scripts/sync-schemas.ts --check
  • bun run scripts/generate-catalog-pages.ts
  • bunx oxfmt --check registry/blocks/experimental-product-launch-portal/registry-item.json registry/blocks/experimental-product-launch-portal/experimental-product-launch-portal.html registry/registry.json docs/docs.json docs/public/catalog-index.json docs/catalog/blocks/experimental-product-launch-portal.mdx
  • git diff --check
  • Commit hooks passed format and commitlint.

Catalog validation

  • bun run --filter @hyperframes/cli dev -- lint /Users/miguel07code/.codex/worktrees/html-in-canvas-launch/hyperframes-oss/.codex-artifacts/catalog-experimental-screen-callout -> 0 errors, 1 expected large-composition warning.
  • bun run --filter @hyperframes/cli dev -- validate /Users/miguel07code/.codex/worktrees/html-in-canvas-launch/hyperframes-oss/.codex-artifacts/catalog-experimental-screen-callout --no-contrast --timeout 6000 -> no console errors.
  • Same screen-callout validate with Chrome Canary/Brave native browser + renderer GPU mode -> no console errors.
  • bun run --filter @hyperframes/cli dev -- lint /Users/miguel07code/.codex/worktrees/html-in-canvas-launch/hyperframes-oss/.codex-artifacts/catalog-experimental-product-launch-portal -> 0 errors, 1 expected large-composition warning.
  • bun run --filter @hyperframes/cli dev -- validate /Users/miguel07code/.codex/worktrees/html-in-canvas-launch/hyperframes-oss/.codex-artifacts/catalog-experimental-product-launch-portal --no-contrast --timeout 8000 --json -> ok: true.
  • Same product-launch-portal validate with Brave 147 + CanvasDrawElement + renderer GPU mode -> ok: true.
  • Determinism grep for product-launch-portal found no Date.now, unseeded Math.random, requestAnimationFrame, setInterval, or fetch(.

Browser verification

  • Used Chrome Canary/Brave native proof page for @hyperframes/shader-transitions: isHtmlInCanvasCaptureSupported() === true, drawElementImageCalls === 2, WebGL transition canvas active at 1920x1080.
  • Used agent-browser for canvas-dom-refractor fallback/native screenshots and recordings.
  • Used agent-browser for experimental-screen-callout fallback: native: false, drawElementCalls: 0, hasDrawElementImage: false, hasLayoutSubtree: false.
  • Used agent-browser for experimental-screen-callout native: native: true, drawElementCalls: 2, hasDrawElementImage: true, hasLayoutSubtree: true.
  • Used agent-browser for experimental-product-launch-portal fallback: native: false, ready: true, drawElementCalls: 0.
  • Product portal fallback screenshot: qa-artifacts/experimental-product-launch-portal/fallback-9.2.png.
  • Product portal fallback recording: qa-artifacts/experimental-product-launch-portal/fallback-flow.webm.
  • Used agent-browser with Brave 147 + --enable-features=CanvasDrawElement for product portal native: native: true, ready: true, drawElementCalls: 7, hasDrawElementImage: true, hasLayoutSubtree: true, data-native-canvas="true".
  • Product portal native screenshots: qa-artifacts/experimental-product-launch-portal/native-4.6-oriented.png, native-9.2-overload.png, native-14.2-oriented.png, native-17.2.png.
  • Product portal native recording: qa-artifacts/experimental-product-launch-portal/native-flow.webm.

Render validation

  • Rendered the original 30s local HTML-in-canvas launch composition with Chrome Canary/Brave native browser + renderer GPU mode: 900/900 frames, 1920x1080, 30fps, 30.000000s.
  • Rendered canvas-dom-refractor: 360/360 frames, 1920x1080, 30fps, 12.000000s.
  • Rendered experimental-screen-callout: 360/360 frames, 1920x1080, 30fps, 12.000000s.
  • Rendered experimental-product-launch-portal: 540/540 frames, 1920x1080, 30fps, 18.000000s.
  • Product portal rendered MP4: qa-artifacts/experimental-product-launch-portal/experimental-product-launch-portal.mp4.
  • Product portal extracted frames: qa-artifacts/experimental-product-launch-portal/frames/frame-4.6.png, frame-9.2.png, frame-14.2.png, frame-17.2.png.

Notes

  • Native HTML-in-canvas remains experimental and browser-gated. Stable browsers keep the fallback paths.
  • Generated docs warn users and agents to enable chrome://flags/#canvas-draw-element, restart the browser, and preserve fallback behavior.
  • Screenshots, recordings, proof pages, extracted frames, and rendered MP4s are local-only QA artifacts and are intentionally not committed.
  • The experimental catalog blocks are rich single-file launch demos. The catalog linter reports the expected file-size warning, but there are no lint errors or runtime console errors.

Comment thread packages/cli/src/commands/validate.ts Outdated
Comment thread packages/cli/src/commands/validate.ts
Comment thread packages/engine/src/services/browserManager.ts
Copy link
Copy Markdown
Collaborator

@jrusso1020 jrusso1020 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we update the version of chrome in our docker images to make sure it includes this feature?

Copy link
Copy Markdown
Collaborator Author

miguel-heygen commented May 3, 2026

should we update the version of chrome in our docker images to make sure it includes this feature?

Not as part of this PR. The current Docker/test paths install Debian chromium plus chrome-headless-shell@stable, and the GitHub preview workflows use stable Chrome. CanvasDrawElement is still Canary/experimental, so a normal stable Docker refresh will not guarantee this API.

This PR is intentionally fallback-safe: the renderer passes the flag, then runtime feature detection only uses native capture if layoutSubtree, requestPaint(), and drawElementImage() exist. Otherwise it stays on html2canvas.

For actually exercising/benchmarking the native path in Docker, we should do a follow-up that pins an opt-in Chrome for Testing/Canary build that has CanvasDrawElement, updates the internal render image the same way, and compares it against the current beginFrame path before making it the default.

@jrusso1020
Copy link
Copy Markdown
Collaborator

@miguel-heygen we may want to update the internal version's docker image as well

then we can test HTML to canvas screenshotting maybe for renders? to see if it's any faster than beginFrame as well

@miguel-heygen miguel-heygen force-pushed the feat/html-in-canvas-launch branch from 93323f6 to 201e4ce Compare May 3, 2026 22:37
@miguel-heygen miguel-heygen changed the base branch from next to main May 3, 2026 22:37
@miguel-heygen
Copy link
Copy Markdown
Collaborator Author

On the internal image: yes, but I would keep it as a follow-up rather than coupling it to this PR. This PR keeps stable/Docker renders fallback-safe. For native-path perf testing we should pin the same Chrome for Testing/Canary build in the external + internal render images, then benchmark HTML-in-canvas screenshotting against beginFrame before changing defaults.

@miguel-heygen miguel-heygen changed the title feat(shader-transitions): add native HTML-in-canvas capture feat(registry): add experimental HTML-in-canvas screen callout May 4, 2026
@miguel-heygen miguel-heygen changed the title feat(registry): add experimental HTML-in-canvas screen callout feat(registry): add experimental HTML-in-canvas launch blocks May 4, 2026
miguel-heygen added a commit that referenced this pull request May 4, 2026
Cinematic 3D spatial UI showcase using the HTML-in-Canvas API.
Live HTML panels rendered as WebGL textures via drawElementImage(),
floating in a Three.js scene with post-processing (bloom, DoF,
chromatic aberration, volumetric light, film grain, vignette),
glass edges, reflection plane, and GSAP-driven camera autopilot
derived from composition timing. Interactive orbit mode via spacebar.

Requires Chrome/Brave with CanvasDrawElement flag enabled.
Depends on PR #611 for engine --enable-features=CanvasDrawElement.
Cinematic 3D spatial UI showcase — live HTML panels as WebGL textures
via drawElementImage(), Three.js scene with bloom, DoF, chromatic
aberration, volumetric light, glass edges, reflection plane, and
camera autopilot derived from composition timing.
miguel-heygen added a commit that referenced this pull request May 6, 2026
Cherry-picked from feat/html-in-canvas-launch (PR #611):

- Enable --enable-features=CanvasDrawElement in Chrome browser args
  so HTML-in-canvas compositions render correctly
- Add native drawElementImage() capture path for shader transitions
  with existing fallback preserved
- Reuse renderer Chrome args in hyperframes validate for consistent
  WebGL/CanvasDrawElement environment
- Add capture.test.ts for the new shader transition capture path
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants