Add vitest plugin for testing full workflows without setting up a server#1237
Add vitest plugin for testing full workflows without setting up a server#1237VaguelySerious wants to merge 46 commits intomainfrom
Conversation
- Add new packages/vitest package with Vitest plugin that reuses @workflow/rollup - Add workbench/vitest app with simple workflow code and unit tests - Add GitHub CI workflow step for vitest plugin tests - Add vitest workbench to changeset ignore list
- Step functions (add, multiply) now tested with actual execution and assertions - Workflow function tested for workflowId property and expected throw behavior - Removed greetWorkflow, simplified to single calculateWorkflow example
- Add test server (src/server.ts) that loads compiled workflow bundles - Update tests to use start() to invoke workflows and verify results - Tests verify calculateWorkflow(2, 7) returns correct sum/product/combined - Build step uses 'wf build' to compile workflow bundles - Add .gitignore for generated files
…ckage - Delete packages/vitest (not needed as separate package) - Update vitest.config.ts to use workflowTransformPlugin from @workflow/rollup - Remove changeset entry for vitest package - Remove duplicate test case
- Use globalSetup to start Nitro dev server before tests - Use workflow/vite plugin for transformations - Tests directly use start() and await run.returnValue - Remove custom server.ts and build step - Much simpler setup matching the sdk-with-workflows example
The vitest workbench spawns a Nitro server which doesn't work in the regular unit test CI context. It has its own dedicated CI job.
- Add testing to docs navigation - Create testing/index.mdx with setup instructions - Link to workbench/vitest example files
Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Signed-off-by: Cursor Agent <cursoragent@cursor.com>
Create new 'testing/' section and update Foundations index page. Slack-Thread: https://vercel.slack.com/archives/C09125LC4AX/p1772375177894629?thread_ts=1772375177.894629&cid=C09125LC4AX Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
Slack-Thread: https://vercel.slack.com/archives/C09125LC4AX/p1772375177894629?thread_ts=1772375177.894629&cid=C09125LC4AX Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
Slack-Thread: https://vercel.slack.com/archives/C09125LC4AX/p1772375177894629?thread_ts=1772375177.894629&cid=C09125LC4AX Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
Update documentation to emphasize unit testing for steps only. Slack-Thread: https://vercel.slack.com/archives/C09125LC4AX/p1772375177894629?thread_ts=1772375177.894629&cid=C09125LC4AX Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
Move testing to top-level, fix content and update meta files. Slack-Thread: https://vercel.slack.com/archives/C09125LC4AX/p1772375177894629?thread_ts=1772375177.894629&cid=C09125LC4AX Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
Add 'globalSetup' script and vitest config, and update docs for test debugging. Slack-Thread: https://vercel.slack.com/archives/C09125LC4AX/p1772375177894629?thread_ts=1772375177.894629&cid=C09125LC4AX Co-authored-by: Pranay Prakash <1797812+pranaygp@users.noreply.github.com>
… sleeps Exposes the existing `wakeUpRun` logic as an instance method on the `Run` class, allowing users to skip sleep() calls in tests or custom UIs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix JSDoc @returns to describe StopSleepResult object instead of number - Clean up global world state in afterEach to prevent test leakage - Add delay before wakeUp() in docs example to allow workflow replay Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The file was removed on main but still exists on the base branch. Restoring it so this PR's diff only contains relevant changes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The docs test type-checks each code block independently, so the standalone wakeUp snippet needs its own imports and run variable. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use the // @setup comment pattern to provide type context without rendering the setup line, keeping the snippet lean. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
🦋 Changeset detectedLatest commit: cc79cd0 The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results❌ Some tests failed Summary
❌ Failed Tests🌍 Community Worlds (48 failed)turso (48 failed):
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
❌ 🌍 Community Worlds
✅ 📋 Other
|
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
docs/content/docs/testing/meta.json
Outdated
| { | ||
| "title": "Testing", | ||
| "pages": ["testing"] | ||
| "pages": ["testing", "server-based"], |
There was a problem hiding this comment.
is this correct, or does it need to be "index"?
| prerequisites: | ||
| - /docs/testing | ||
| - /docs/foundations/workflows-and-steps | ||
| related: | ||
| - /docs/testing | ||
| - /docs/getting-started/vite |
There was a problem hiding this comment.
nit: I don't think we actually do anything with these rn? I think claude hallucinated this front matter.
Would be nice to actually use them in the future though. Maybe a geistdocs thing
|
|
||
| The [Vitest plugin](/docs/testing#integration-testing-with-the-vitest-plugin) runs workflows entirely in-process and is the recommended approach for most testing scenarios. However, there are cases where you may want to test against a running server: | ||
|
|
||
| - Testing the full HTTP layer (middleware, authentication, request handling) |
There was a problem hiding this comment.
I think the server example wouldn't actually do this since it wouldn't reuse the http layer of the actual app the user is going to productionize against no? the nitro sidecar is in itself a separate server
I still see the value of it if people want to test against an actual server and then they can customize the nitro server to match their setup?
It's maybe at least worth noting that, this i an example with a nitro server for a full simulated test and is meant to be a starting point for users to further customize for their own testing needs
docs/content/docs/testing/index.mdx
Outdated
|
|
||
| 1. **Unit testing** - Test individual steps as plain functions, without the workflow runtime. | ||
| 2. **Integration testing** - Test entire workflows against a real workflow setup using the Vite plugin. Required for workflows that use [hooks](/docs/foundations/hooks), webhooks, [`sleep()`](/docs/api-reference/workflow/sleep), or retries. | ||
| 2. **Integration testing** - Test entire workflows in-process using the `workflow()` Vitest plugin. Required for workflows that use [hooks](/docs/foundations/hooks), webhooks, [`sleep()`](/docs/api-reference/workflow/sleep), or retries. |
There was a problem hiding this comment.
| 2. **Integration testing** - Test entire workflows in-process using the `workflow()` Vitest plugin. Required for workflows that use [hooks](/docs/foundations/hooks), webhooks, [`sleep()`](/docs/api-reference/workflow/sleep), or retries. | |
| 2. **Integration testing** - Test entire workflows in-process using the `workflow()` Vitest plugin. Required when you want to test workflow specific code paths, like those using [hooks](/docs/foundations/hooks), webhooks, [`sleep()`](/docs/api-reference/workflow/sleep), retries, etc. |
(metadata, locks, etc. are all relevant here but we dont' need to list everything)
| <Callout type="warn"> | ||
| Inside integration tests, which run the full workflow runtime, `vi.mock()` and related calls do not work. To test steps with mocked step dependencies, use [unit tests](#unit-testing-steps) instead. Instead of using mocks in integration tests, consider dependency injection or environment variable-based conditional logic. | ||
| </Callout> |
There was a problem hiding this comment.
I don't understand this? why would mocks not work in the vitest runtime? is this not a deal breaker?
There was a problem hiding this comment.
It doesn't work for first-party code, but it might work for third-party dependencies. We should extend the vitest workbench app to add an import for an npm package and then mock it out, and then test whether the mock works. If yes, we should update this Callout to be more specific. If not, let's document that case too in the markdown file inside the vitest workbench.
docs/content/docs/testing/index.mdx
Outdated
|
|
||
| 1. Transforms `"use workflow"` and `"use step"` directives via SWC | ||
| 2. Builds workflow and step bundles before tests run | ||
| 3. Sets up an in-process workflow runtime using a fresh [Local World](/docs/worlds/local) instance in each test worker |
There was a problem hiding this comment.
question: do we clean up local filesystem across individual tests? or across test runs? etc.
in other words, what's the test isolation story?
for example, a hook token won't be reusable if you have a test run that didn't complete in the past right (remember that in local world, you could Ctrl+C a run midway, and it won't resume automatically in future restarts)
maybe we should clean up the local world data directory on test start/(i.e. before all)? That way the runs are still observable after a test run is complete, but each run is isolated
There was a problem hiding this comment.
(if you've already implemented this, ignore me. I'm just reading the docs rn - not the code yet).
consider any doc updates based on my questions as you see fit :)
| console.error("Failed to start server:", error); | ||
| clearTimeout(timeout); | ||
| resolve(false); | ||
| const result = await run.returnValue; // [!code highlight] |
There was a problem hiding this comment.
| const result = await run.returnValue; // [!code highlight] | |
| // waiting for the return value will block till the run completes (fails or succeeds) | |
| const result = await run.returnValue; // [!code highlight] | |
| const status = await run.status; | |
| expect(status).toEqual("completed"); |
(suggestion written by hand - pls validate types/typos when incorporating)
docs/content/docs/testing/index.mdx
Outdated
|
|
||
| let server: ChildProcess | null = null; | ||
| const PORT = "4000"; | ||
| Use [`start()`](/docs/api-reference/workflow-api/start) to trigger a workflow and `await run.returnValue` to get the result: |
There was a problem hiding this comment.
let's cross link to run.returnValue docs in API reference? (so people/agents can figure out that it will block till ready, and throw if the workflow fails, etc.)
| // waitForSleep returns the sleep's correlation ID, which can be | ||
| // passed to wakeUp() to target a specific sleep in the workflow. | ||
| const sleepId = await waitForSleep(run); // [!code highlight] | ||
| await getRun(run.runId).wakeUp({ correlationIds: [sleepId] }); // [!code highlight] |
There was a problem hiding this comment.
comment here that calling wakeUp() without correlation IDs will resume all/any active sleeps at this time
| // Wait for the sleep to be reached, then skip it. | ||
| // waitForSleep returns the sleep's correlation ID, which can be | ||
| // passed to wakeUp() to target a specific sleep in the workflow. | ||
| const sleepId = await waitForSleep(run); // [!code highlight] |
There was a problem hiding this comment.
question: does waitForSleep return immediately if there are previous sleeps already active?
psudeocode:
startRun that triggers 2 sleeps in parallel
waitFor Sleep().. (how many sleeps will it return? 1 or both?)
wakeUp(1 of the sleeps)
waitForSleep() (will this immediately return the other sleep pending?
wakeUp(the other sleep)
waitForSleep() (will this block indefinitely now since all sleeps are resolved?
docs/content/docs/testing/index.mdx
Outdated
|
|
||
| ### Testing Webhooks | ||
|
|
||
| Webhooks are hooks that receive HTTP `Request` objects. In tests, resume them using `resumeHook()` with a `Request` payload — no HTTP server needed: |
There was a problem hiding this comment.
we have resumeWebhook actually and we should use that. I think this is the only valid usecase for resumeWebhook.
- it's already type safe
- it will correctly only resume webhooks (and not normal hook)
docs/content/docs/testing/index.mdx
Outdated
| - [`start()` API Reference](/docs/api-reference/workflow-api/start) - Start workflows programmatically | ||
| - [`resumeHook()` API Reference](/docs/api-reference/workflow-api/resume-hook) - Resume hooks with data | ||
| - [`getRun()` API Reference](/docs/api-reference/workflow-api/get-run) - Check workflow run status and wake up sleeping runs | ||
| - [`waitForSleep()` and `waitForHook()`](#testing-hooks-and-waits) - Test helpers for polling workflow state |
There was a problem hiding this comment.
can we please add an API reference section for the @workflow/vitest package that includes API reference sections for waitForSleep and waitForHook
at least so that the TSDoc generation works and those pages have something populated
| ## Debugging Test Runs | ||
|
|
||
| When integration tests fail, the [Workflow DevKit CLI and Web UI](/docs/observability) can help you inspect what happened. Because integration tests persist workflow state to `WORKFLOW_LOCAL_DATA_DIR`, you can use the same observability tools you would use in development. | ||
| When integration tests fail, the [Workflow DevKit CLI and Web UI](/docs/observability) can help you inspect what happened. Because integration tests persist workflow state locally, you can use the same observability tools you would use in development. |
| @@ -0,0 +1,3 @@ | |||
| import { setupWorkflowTests } from './index.js'; | |||
|
|
|||
| await setupWorkflowTests(); | |||
There was a problem hiding this comment.
setup-file.ts calls setupWorkflowTests() but never calls teardownWorkflowTests(). The manual setup example in the docs shows afterAll(() => teardownWorkflowTests()), but the automatic path skips this. This could leak resources or let state bleed between test files in a multi-file suite.
Consider adding teardown registration here:
import { setupWorkflowTests, teardownWorkflowTests } from "./index.js";
import { afterAll } from "vitest";
await setupWorkflowTests();
afterAll(async () => {
await teardownWorkflowTests();
});| ); | ||
| const stepsModule = await import( | ||
| /* @vite-ignore */ join(outDir, 'steps.mjs') | ||
| ); |
There was a problem hiding this comment.
The data directory at join(outDir, "data") persists across test runs. Events and hooks from previous runs remain on disk, which could cause waitForSleep / waitForHook to match stale events from a prior run if the same runId prefix collides. Consider clearing or recreating the data directory at the start of setupWorkflowTests (or in buildWorkflowTests) to guarantee test isolation.
| "@types/node": "catalog:", | ||
| "@workflow/tsconfig": "workspace:*", | ||
| "vite": "7.1.12" | ||
| } |
There was a problem hiding this comment.
The plugin returns Vite Plugin[] and the test helpers use types from @workflow/core, but vite and vitest are not listed as peerDependencies. Users need compatible versions for the plugin to work. Consider adding:
"peerDependencies": {
"vite": ">=6.0.0",
"vitest": ">=3.0.0"
}(with appropriate version ranges for your supported matrix)
| ├── nitro.config.ts # Nitro config with workflow module | ||
| ├── vitest.config.ts # Vitest config with workflow plugin | ||
| └── vitest.setup.ts # Global setup to start Nitro server | ||
| ``` |
There was a problem hiding this comment.
This README appears stale — it references a Nitro server-based approach with vitest.setup.ts (global setup) and a nitro.config.ts-centric project structure, but the workbench actually uses the new in-process @workflow/vitest plugin. The project structure section lists files that do not exist (e.g. vitest.setup.ts) and omits real ones (workflows/hooks.ts, workflows/sleeping.ts, workflows/webhook.ts). Worth updating to match the actual setup.
|
|
||
| /** Direct in-process handlers by queue prefix, bypassing HTTP when set. */ | ||
| const directHandlers = new Map<string, DirectHandler>(); | ||
|
|
There was a problem hiding this comment.
This assumes any queue name that does not start with __wkf_step_ must be __wkf_workflow_. If the queue name has a different prefix entirely, it will still look up __wkf_workflow_ in the direct handlers map and likely get undefined, which happens to fall through to the HTTP path — so it works by accident. But the intent would be clearer with explicit checks:
const prefix = queueName.startsWith("__wkf_step_")
? "__wkf_step_"
: queueName.startsWith("__wkf_workflow_")
? "__wkf_workflow_"
: undefined;
const directHandler = prefix ? directHandlers.get(prefix) : undefined;
TooTallNate
left a comment
There was a problem hiding this comment.
Review Summary
Great feature — running full workflow integration tests in-process without a server is a significant DX improvement. The core approach of registering direct handlers on the local world queue is elegant and minimal.
What looks good
- Clean architecture:
globalSetupbuilds bundles,setupFilewires up in-process handlers viaregisterHandler() waitForSleep()andwaitForHook()test helpers are well-designed with sensible polling + timeout defaults- Docs rewrite is excellent — leading with the simpler in-process approach and moving server-based to its own page
- Honest callout about
vi.mock()limitations, with thorough internal analysis inMOCKING.md - CI properly isolates the vitest workbench tests into their own job
- Good test coverage in the workbench (simple, sleeping, hooks, webhooks)
Items to address
- Missing teardown in
setup-file.ts— resources/state may leak between test files - Prefix detection in
queue.ts— falls through to__wkf_workflow_for unknown queue names (works by accident) - Stale data directory —
.workflow-vitest/datapersists between runs, risking event collisions - Missing peer dependencies —
viteandvitestshould be peer deps in@workflow/vitest - Stale workbench README — references Nitro server approach and files that no longer exist
Minor / non-blocking
resolveBaseUrlmoved inside the retry loop for the HTTP path (was previously hoisted)- Incidental formatting changes in
.changeset/config.json undici7.16.0 -> 7.22.0 bump in lockfile (harmless but unrelated)
Overall this is close to ready. The teardown and data isolation issues are the most important to address before merge.
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
- Add afterAll teardown to setup-file.ts (TooTallNate) - Add vite/vitest peerDependencies to @workflow/vitest (TooTallNate) - Fix queue.ts prefix check to be explicit (TooTallNate) - Update stale workbench README to match current setup (TooTallNate) - Switch webhook test to use resumeWebhook (pranaygp) - Fix meta.json page reference from "testing" to "index" (pranaygp) - Update integration test description (pranaygp) - Document test isolation: world data cleared on setup (pranaygp) - Add run.status example and cross-link to run.returnValue (pranaygp) - Document wakeUp without correlationIds behavior (pranaygp) - Document waitForSleep parallel/sequential behavior (pranaygp) - Use resumeWebhook in webhook docs example (pranaygp) - Add @workflow/vitest API reference page (pranaygp) - Clean up server-based.mdx: remove hallucinated frontmatter, add hooks/waits info, note about customization (pranaygp) - Fix prose references to workflow/vitest → @workflow/vitest (vercel[bot]) - Test third-party mocking: confirmed vi.mock does not work for npm packages either, updated MOCKING.md and callout (VaguelySerious) - Fix per-worker data directory isolation (VITEST_POOL_ID) - Add changesets for @workflow/vitest and @workflow/world-local Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The docs-typecheck project doesn't have @workflow/vitest installed, so code samples using its exports fail type checking. Add @skip-typecheck annotations to those blocks. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ith "Unexpected character `!` (U+0021) before name" error.
This commit fixes the issue reported at docs/content/docs/api-reference/vitest/index.mdx:88
**Bug:** The file `docs/content/docs/api-reference/vitest/index.mdx` uses HTML-style comments (`<!-- @skip-typecheck ... -->`) on lines 88 and 117. MDX does not support HTML comment syntax. When the MDX compiler encounters `<!--`, it sees `!` after `<` and expects a valid JSX element name (starting with a letter, `$`, or `_`), causing a parse error.
The build log from deployment `dpl_AcUbRm42SaELfM8UKd2nrVPDsxEE` confirms this:
```
docs:build: 84:2: index.mdx:84:2: Unexpected character `!` (U+0021) before name, expected a character that can start a name, such as a letter, `$`, or `_` (note: to create a comment in MDX, use `{/* text */}`)
```
This causes the entire docs build to fail (`docs#build` is listed as the failed task).
**Fix:** Replaced both HTML-style comments with the correct MDX/JSX comment syntax:
- `<!-- @skip-typecheck - @workflow/vitest not available in docs-typecheck -->` → `{/* @skip-typecheck - @workflow/vitest not available in docs-typecheck */}`
This is consistent with how all other MDX files in the `docs/content` directory handle `@skip-typecheck` comments (e.g., `docs/content/docs/how-it-works/understanding-directives.mdx` uses `{/* @skip-typecheck: ... */}` throughout).
Co-authored-by: Vercel <vercel[bot]@users.noreply.github.com>
Co-authored-by: VaguelySerious <mittgfu@gmail.com>
Superseeds #822. This allows users to add a vitest plugin enabling running of a workflow without a server instance, so you can e2e test a full workflow with given inputs.
Also introduces test helpers for "expect and skip next wait" and "expect and resolve next hook with ".
New docs link
Note this doesn't allow mocking due to how SWC bundles the step/run separately. There's a callout for that and it's already not supported in the existing docs/setup. I added a separate markdown file to the workbench with what Claude thinks is the way forward for that, and my personal take is that it's hard and we should not worry about it right now