feat(ai-grok): video generation adapter for the grok-imagine video models#742
Open
tombeckenham wants to merge 3 commits into
Open
Conversation
…dels
Adds a grokVideo adapter to @tanstack/ai-grok for xAI's Imagine video
models (grok-imagine-video at $0.05/s, grok-imagine-video-1.5-preview at
$0.08/s) using the experimental generateVideo() jobs/polling
architecture: POST /v1/videos/generations to create, GET
/v1/videos/{request_id} to poll, hosted mp4 URL plus usage (billed
seconds + exact USD cost) on completion.
Sizing follows the grok-imagine aspect-ratio template ('16:9_720p' →
aspect_ratio/resolution); durations are 1-15 integer seconds;
image-to-video starting frames go through modelOptions.image. The
Imagine video endpoints are plain JSON (not in the OpenAI SDK), so the
adapter issues direct requests with an injectable fetch seam.
Closes #705.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
📝 WalkthroughWalkthroughAdds Grok Imagine video generation: model metadata and provider option types, a GrokVideoAdapter implementing create/poll/getUrl flows (with usage/cost mapping), factories and exports, tests, docs, and example app wiring. ChangesGrok Imagine Video Adapter
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related issues
Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
🚀 Changeset Version Preview7 package(s) bumped directly, 24 bumped as dependents. 🟥 Major bumps
🟨 Minor bumps
🟩 Patch bumps
|
|
View your CI Pipeline Execution ↗ for commit 49a7c09
☁️ Nx Cloud last updated this comment at |
@tanstack/ai
@tanstack/ai-anthropic
@tanstack/ai-client
@tanstack/ai-code-mode
@tanstack/ai-code-mode-skills
@tanstack/ai-devtools-core
@tanstack/ai-elevenlabs
@tanstack/ai-event-client
@tanstack/ai-fal
@tanstack/ai-gemini
@tanstack/ai-grok
@tanstack/ai-groq
@tanstack/ai-isolate-cloudflare
@tanstack/ai-isolate-node
@tanstack/ai-isolate-quickjs
@tanstack/ai-mcp
@tanstack/ai-ollama
@tanstack/ai-openai
@tanstack/ai-openrouter
@tanstack/ai-preact
@tanstack/ai-react
@tanstack/ai-react-ui
@tanstack/ai-solid
@tanstack/ai-solid-ui
@tanstack/ai-svelte
@tanstack/ai-utils
@tanstack/ai-vue
@tanstack/ai-vue-ui
@tanstack/openai-base
@tanstack/preact-ai-devtools
@tanstack/react-ai-devtools
@tanstack/solid-ai-devtools
commit: |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
docs/media/video-generation.md (1)
315-315:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winRemove
as anytype assertion from documentation code sample.The coding guidelines explicitly prohibit
astype-assertion casts in documentation code samples (exceptas const). Examples must type-check without type casts.Consider properly typing the
sizeparameter in the input validator or using a type guard instead:.inputValidator((data: { prompt: string; size?: '1280x720' | '720x1280' | '1792x1024' | '1024x1792'; duration?: number }) => data)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/media/video-generation.md` at line 315, Remove the "as any" cast on the size assignment in the docs sample and ensure the sample types-check without casts: update the .inputValidator signature to type the incoming data.size as the allowed union ('1280x720' | '720x1280' | '1792x1024' | '1024x1792') or add a short type guard that validates/normalizes data.size before assigning to size; reference the .inputValidator callback and the size property so the sample compiles without using "as any".Source: Coding guidelines
🧹 Nitpick comments (1)
packages/ai-grok/tests/video-adapter.test.ts (1)
1-1: ⚡ Quick winPlace this unit test alongside the source file per repo rule.
video-adapter.test.tscurrently lives underpackages/ai-grok/tests/, but the guideline requires*.test.tsfiles to be colocated with source. Please move it next to the adapter/provider source it validates (for example underpackages/ai-grok/src/...).As per coding guidelines, "
**/*.test.ts: Place unit tests alongside source code in*.test.tsfiles".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/ai-grok/tests/video-adapter.test.ts` at line 1, The test file video-adapter.test.ts must be moved from packages/ai-grok/tests/ to be colocated with the adapter/provider source it validates (e.g., under packages/ai-grok/src/ alongside the video adapter implementation), update any relative import paths in video-adapter.test.ts to reflect the new location, and adjust any test-runner or build config references if they rely on the old tests/ directory; locate the adapter/provider source by name (video-adapter, VideoAdapter, or similar) and place the test file next to that module so imports and module resolution remain correct.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/ai-grok/src/adapters/video.ts`:
- Around line 151-208: The validation calls (validateVideoSize,
validateVideoDuration) in createVideoJob can throw before the existing
try/catch, so wrap those validations and the rest of the function body inside
the same try block (or add an outer try that encloses validations) so any thrown
validation errors are caught; in the catch call logger.errors with
toRunErrorPayload and source `${this.name}.createVideoJob` (same shape as
current catch) and rethrow the error to preserve behavior.
---
Outside diff comments:
In `@docs/media/video-generation.md`:
- Line 315: Remove the "as any" cast on the size assignment in the docs sample
and ensure the sample types-check without casts: update the .inputValidator
signature to type the incoming data.size as the allowed union ('1280x720' |
'720x1280' | '1792x1024' | '1024x1792') or add a short type guard that
validates/normalizes data.size before assigning to size; reference the
.inputValidator callback and the size property so the sample compiles without
using "as any".
---
Nitpick comments:
In `@packages/ai-grok/tests/video-adapter.test.ts`:
- Line 1: The test file video-adapter.test.ts must be moved from
packages/ai-grok/tests/ to be colocated with the adapter/provider source it
validates (e.g., under packages/ai-grok/src/ alongside the video adapter
implementation), update any relative import paths in video-adapter.test.ts to
reflect the new location, and adjust any test-runner or build config references
if they rely on the old tests/ directory; locate the adapter/provider source by
name (video-adapter, VideoAdapter, or similar) and place the test file next to
that module so imports and module resolution remain correct.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: b6eef03c-6d23-44d3-b96b-75cc68878c1b
📒 Files selected for processing (10)
.changeset/grok-imagine-video-adapter.mddocs/adapters/grok.mddocs/config.jsondocs/media/video-generation.mdpackages/ai-grok/src/adapters/video.tspackages/ai-grok/src/index.tspackages/ai-grok/src/model-meta.tspackages/ai-grok/src/video/video-provider-options.tspackages/ai-grok/tests/video-adapter.test.tspackages/ai/skills/ai-core/media-generation/SKILL.md
Comment on lines
+151
to
+208
| async createVideoJob( | ||
| options: VideoGenerationOptions<GrokVideoProviderOptions>, | ||
| ): Promise<VideoJobResult> { | ||
| const { model, prompt, size, modelOptions, logger } = options | ||
|
|
||
| validateVideoSize(model, size) | ||
| validateVideoDuration(model, options.duration) | ||
| validateVideoDuration(model, modelOptions?.duration) | ||
| const duration = options.duration ?? modelOptions?.duration | ||
|
|
||
| // The generic `size` option carries an "aspectRatio_resolution" template | ||
| // (e.g. '16:9_720p') and maps to the Imagine API's `aspect_ratio` / | ||
| // `resolution` parameters; explicit modelOptions win over the template. | ||
| const parsedSize = size !== undefined ? parseGrokVideoSize(size) : undefined | ||
| const request = { | ||
| model, | ||
| prompt, | ||
| ...(parsedSize && { | ||
| aspect_ratio: parsedSize.aspectRatio, | ||
| ...(parsedSize.resolution !== undefined && { | ||
| resolution: parsedSize.resolution, | ||
| }), | ||
| }), | ||
| ...(duration !== undefined && { duration }), | ||
| ...modelOptions, | ||
| } | ||
|
|
||
| try { | ||
| logger.request( | ||
| `activity=video.create provider=${this.name} model=${model} size=${size ?? 'default'} duration=${duration ?? 'default'}`, | ||
| { provider: this.name, model }, | ||
| ) | ||
|
|
||
| const response = await this.request('/videos/generations', { | ||
| method: 'POST', | ||
| body: JSON.stringify(request), | ||
| }) | ||
| if (!response.ok) { | ||
| throw new Error( | ||
| `grok: video generation request failed (${response.status} ${response.statusText}): ${await this.errorMessage(response)}`, | ||
| ) | ||
| } | ||
|
|
||
| const result = (await response.json()) as GrokVideoCreateResponse | ||
| if (!result.request_id) { | ||
| throw new Error( | ||
| 'grok: video generation response contained no request_id', | ||
| ) | ||
| } | ||
| return { jobId: result.request_id, model } | ||
| } catch (error: unknown) { | ||
| logger.errors(`${this.name}.createVideoJob fatal`, { | ||
| error: toRunErrorPayload(error, `${this.name}.createVideoJob failed`), | ||
| source: `${this.name}.createVideoJob`, | ||
| }) | ||
| throw error | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
Verify that validation errors before the API call are properly logged.
The validation calls validateVideoSize and validateVideoDuration (lines 156-158) can throw errors before the try block starts at line 178. If validation fails, the error bypasses the catch block's logger, so validation failures won't be logged with the structured error format.
Consider wrapping the entire function body in the try-catch or adding a separate catch wrapper:
🛡️ Proposed fix to ensure validation errors are logged
async createVideoJob(
options: VideoGenerationOptions<GrokVideoProviderOptions>,
): Promise<VideoJobResult> {
+ try {
const { model, prompt, size, modelOptions, logger } = options
validateVideoSize(model, size)
validateVideoDuration(model, options.duration)
validateVideoDuration(model, modelOptions?.duration)
const duration = options.duration ?? modelOptions?.duration
// The generic `size` option carries an "aspectRatio_resolution" template
// (e.g. '16:9_720p') and maps to the Imagine API's `aspect_ratio` /
// `resolution` parameters; explicit modelOptions win over the template.
const parsedSize = size !== undefined ? parseGrokVideoSize(size) : undefined
const request = {
model,
prompt,
...(parsedSize && {
aspect_ratio: parsedSize.aspectRatio,
...(parsedSize.resolution !== undefined && {
resolution: parsedSize.resolution,
}),
}),
...(duration !== undefined && { duration }),
...modelOptions,
}
- try {
logger.request(
`activity=video.create provider=${this.name} model=${model} size=${size ?? 'default'} duration=${duration ?? 'default'}`,
{ provider: this.name, model },
)
const response = await this.request('/videos/generations', {
method: 'POST',
body: JSON.stringify(request),
})
if (!response.ok) {
throw new Error(
`grok: video generation request failed (${response.status} ${response.statusText}): ${await this.errorMessage(response)}`,
)
}
const result = (await response.json()) as GrokVideoCreateResponse
if (!result.request_id) {
throw new Error(
'grok: video generation response contained no request_id',
)
}
return { jobId: result.request_id, model }
- } catch (error: unknown) {
+ } catch (error: unknown) {
logger.errors(`${this.name}.createVideoJob fatal`, {
error: toRunErrorPayload(error, `${this.name}.createVideoJob failed`),
source: `${this.name}.createVideoJob`,
})
throw error
- }
+ }
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| async createVideoJob( | |
| options: VideoGenerationOptions<GrokVideoProviderOptions>, | |
| ): Promise<VideoJobResult> { | |
| const { model, prompt, size, modelOptions, logger } = options | |
| validateVideoSize(model, size) | |
| validateVideoDuration(model, options.duration) | |
| validateVideoDuration(model, modelOptions?.duration) | |
| const duration = options.duration ?? modelOptions?.duration | |
| // The generic `size` option carries an "aspectRatio_resolution" template | |
| // (e.g. '16:9_720p') and maps to the Imagine API's `aspect_ratio` / | |
| // `resolution` parameters; explicit modelOptions win over the template. | |
| const parsedSize = size !== undefined ? parseGrokVideoSize(size) : undefined | |
| const request = { | |
| model, | |
| prompt, | |
| ...(parsedSize && { | |
| aspect_ratio: parsedSize.aspectRatio, | |
| ...(parsedSize.resolution !== undefined && { | |
| resolution: parsedSize.resolution, | |
| }), | |
| }), | |
| ...(duration !== undefined && { duration }), | |
| ...modelOptions, | |
| } | |
| try { | |
| logger.request( | |
| `activity=video.create provider=${this.name} model=${model} size=${size ?? 'default'} duration=${duration ?? 'default'}`, | |
| { provider: this.name, model }, | |
| ) | |
| const response = await this.request('/videos/generations', { | |
| method: 'POST', | |
| body: JSON.stringify(request), | |
| }) | |
| if (!response.ok) { | |
| throw new Error( | |
| `grok: video generation request failed (${response.status} ${response.statusText}): ${await this.errorMessage(response)}`, | |
| ) | |
| } | |
| const result = (await response.json()) as GrokVideoCreateResponse | |
| if (!result.request_id) { | |
| throw new Error( | |
| 'grok: video generation response contained no request_id', | |
| ) | |
| } | |
| return { jobId: result.request_id, model } | |
| } catch (error: unknown) { | |
| logger.errors(`${this.name}.createVideoJob fatal`, { | |
| error: toRunErrorPayload(error, `${this.name}.createVideoJob failed`), | |
| source: `${this.name}.createVideoJob`, | |
| }) | |
| throw error | |
| } | |
| } | |
| async createVideoJob( | |
| options: VideoGenerationOptions<GrokVideoProviderOptions>, | |
| ): Promise<VideoJobResult> { | |
| try { | |
| const { model, prompt, size, modelOptions, logger } = options | |
| validateVideoSize(model, size) | |
| validateVideoDuration(model, options.duration) | |
| validateVideoDuration(model, modelOptions?.duration) | |
| const duration = options.duration ?? modelOptions?.duration | |
| // The generic `size` option carries an "aspectRatio_resolution" template | |
| // (e.g. '16:9_720p') and maps to the Imagine API's `aspect_ratio` / | |
| // `resolution` parameters; explicit modelOptions win over the template. | |
| const parsedSize = size !== undefined ? parseGrokVideoSize(size) : undefined | |
| const request = { | |
| model, | |
| prompt, | |
| ...(parsedSize && { | |
| aspect_ratio: parsedSize.aspectRatio, | |
| ...(parsedSize.resolution !== undefined && { | |
| resolution: parsedSize.resolution, | |
| }), | |
| }), | |
| ...(duration !== undefined && { duration }), | |
| ...modelOptions, | |
| } | |
| logger.request( | |
| `activity=video.create provider=${this.name} model=${model} size=${size ?? 'default'} duration=${duration ?? 'default'}`, | |
| { provider: this.name, model }, | |
| ) | |
| const response = await this.request('/videos/generations', { | |
| method: 'POST', | |
| body: JSON.stringify(request), | |
| }) | |
| if (!response.ok) { | |
| throw new Error( | |
| `grok: video generation request failed (${response.status} ${response.statusText}): ${await this.errorMessage(response)}`, | |
| ) | |
| } | |
| const result = (await response.json()) as GrokVideoCreateResponse | |
| if (!result.request_id) { | |
| throw new Error( | |
| 'grok: video generation response contained no request_id', | |
| ) | |
| } | |
| return { jobId: result.request_id, model } | |
| } catch (error: unknown) { | |
| logger.errors(`${this.name}.createVideoJob fatal`, { | |
| error: toRunErrorPayload(error, `${this.name}.createVideoJob failed`), | |
| source: `${this.name}.createVideoJob`, | |
| }) | |
| throw error | |
| } | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/ai-grok/src/adapters/video.ts` around lines 151 - 208, The
validation calls (validateVideoSize, validateVideoDuration) in createVideoJob
can throw before the existing try/catch, so wrap those validations and the rest
of the function body inside the same try block (or add an outer try that
encloses validations) so any thrown validation errors are caught; in the catch
call logger.errors with toRunErrorPayload and source
`${this.name}.createVideoJob` (same shape as current catch) and rethrow the
error to preserve behavior.
Adds 'xAI Direct' text-to-video and image-to-video entries to the video generator that use the native grokVideo adapter against xAI's Imagine API (XAI_API_KEY), alongside the existing fal-hosted grok-imagine entries. Polling is now keyed by the UI model id (the direct entries share one adapter model), and the completed card shows the exact USD cost when the adapter reports it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…imagine-video-1.5-preview Groups the video model dropdown by provider (fal.ai / xAI direct) the same way the image generator does, adds the model version to the direct-xAI entry names, and switches them to the grok-imagine-video-1.5-preview model. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 Changes
Adds a
grokVideoadapter to@tanstack/ai-grokfor xAI's Imagine video models, closing #705:grok-imagine-video($0.05/s) andgrok-imagine-video-1.5-preview($0.08/s), added tomodel-meta.tswithGROK_VIDEO_MODELS/GrokVideoModelexports.GrokVideoAdapter(+grokVideo/createGrokVideofactories) extendsBaseVideoAdapterwith the jobs/polling pattern —createVideoJobposts to/v1/videos/generations,getVideoStatus/getVideoUrlread/v1/videos/{request_id}. The Imagine video endpoints are plain JSON and not part of the OpenAI SDK surface, so the adapter issues direct requests with an injectablefetchseam (no global stubs needed in tests).sizetemplate consistent with the grok-imagine image models —'16:9_720p'→aspect_ratio/resolution. Video accepts1:1 | 16:9 | 9:16 | 4:3 | 3:4 | 3:2 | 2:3and480p | 720p | 1080p(a narrower set than the image models — verified against the live API, which rejects phone ratios andauto).modelOptions.image: { url }passes a starting frame (public URL or base64 data URI).usage.unitsBilled(billed seconds) andusage.cost(exact USD from the API'scost_in_usd_ticks; 10^10 ticks per dollar).examples/ts-react-mediagains "xAI Direct" text-to-video and image-to-video entries using the nativegrokVideoadapter (alongside the existing fal-hosted grok-imagine entries), with exact USD cost shown on completion.docs/adapters/grok.md,docs/media/video-generation.md(+config.jsonupdatedAt), and themedia-generationagent skill.The issue assumed the Imagine plumbing from #624 (issue #618 ) had landed; that PR is still open, so this is built directly against
mainand is independently mergeable. The coreimageInputs/metadata.roleflow from #618/#624 isn't available onmainyet — image-to-video is exposed through typedmodelOptions.imagefor now. If #624 merges before this, this PR can be rebased and theimageInputs/ media-prompt flow wired into the adapter here before merging. Otherwise this can be merged as is, and 624 could be updated to rewire this adapterTesting
packages/ai-grok/tests/video-adapter.test.ts, 27 tests) cover request shape, size-template mapping, validation, status mapping, usage, and error paths via the injected fetch seam. aimock doesn't mock the Imagine video endpoints, so coverage is unit-test based (same approach the issue calls out forimage-to-image/image-to-video).generateVideo()/getVideoJobStatus(): text-to-video (16:9_480p, 1s,unitsBilled: 1,cost: 0.05) and image-to-video viamodelOptions.image(cost: 0.052— the API bills slightly more with an image input, which is why cost is read from the API rather than computed).✅ Checklist
pnpm run test:pr.🚀 Release Impact
🤖 Generated with Claude Code
Summary by CodeRabbit