Add Phase 1 schema-inference CLI (prompt → DatasetSchema)#16
Conversation
📝 WalkthroughWalkthroughThis PR adds a schema-inference feature: Zod schemas and TypeScript types for dataset contracts, a system prompt, an OpenRouter-backed Claude implementation that generates and validates JSON schemas, a Node.js CLI (npm script) that accepts --prompt, writes results to timestamped output directories, and repository changes (deps, env example, .gitignore) to support the feature. Sequence Diagram(s)sequenceDiagram
participant CLI
participant inferSchema
participant callOnce
participant generateText
participant Claude
CLI->>inferSchema: inferSchema(prompt)
inferSchema->>callOnce: callOnce(model, prompt)
callOnce->>generateText: generateText with datasetSchemaSchema
generateText->>Claude: request + system prompt
Claude->>generateText: JSON schema output
generateText->>callOnce: validated DatasetSchema object
callOnce->>inferSchema: return schema
inferSchema->>CLI: DatasetSchema
CLI->>Filesystem: create output/<runId>/ and write schema.json
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Adds zod-validated contracts for Phase 1↔2 seams per project_spec.md. Cross-field invariants: exactly one column with is_primary_key=true, its name matches DatasetSchema.primary_key, it is non-nullable and enumerable. No implementation yet — contracts only. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements input → schema → primary-key identification per project_spec.md: - `inferSchema(prompt)` calls Claude (claude-sonnet-4-6) with tool-forced JSON output, validates the result against `datasetSchemaSchema`, and retries once with the formatted zod error appended on failure. - CLI entry `npm run infer-schema -- --prompt "..."` writes `backend/output/<run_id>/schema.json`. - System prompt at `backend/prompts/schema-inference.txt`, loaded via `import.meta.url` (ESM-safe, works in dev and built). - `ANTHROPIC_API_KEY` is optional in `env.ts` (so the Fastify server still boots without it) and asserted at module load in `schema-inference.ts` so the CLI fails fast when missing. Placeholder added to `backend/.env.example`. - Generated `backend/output/` is gitignored. Phase 2 (row enumeration, TinyFish tools, Mastra agent) is deferred to a follow-up PR. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
DESU-CLUB
force-pushed
the
feat/phase1-schema-pk
branch
from
3d3636f to
e7dfe58
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@project_spec.md`:
- Around line 124-128: The project spec references incorrect source and asset
paths (e.g., src/..., prompts/..., output/..., src/index.ts) while the
implementation lives under backend/ (backend/src/..., backend/prompts/...,
backend/output/...) and the CLI is backend/src/cli.ts; update all occurrences in
the spec (including the inferSchema entry for export async function
inferSchema(prompt: string): Promise<DatasetSchema> and the ranges noted at
202-228 and 252-258) to use the backend/... paths and correct CLI entrypoint so
the spec matches the repository layout and exports.
- Around line 12-14: Multiple fenced code blocks in project_spec.md lack
language labels causing markdownlint MD040; update each affected triple-backtick
block (e.g., the blocks containing "Prompt → Schema inference → Primary key
enumeration → Skeleton dataset", the lines starting with
"schema.retrieval_strategy === ..." , the repository tree block beginning with
"/", and the "Run ID: abc123 ..." block) to include an appropriate language tag
(suggest: text for plain prose/tree, bash for shell examples, json for JSON
snippets) so each ``` becomes something like ```text, ```bash or ```json; apply
the same change to the other reported ranges (around lines 152-156, 204-228,
263-268) to resolve MD040 across the file.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: b6f98900-1584-476a-89a2-f670e21a0684
⛔ Files ignored due to path filters (1)
backend/package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (9)
.gitignorebackend/.env.examplebackend/package.jsonbackend/prompts/schema-inference.txtbackend/src/cli.tsbackend/src/env.tsbackend/src/pipeline/schema-inference.tsbackend/src/pipeline/types.tsproject_spec.md
simantak-dabhade
requested review from
MMeteorL,
giaphutran12 and
simantak-dabhade
| @@ -0,0 +1,132 @@ | |||
| import { readFileSync } from "node:fs"; | |||
| import Anthropic from "@anthropic-ai/sdk"; | |||
There was a problem hiding this comment.
we're using vercel ai sdk and openrouter key actually, we're not locking in anthropic models at this point
There was a problem hiding this comment.
Good catch — swapped it out. Now using Vercel AI SDK + OpenRouter with @openrouter/ai-sdk-provider. The model string is just openrouter("anthropic/claude-sonnet-4-6") so we can swap to any provider/model by changing one string. See commit f544da7.
| 1. Identify the universe of entities the user wants to collect. Each entity becomes one row in the dataset. | ||
| 2. Pick a clear primary key — the column whose values uniquely identify each row. This is usually a name, ID, or canonical URL. Exactly one column must have `is_primary_key: true`, and its `name` must equal `primary_key`. The primary key column must have `nullable: false` and `is_enumerable: true`. | ||
| 3. Choose useful columns. Each column captures one fact about the entity. Use snake_case names. Mark `is_enumerable: true` only on columns whose values can be used to list all rows (typically just the primary key, and occasionally one or two others when a source page lists them alongside the primary key). | ||
| 4. Set `retrieval_strategy`: |
There was a problem hiding this comment.
This may be too early in the agent workflow to suggest these without more context. In the current agent system, the agent would first use Tinyfish search to search for candidate urls and fetch those results for analysis of what is the best way to retrieve.
For the same reason, instead of source_hint, it may be helpful to generate search_query_hint.
Retrieval_hint is a great idea, however, because of the varied sources that Tinyfish search would propose, this may not be the best fit. This should be a decision left for downstream agent with more information.
My recommendation is to produce 4: 'search_queries": generate [X number] of search queries based on the user prompt and data spec. These search queries should be differentiated from each other.
There was a problem hiding this comment.
Agreed — added your comments verbatim to the prompt file so we don't lose this context (commit f54e368). We'll revisit the schema fields (replacing retrieval_strategy/source_hint/retrieval_hint with search_queries) once the downstream agent work is further along and we have a clearer picture of how TinyFish Search fits into the flow.
Replaces @anthropic-ai/sdk with ai (Vercel AI SDK 6) and @openrouter/ai-sdk-provider so the schema-inference CLI is not locked to a single model provider. Uses generateText + Output.object() which passes the Zod schema directly, eliminating the duplicate JSON Schema definition and manual tool_use extraction. API key check moved from module-level to getModel() so importing this module won't crash the backend server when OPENROUTER_API_KEY is unset. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/src/cli.ts`:
- Around line 28-40: The CLI currently prints the raw prompt and raw error
objects; update the main() flow so you no longer log the full prompt variable or
entire error objects: in the area around inferSchema(prompt) and the console.log
statements referencing prompt and runId, replace printing the raw prompt with a
redacted or truncated preview (e.g., show only length or first N characters with
"[REDACTED]" suffix) and keep runId only, and in the main().catch handler
replace console.error(err) with a sanitized error log that emits only a short
message, an error code/type, and optionally a truncated stack or hashed
fingerprint rather than the full error object. Ensure changes reference the
inferSchema call, runId, prompt variable, and main() catch block so the reviewer
can find and update the exact logging lines.
In `@backend/src/pipeline/types.ts`:
- Line 28: The retrieval_hint field in the Zod schema currently allows empty
strings; update the schema entry for retrieval_hint in
backend/src/pipeline/types.ts to enforce non-empty values by adding .min(1) like
the existing display_name and source_hint fields (i.e., adjust the z.string()
call for retrieval_hint to z.string().min(1)) so the field matches the intent of
the schema-inference prompt.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 574ecc93-9d96-44f5-84cd-13eb822ab349
⛔ Files ignored due to path filters (1)
backend/package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (8)
.gitignorebackend/.env.examplebackend/package.jsonbackend/prompts/schema-inference.txtbackend/src/cli.tsbackend/src/env.tsbackend/src/pipeline/schema-inference.tsbackend/src/pipeline/types.ts
✅ Files skipped from review due to trivial changes (3)
- .gitignore
- backend/.env.example
- backend/src/env.ts
| console.log(`Inferring schema for: "${prompt}"`); | ||
| const schema = await inferSchema(prompt); | ||
|
|
||
| mkdirSync(outDir, { recursive: true }); | ||
| const schemaPath = join(outDir, "schema.json"); | ||
| writeFileSync(schemaPath, JSON.stringify(schema, null, 2) + "\n"); | ||
|
|
||
| console.log(`Run ID: ${runId}`); | ||
| console.log(`Schema: backend/${schemaPath}`); | ||
| } | ||
|
|
||
| main().catch((err) => { | ||
| console.error(err); |
There was a problem hiding this comment.
Avoid logging raw prompts and raw error objects.
This CLI can handle sensitive prompt content; printing full prompt and unsanitized errors increases leakage risk in shared logs.
Suggested hardening
- console.log(`Inferring schema for: "${prompt}"`);
+ console.log(`Inferring schema (prompt length: ${prompt.length})`);
main().catch((err) => {
- console.error(err);
+ const message = err instanceof Error ? err.message : String(err);
+ console.error(`infer-schema failed: ${message}`);
process.exit(1);
});🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/src/cli.ts` around lines 28 - 40, The CLI currently prints the raw
prompt and raw error objects; update the main() flow so you no longer log the
full prompt variable or entire error objects: in the area around
inferSchema(prompt) and the console.log statements referencing prompt and runId,
replace printing the raw prompt with a redacted or truncated preview (e.g., show
only length or first N characters with "[REDACTED]" suffix) and keep runId only,
and in the main().catch handler replace console.error(err) with a sanitized
error log that emits only a short message, an error code/type, and optionally a
truncated stack or hashed fingerprint rather than the full error object. Ensure
changes reference the inferSchema call, runId, prompt variable, and main() catch
block so the reviewer can find and update the exact logging lines.
| type: columnTypeSchema, | ||
| is_primary_key: z.boolean(), | ||
| is_enumerable: z.boolean(), | ||
| retrieval_hint: z.string(), |
There was a problem hiding this comment.
Add .min(1) constraint to retrieval_hint for consistency.
The retrieval_hint field accepts empty strings, while similar fields like display_name (line 24) and source_hint (line 40) enforce .min(1). The schema-inference prompt (line 13) instructs "Write a retrieval_hint for each column describing where/how the value can be found later," which implies non-empty content is expected.
🔧 Proposed fix
- retrieval_hint: z.string(),
+ retrieval_hint: z.string().min(1),📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| retrieval_hint: z.string(), | |
| retrieval_hint: z.string().min(1), |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/src/pipeline/types.ts` at line 28, The retrieval_hint field in the
Zod schema currently allows empty strings; update the schema entry for
retrieval_hint in backend/src/pipeline/types.ts to enforce non-empty values by
adding .min(1) like the existing display_name and source_hint fields (i.e.,
adjust the z.string() call for retrieval_hint to z.string().min(1)) so the field
matches the intent of the schema-inference prompt.
Preserves suggestions about replacing retrieval_strategy/source_hint with search_queries for future implementation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@backend/prompts/schema-inference.txt`:
- Around line 20-36: The production prompt contains inline review-discussion
comments (the block beginning with "# `@MMeteorL`'s comments/suggestions:"
referencing Tinyfish, source_hint, retrieval_hint, and recommending
'search_queries') that must be removed from
backend/prompts/schema-inference.txt; delete that review text so the file only
contains the formal schema-inference prompt content (move any useful reviewer
suggestions into the PR or separate design notes), and if you want to preserve
the actionable suggestion, replace the removed discussion with a concise, formal
prompt field such as "search_queries" guidance (not the reviewer conversation)
so functions like schema inference, search_queries, source_hint, and
retrieval_hint are defined explicitly and consistently.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: b438a317-791b-467a-9aa0-9b3da82d1eeb
📒 Files selected for processing (1)
backend/prompts/schema-inference.txt
|
|
||
| # @MMeteorL's comments/suggestions: | ||
| # This may be too early in the agent workflow to suggest these without more | ||
| # context. In the current agent system, the agent would first use Tinyfish | ||
| # search to search for candidate urls and fetch those results for analysis | ||
| # of what is the best way to retrieve. | ||
| # | ||
| # For the same reason, instead of source_hint, it may be helpful to generate | ||
| # search_query_hint. | ||
| # | ||
| # Retrieval_hint is a great idea, however, because of the varied sources | ||
| # that Tinyfish search would propose, this may not be the best fit. This | ||
| # should be a decision left for downstream agent with more information. | ||
| # | ||
| # My recommendation is to produce: 'search_queries': generate [X number] of | ||
| # search queries based on the user prompt and data spec. These search queries | ||
| # should be differentiated from each other. |
There was a problem hiding this comment.
Remove inline review-discussion text from the production prompt.
These lines are not ignored by the LLM; they become active prompt content and can conflict with the required schema contract, causing inconsistent outputs.
Suggested fix
-
-# `@MMeteorL`'s comments/suggestions:
-# This may be too early in the agent workflow to suggest these without more
-# context. In the current agent system, the agent would first use Tinyfish
-# search to search for candidate urls and fetch those results for analysis
-# of what is the best way to retrieve.
-#
-# For the same reason, instead of source_hint, it may be helpful to generate
-# search_query_hint.
-#
-# Retrieval_hint is a great idea, however, because of the varied sources
-# that Tinyfish search would propose, this may not be the best fit. This
-# should be a decision left for downstream agent with more information.
-#
-# My recommendation is to produce: 'search_queries': generate [X number] of
-# search queries based on the user prompt and data spec. These search queries
-# should be differentiated from each other.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # @MMeteorL's comments/suggestions: | |
| # This may be too early in the agent workflow to suggest these without more | |
| # context. In the current agent system, the agent would first use Tinyfish | |
| # search to search for candidate urls and fetch those results for analysis | |
| # of what is the best way to retrieve. | |
| # | |
| # For the same reason, instead of source_hint, it may be helpful to generate | |
| # search_query_hint. | |
| # | |
| # Retrieval_hint is a great idea, however, because of the varied sources | |
| # that Tinyfish search would propose, this may not be the best fit. This | |
| # should be a decision left for downstream agent with more information. | |
| # | |
| # My recommendation is to produce: 'search_queries': generate [X number] of | |
| # search queries based on the user prompt and data spec. These search queries | |
| # should be differentiated from each other. |
🧰 Tools
🪛 LanguageTool
[style] ~32-~32: Consider using a more formal alternative.
Context: ...decision left for downstream agent with more information. # # My recommendation is to produce: '...
(MORE_INFO)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@backend/prompts/schema-inference.txt` around lines 20 - 36, The production
prompt contains inline review-discussion comments (the block beginning with "#
`@MMeteorL`'s comments/suggestions:" referencing Tinyfish, source_hint,
retrieval_hint, and recommending 'search_queries') that must be removed from
backend/prompts/schema-inference.txt; delete that review text so the file only
contains the formal schema-inference prompt content (move any useful reviewer
suggestions into the PR or separate design notes), and if you want to preserve
the actionable suggestion, replace the removed discussion with a concise, formal
prompt field such as "search_queries" guidance (not the reviewer conversation)
so functions like schema inference, search_queries, source_hint, and
retrieval_hint are defined explicitly and consistently.
4 participants
Summary
backend/src/pipeline/types.ts) — Zod schemas + inferred TS types forDatasetSchema,DatasetRow, andRunManifest. Locked early so retrieval/normalization phases can be built against a stable interfacenpm run infer-schema) — Wraps@anthropic-ai/sdk+ the prompt template atbackend/prompts/schema-inference.txtto turn a natural-language prompt into a validatedDatasetSchema. Writesoutput/<timestamp>-<hash>/schema.jsonANTHROPIC_API_KEY(only the CLI reads it; backend/server boot is unaffected)Verification
Command run:
Output (
backend/output/20260519235614-ffdd70/schema.json):{ "dataset_name": "yc_fintech_companies", "description": "Y Combinator companies in the Fintech industry, including company name, batch, and website URL.", "columns": [ { "name": "company_name", "display_name": "Company Name", "type": "string", "is_primary_key": true, "is_enumerable": true, "retrieval_hint": "The name of the YC-backed fintech company, listed on the YC company directory page.", "nullable": false }, { "name": "batch", "display_name": "Batch", "type": "string", "is_primary_key": false, "is_enumerable": true, "retrieval_hint": "The YC batch the company participated in (e.g. S21, W22), found on the company's YC profile page or the directory listing.", "nullable": true }, { "name": "company_url", "display_name": "Company URL", "type": "url", "is_primary_key": false, "is_enumerable": false, "retrieval_hint": "The company's official website URL, listed on the YC company directory or the company's individual YC profile page.", "nullable": true } ], "primary_key": "company_name", "retrieval_strategy": "browser", "source_hint": "https://www.ycombinator.com/companies?industry=Fintech" }Test plan
ANTHROPIC_API_KEYinbackend/.envnpm run infer-schema -- --prompt "<your prompt>"frombackend/— confirms a freshbackend/output/<timestamp>-<hash>/schema.jsonis writtenschema.json— confirms it parses as a validDatasetSchema(columns array, oneis_primary_key: true,retrieval_strategyset)make devfrom repo root still boots frontend (:3500), backend (:3501), Convex (:3210) — Phase 1 changes are CLI-only and shouldn't touch the running servicesNotes
tscerror onmain(convex.setAdminAuthnot onConvexHttpClient) is unrelated to this PRbackend/output/is gitignored — only the path convention is exercised here🤖 Generated with Claude Code