Skip to content

Add collect tool type#1

Merged
mme merged 7 commits into
mainfrom
mme/collect
Apr 2, 2026
Merged

Add collect tool type#1
mme merged 7 commits into
mainfrom
mme/collect

Conversation

@mme
Copy link
Copy Markdown
Contributor

@mme mme commented Apr 1, 2026

Summary

Adds a collect tool type that lets agents write structured data back to the database, configured entirely in YAML — no code changes needed to define new collect tools.

  • Schema validation: Field types (string, number, enum) validated at config load via Zod, with runtime input validation on every tool call
  • Generic error handling: Both search and collect tools return generic error messages to agents, preventing internal detail leaks (PG hosts, API keys)
  • Config hardening: Tool name uniqueness validation, backwards-compatible type defaulting with deprecation warning, exhaustive tool type dispatch
  • First use case: submit-feedback tool for agents to report whether search results were helpful

What changed

Area Files What
Types src/types.ts CollectToolConfigSchema, discriminated union, field refinements (enum values, non-enum values rejection, min 1 field)
DB src/db/schema.ts, src/db/queries.ts collected_data table (JSONB), parameterized insertCollectedData
Implementation src/mcp/tools/collect.ts, src/mcp/server.ts yamlSchemaToZod, registerCollectTool, exhaustive switch dispatch
Config src/config.ts Tool name uniqueness check, backwards-compat type defaulting, scoped source validation
Error handling src/mcp/tools/search.ts Generic error response (was leaking raw error.message)
Logging src/index.ts Collect tool call logging with truncated data preview
Config files mcp-docs.yaml, mcp-docs.example.yaml submit-feedback tool definition
Tests src/__tests__/collect*.ts, src/__tests__/tool-config.test.ts 29 tests: schema conversion, config validation, MCP protocol round-trips, DB failure handling
Docs README.md Collect tool documentation and examples

Test plan

  • npx tsc --noEmit passes
  • 29 unit tests via vitest (npm test):
    • yamlSchemaToZod conversion (string, number, enum, required/optional, descriptions, unsupported type rejection)
    • CollectToolConfigSchema and AnyToolConfigSchema validation (empty schema, enum without values, values on non-enum, unknown types)
    • ServerConfigSchema.superRefine for default_limit <= max_limit
    • Full MCP protocol round-trip (list, call, DB write, DB failure with generic error, invalid input rejection)

🤖 Generated with Claude Code

jpr5
jpr5 previously requested changes Apr 1, 2026
Copy link
Copy Markdown
Contributor

@jpr5 jpr5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review — Add Collect Tool Type

Clean, well-scoped PR. The collect tool concept is sound — YAML-defined schemas, Zod validation, parameterized DB writes, good test coverage for a first pass. Found no critical bugs, but several important issues worth fixing before merge.


🟡 Important — Should Fix

I1. Error messages from insertCollectedData are exposed verbatim to the MCP agent

src/mcp/tools/collect.ts: when the DB insert fails, the raw error.message is returned in the MCP tool response. PostgreSQL errors routinely include hostnames, ports, usernames, and partial query text. This leaks directly to the LLM agent (and potentially to end users).

Fix: return a generic message in the MCP response, keep the detail in console.error only:

catch (error) {
    const message = error instanceof Error ? error.message : String(error);
    console.error(`[${toolConfig.name}] Error inserting collected data: ${message}`);
    return {
        content: [{ type: "text" as const, text: "Error: Failed to store data. Please try again later." }],
        isError: true,
    };
}

I2. yamlSchemaToZod switch has no default branch

src/mcp/tools/collect.ts: the switch handles 'string', 'number', 'enum' but has no default. If someone adds a new type to the Zod enum in types.ts but forgets to update the switch, fieldSchema is used uninitialized — the server crashes at startup with an incomprehensible TypeError: Cannot read properties of undefined.

Fix: add a default that throws with the field name and type:

default:
    throw new Error(`Unsupported field type "${field.type}" for field "${fieldName}"`);

I3. Empty schema: {} silently accepts and stores arbitrary data

CollectToolConfigSchema allows schema: {}. When registered, this creates a tool with zero input fields that accepts any arguments and writes {} to the database. Almost certainly a misconfiguration.

Fix: add a minimum size on the schema record:

schema: z.record(...).refine(
    s => Object.keys(s).length > 0,
    { message: 'collect tool schema must define at least one field' }
)

The test at tool-config.test.ts that asserts schema: {} is valid should be updated to expect failure.

I4. .default('search') on SearchToolConfigObjectSchema is dead code inside discriminatedUnion

src/types.ts: z.discriminatedUnion inspects the discriminator value before applying defaults. When the type field is absent, Zod can't match any variant — the .default('search') never fires. The actual backward-compat handling is the imperative tool.type = 'search' in config.ts.

Fix: change .default('search') to just z.literal('search') to make the schema honest. The pre-processing in config.ts handles backward compat, and that's fine.

I5. default_limit <= max_limit not enforced by AnyToolConfigSchema

The SearchToolConfigSchema.refine() is only used for type inference, not in the actual parse path. The real enforcement is in ServerConfigSchema.superRefine(). But AnyToolConfigSchema — which looks like the canonical "parse any tool config" entry point — silently allows default_limit > max_limit.

Fix: add a comment on AnyToolConfigSchema noting the cross-field constraint lives in ServerConfigSchema.superRefine, not here. And delete the dead .refine() on SearchToolConfigSchema since it never fires in the parsing pipeline.

I6. MCP test "writes data to the database" is order-dependent

collect-mcp.test.ts: this test asserts insertCollectedData was called, but doesn't invoke the tool itself — it relies on the previous test having run. No vi.clearAllMocks() between tests.

Fix: add beforeEach(() => vi.clearAllMocks()), and either merge the assertion into the "calls the tool" test or make it self-contained.


🔵 Suggestions

S1. values accepted on non-enum fields. { type: 'string', values: ['a', 'b'] } parses successfully — values is silently ignored. Consider adding a refine: f => f.type === 'enum' || !f.values.

S2. Config backwards-compat should log a deprecation warning when defaulting type: 'search' so users know to add type: search explicitly.

S3. README says "store it as JSON" but the column is JSONB. Meaningful distinction for PG users.

S4. README description of "Data is written to the collected_data table with the tool name and a timestamp" omits the most important thing stored — the data payload itself.

S5. yamlSchemaToZod JSDoc says "into a Zod object schema" but it returns a shape record, not a z.ZodObject.

S6. DB failure path in registerCollectTool untested. Mock insertCollectedData to reject and verify isError: true.

S7. superRefine for default_limit <= max_limit untested at the ServerConfigSchema level.


✅ What's Working Well

  • SQL injection safe: insertCollectedData uses parameterized queries ($1, $2)
  • Clean discriminated union: AnyToolConfigSchema correctly separates search and collect concerns
  • Backwards-compat: config.ts handles existing YAML files without type: field
  • Thorough schema validation tests: collect.test.ts covers all field types, required/optional, descriptions, and invalid input rejection
  • MCP protocol round-trip test: uses InMemoryTransport to test the full client-server path
  • Good first use case: search feedback is genuinely useful and demonstrates the pattern
  • Cross-validation scoped correctly: source reference check now only applies to search tools

Summary: 6 important issues, 7 suggestions. No critical bugs. The most actionable are I1 (PG error leak to agents) and I2 (switch without default). The schema validation issues (I3-I5) are more about correctness of the validation boundaries.

@mme mme force-pushed the mme/collect branch 3 times, most recently from 38929cf to 82c48a0 Compare April 1, 2026 20:38
@mme mme dismissed jpr5’s stale review April 1, 2026 20:41

All 6 requested changes and 7 suggestions addressed

Copy link
Copy Markdown
Contributor

@jpr5 jpr5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 2 Review — Collect Tool Type

Nice work addressing the Round 1 feedback — 5 of 6 findings fully fixed, 1 documented as intentional.

Fixed from Round 1

Finding Status
PG error leak to agents FIXED — generic message returned, detail logged server-side, test verifies no IP leak
switch without default in yamlSchemaToZod FIXED — throws with field name and type
Empty schema: {} accepted FIXED — refine requires at least one field
.default('search') dead code in discriminatedUnion FIXED — z.literal('search'), compat in config.ts with warning
values on non-enum fields FIXED — refine rejects
default_limit <= max_limit location Documented — comment explains it lives in ServerConfigSchema.superRefine

Test Gaps Fixed

Gap Status
DB failure path FIXED — mocks rejection, verifies sanitized error
superRefine limit validation FIXED — two tests at ServerConfigSchema level
Order-dependent MCP test FIXED — vi.clearAllMocks + merged assertions
Empty schema FIXED — asserts rejection
Backwards-compat config defaulting Not tested — loadServerConfig() type-injection has no coverage

🟡 New Finding — Should Fix

createMcpServer switch has no default case — future tool types silently skipped

src/mcp/server.ts: the switch (tool.type) handles 'collect' and 'search' but has no default. If a new tool type passes Zod validation but isn't wired here, the tool silently doesn't register. Server starts fine, tool never appears, no error.

default:
    throw new Error(`Unknown tool type "${tool.type}" for tool "${tool.name}"`);

🔵 Suggestions

  • README inconsistency: opening paragraph says "store it as JSON", closing paragraph correctly says "JSONB". Column is JSONB — make consistent.
  • registerCollectTool JSDoc: doesn't mention the sanitized error pattern (log detail, return generic message). Worth a one-liner.
  • Backwards-compat defaulting untested: the loadServerConfig() type-injection logic has no test. If broken, users without type: in YAML get cryptic Zod errors on startup.

✅ Overall

Significant improvement from Round 1. All original important issues addressed. The server.ts default case is the only remaining actionable item — everything else is minor.

@jpr5 jpr5 requested a review from AlemTuzlak April 2, 2026 17:10
@jpr5
Copy link
Copy Markdown
Contributor

jpr5 commented Apr 2, 2026

Fixing.

Copy link
Copy Markdown
Contributor

@jpr5 jpr5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All Round 2 findings addressed:

  • createMcpServer switch now has exhaustive default case (d4d11d2)
  • README JSON → JSONB consistency (fd38593)
  • registerCollectTool JSDoc documents error handling pattern (b999f1e)
  • Backwards-compat config defaulting now tested (6a8bd52)

Build passes, all 31 tests pass. Ready for @AlemTuzlak to review.

@jpr5 jpr5 requested review from jpr5 and removed request for AlemTuzlak April 2, 2026 17:34
mme added 7 commits April 2, 2026 19:48
Introduces CollectToolConfig Zod schema with YAML-defined field
validation, a discriminated union (AnyToolConfigSchema) for tool
type dispatch, the collected_data DB table, and insertCollectedData
query. Renames ToolConfig -> SearchToolConfig and moves cross-field
validation to ServerConfigSchema.superRefine.
Adds registerCollectTool which converts YAML field definitions to Zod
shapes at registration time and writes validated input to the DB.
Updates createMcpServer with an exhaustive switch on tool type.
Renames ToolConfig -> SearchToolConfig in search tool and hardens
its error response to avoid leaking internal details.
Defaults tool type to 'search' for backwards compatibility, validates
tool name uniqueness, scopes source cross-validation to search tools
only. Adds submit-feedback collect tool to mcp-docs.yaml and example.
Branches MCP request logging to show a JSON data preview for collect
tools instead of the query/limit format used for search tools.
Covers yamlSchemaToZod conversion, MCP protocol round-trip (tool
listing, successful call, DB failure, invalid input), CollectToolConfig
and AnyToolConfigSchema validation, backwards-compat type defaulting,
and ServerConfigSchema cross-field checks.
Adds Collect Tools section with schema field reference and example
config. Renames 'Tools' header to 'Search Tools' for clarity. Adds
npm test to the development commands.
@mme mme merged commit 3940509 into main Apr 2, 2026
@jpr5 jpr5 mentioned this pull request Apr 2, 2026
jpr5 added a commit that referenced this pull request Jun 8, 2026
…ui sources + chunker/durability (#93)

## What this PR actually is
This is the **Pathfinder gap-report remediation** (the 30-day docs-MCP
gap analysis). Merging it **deploys to the live docs-MCP**
(`mcp.copilotkit.ai`) via push-to-main → Docker → Railway, and runs a
small **additive** DB migration. It bundles the engine-code fixes
**and** the `deploy/copilotkit-docs.yaml` config quick-wins — so merging
flips production behavior, not just code.

## Config quick-wins (`deploy/copilotkit-docs.yaml`)
- **Hybrid search ON** for all 4 search tools (`search_mode: hybrid`) +
`min_score: 0.3` floor — the report's #1 lever (prod was pure-vector).
- **Docs source repointed** from the retired `docs/content/docs/` tree
to the live `showcase/shell-docs/src/content/docs/` (+ `strip_prefix` /
`webhook.path_triggers`).
- **Code source de-polluted**: exclude `examples/**`, `showcase/**`,
`**/.next/**`, `**/*.d.ts` (was ~79% boilerplate).
- **ag-ui code sources added**: `integrations/**` (Python adapters),
`sdks/python/`, `sdks/community/` (JVM + ports); exclude `generated/` +
`*.pb.*`.
- **Tool descriptions sharpened** with scope/exclusions (fixes
cross-tool mis-routing).
- Drop the two non-`.mdx` docs files that derived 404 URLs.

## Engine changes (`src/`)
- **Markdown chunker hardening** + large invariant test corpus (fence
integrity, verbatim-substring fidelity, heading-path soundness, robust
LLM-array recovery).
- **MDX `<Snippet/>` inlining** (`snippets.ts`) — recovers
snippet-composed pages (v2 Migration Guide, etc.).
- **Embed `title` + `headingPath`** alongside content (`pipeline.ts`),
not content-only.
- **Indexing durability**: per-item partial-failure handling, composed
with Atlas cache invalidation.
- **Request-source tagging** (`X-Pathfinder-Source` →
`query_log.request_source` + `session_id`) — anti-self-inflation
groundwork. Migration is `ADD COLUMN IF NOT EXISTS` + `CREATE INDEX IF
NOT EXISTS` on the append-only `query_log` (additive, idempotent,
online-safe).

## After merge
- Watch `deploy-health-check` (auto on push-to-main) +
`index-health-monitor` (4h cron).
- Retrieval gains (hybrid, repoint, new sources, embed-text) take full
effect on the **next nightly reindex (03:00 UTC)** which re-chunks +
re-embeds.
- 4234 tests green; CI green; rebased onto current main (Atlas
integrated).

## Not included (follow-up)
- Query-time alias/acronym expansion (HITL↔human-in-the-loop, etc.) —
gap-report QW7.
- Structural: split examples into their own source, content-hash dedup,
AST chunking, cross-encoder reranker.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants