Skip to content

Add Supabase vector storage with pgvector support#578

Merged
sroussey merged 13 commits into
mainfrom
claude/hopeful-babbage-vd5ory
Jun 12, 2026
Merged

Add Supabase vector storage with pgvector support#578
sroussey merged 13 commits into
mainfrom
claude/hopeful-babbage-vd5ory

Conversation

@sroussey

Copy link
Copy Markdown
Collaborator

Implements SupabaseVectorStorage, a scoped vector storage backend for Supabase using pgvector for similarity search. Extends the existing SupabaseTabularStorage to add vector-specific operations while enforcing tenant/project/knowledge-base isolation at the application layer.

Key Changes

  • SupabaseVectorStorage class (providers/supabase/src/storage/SupabaseVectorStorage.ts): New vector storage implementation that:

    • Extends SupabaseTabularStorage for row persistence
    • Delegates similarity search to server-side match_<table> RPC functions (pgvector)
    • Enforces scope binding via withScope() — all reads/writes are constrained to a tenant/project/knowledge-base scope
    • Converts TypedArray vectors to pgvector text format ([a,b,c]) on write and restores them on read
    • Validates metadata filter keys to prevent injection attacks
    • Overrides all inherited tabular methods to inject scope criteria, preventing cross-scope data leakage
  • Database schema and RPCs (providers/supabase/migrations/0001_kb_chunks_pgvector.sql):

    • Creates kb_chunks_* tables for 5 vector dimensions (384, 768, 1024, 1536, 3072)
    • Each table has scope columns (tenant_id, project_id, kb_id) and a pgvector column
    • HNSW indexes for fast similarity search; 3072-dim table uses halfvec cast to work around pgvector's 2000-dim HNSW limit
    • match_kb_chunks_* RPC functions that perform cosine similarity search with scope filtering and optional metadata filtering
    • Row-level security policies for tenant isolation (defense-in-depth; service role bypasses RLS)
  • Shared documents table (providers/supabase/migrations/0002_shared_documents.sql): Simple table for document metadata with RLS tenant isolation

  • Integration tests:

    • SupabaseVectorStorage.integration.test.ts: Tests vector persistence, similarity ranking, scope isolation, and metadata filter validation against a PGlite-backed mock
    • kb_chunks_rls.test.ts: Verifies RLS policies enforce tenant isolation at the database level
    • shared_documents_rls.test.ts: Verifies RLS on the shared documents table
    • SupabaseMockClient.pgvector.test.ts: Tests the mock client's pgvector extension and RPC routing
  • Mock client enhancements (packages/test/src/test/helpers/SupabaseMockClient.ts):

    • Added pgvector extension support to PGlite
    • Enhanced RPC routing to handle multi-statement SQL (migrations) via exec_sql RPC
    • Positional parameter routing for RPC functions (required for PGlite test mocks)
  • Dependencies: Added @electric-sql/pglite-pgvector for pgvector support in tests

Notable Implementation Details

  • Scope enforcement: Scope columns are always stamped from the resolved scope, never from client input. All inherited read/delete methods are overridden to inject scope criteria, preventing a single knowledge base from accessing data across scopes.
  • Vector serialization: Vectors are stored as pgvector text in the database but returned to callers as their original TypedArray type, preserving the schema contract.
  • Metadata filtering: Uses JSONB containment (@>) for flexible filtering; filter keys are validated against /^[a-zA-Z_][a-zA-Z0-9_]*$/ to prevent injection.
  • Score threshold handling: Accepts NULL for no threshold (returns all results), matching pgvector conventions.
  • Independent scoped instances: withScope() constructs a fresh instance rather than cloning the prototype, avoiding aliasing of mutable state (realtime channels, event listeners).

https://claude.ai/code/session_01VHcJsxLqN1yMVghTLXfP1k

@github-actions

github-actions Bot commented Jun 12, 2026

Copy link
Copy Markdown

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 62.17% 24969 / 40156
🔵 Statements 62.02% 25827 / 41643
🔵 Functions 63.11% 4721 / 7480
🔵 Branches 50.82% 12243 / 24088
File CoverageNo changed files found.
Generated in workflow #2560 for commit c4d8919 by the Vitest Coverage Report Action

claude added 13 commits June 12, 2026 20:33
Extends the pgvector migration to add tables, indexes, match functions,
RLS policies, and integration tests for all five supported embedding
dimensions (384, 768, 1024, 1536, 3072).

https://claude.ai/code/session_01L7bTPtkGxEdXxEToZe9Tu1
…ore reads & deletes

SupabaseVectorStorage only overrode put/putBulk/similaritySearch, so the
inherited tabular methods (get, getAll, deleteAll, query, deleteSearch, size,
getBulk, paging) ran table-wide. Database RLS constrains only the tenant, so a
single knowledge base's getAll/clearChunks could read or delete every kb the
tenant owns at that embedding dimension (clearChunks -> deleteAll was tenant-wide
data loss). Override the scope-sensitive accessors to narrow each one to the
bound (tenant, project, kb) scope; scope values win over caller criteria.
…pp KB schema

SupabaseVectorStorage is now a generic pgvector repository: the scope is a
generic Record<string,string> of equality columns (optional) instead of a fixed
tenant/project/kb shape, and similarity search delegates to a match_<table> RPC
with a generic p_scope jsonb param. Stamping, filtering, and the scope-narrowed
inherited reads/deletes all operate over the bound column map; with no scope
bound the store is plain table-wide.

The cloud knowledge-base schema (kb_chunks_<dim> + shared_documents migrations
and their tenant-RLS tests) is consumer-specific and no longer ships in the
generic provider; it moves to the consuming application. The integration test
now exercises the storage against a generic table + match RPC.
… scoping

Remove all scoping from the generic provider: no VectorScope type, no withScope,
no scope stamping, no p_scope RPC arg, and no scope-narrowing overrides. The base
is now a plain table-wide pgvector store whose match_<table> RPC takes only
(query_embedding, match_count, score_threshold, p_filter).

prepareForWrite and matchArgs are protected seams (and the vector/metadata
helpers are protected) so a consumer that needs scoping can subclass: decorate
writes and pass extra arguments to a wider match_<table>, without the base
knowing anything about scope. The integration test now covers the plain store
(ranking, hydration, metadata default + containment filter, filter-key guard).
@sroussey sroussey force-pushed the claude/hopeful-babbage-vd5ory branch from c4d8919 to 23dbfb5 Compare June 12, 2026 20:34
@sroussey sroussey merged commit fa16883 into main Jun 12, 2026
12 of 13 checks passed
@sroussey sroussey deleted the claude/hopeful-babbage-vd5ory branch June 12, 2026 20:34
sroussey added a commit that referenced this pull request Jun 12, 2026
## @workglow/browser-control

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/task-graph

### Features

- add bugs URL to package.json files across all packages and providers

### Bug Fixes

#### task-graph,storage

- cache restart-resume + SharedInMemory sync barrier (#552)

### Documentation

#### task-graph

- fix TaskOutputTabularRepository README examples for new constructor signature

## @workglow/javascript

### Features

- add bugs URL to package.json files across all packages and providers

### Bug Fixes

- tsgo issue

## @workglow/ai

### Features

- add typecheck budget guard to catch type-instantiation regressions (#555)
- add bugs URL to package.json files across all packages and providers

### Bug Fixes

#### ai

- export ChunkRetrievalInputSchema + nightly schema-vs-type drift guard (#565)

## @workglow/knowledge-base

### Features

- add bugs URL to package.json files across all packages and providers

## workglow

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/storage

### Features

- add bugs URL to package.json files across all packages and providers

### Bug Fixes

#### task-graph,storage

- cache restart-resume + SharedInMemory sync barrier (#552)

## @workglow/mcp

### Features

- add bugs URL to package.json files across all packages and providers

### Bug Fixes

#### mcp

- thread run-scoped registry through discoverSchemas (#577)
- resolve auth credentials through the run-scoped registry

## @workglow/util

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/test

### Features

- add bugs URL to package.json files across all packages and providers

#### supabase

- add Supabase vector storage with pgvector support (#578)

### Bug Fixes

#### mcp

- thread run-scoped registry through discoverSchemas (#577)
- resolve auth credentials through the run-scoped registry

#### task-graph,storage

- cache restart-resume + SharedInMemory sync barrier (#552)

### Tests

- pin HF router provider for tool-calling conformance tests (#564)

### Chores

- update deps

### Updated Dependencies

- `@aws-sdk/client-sqs`: ^3.1068.0
- `@cloudflare/workers-types`: ^4.20260612.1
- `@types/dom-chromium-ai`: ^0.0.17
- `miniflare`: ^4.20260611.0

## @workglow/tasks

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/job-queue

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/indexeddb

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/openai

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/llamacpp-server

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/mlx

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/electron

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/ollama

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/node-llama-cpp

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/aws

### Features

- add bugs URL to package.json files across all packages and providers

### Chores

- update deps

### Updated Dependencies

- `@aws-sdk/client-sqs`: ^3.1068.0

## @workglow/anthropic

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/google-gemini

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/postgres

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/stable-diffusion-server

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/supabase

### Features

- add bugs URL to package.json files across all packages and providers

#### supabase

- add Supabase vector storage with pgvector support (#578)

## @workglow/playwright

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/sqlite

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/cloudflare

### Features

- add bugs URL to package.json files across all packages and providers

### Chores

- update deps

### Updated Dependencies

- `@cloudflare/workers-types`: ^4.20260612.1

## @workglow/huggingface-transformers

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/tf-mediapipe

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/chrome-ai

### Features

- add bugs URL to package.json files across all packages and providers

### Chores

- update deps

### Updated Dependencies

- `@types/dom-chromium-ai`: ^0.0.17

## @workglow/huggingface-inference

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/cactus

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/bun-webview

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/cli

### Features

- add bugs URL to package.json files across all packages and providers

## @workglow/web

### Features

- add bugs URL to package.json files across all packages and providers

### Chores

- update deps

### Updated Dependencies

- `@tailwindcss/vite`: ^4.3.1
- `tailwindcss`: ^4.3.1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants