Multi-tenancy seam: tenant columns, resolver, quotas, sharded ingest

[gaurav0107]

Implements the multi-tenancy spec (docs/multi-tenancy-spec.md) end-to-end. Cloud-only deployment per the office-hours decisions; on-prem / BYOC deferred.

Summary

• Schema: postgres 0024_multitenancy_seam.sql (plan/plan_meter_limit/quota_period/org.retention_days), ClickHouse 0006_tenant_columns.sql (8 trace tables swap to (org_id, project_id, …) ORDER BY via CREATE-INSERT-RENAME), ClickHouse 0007_audit_log.sql (audit_log + audit_reconciliation_gap).
• New shared module services/_shared/tenant/ (tracebility-tenant): TenantContext, Redis-cached Resolver with pubsub invalidation, GCRA RateLimiter, QuotaMeter, ShardRouter (blake2b mod 16), AuditWriter, EvalConcurrency semaphore, plus the quota + audit reconciler loops.
• ingest-api: auth chains through the resolver, every envelope stamps (org_id, workspace_id, project_id), enqueue routes to tracebility:ingest:v1:{0..15} keyed by hash(org_id). Per-plan rate limit (429) and per-org hard quota (402) at the edge.
• ingest-worker: writer threads tenant tuple into all CH row inserts; consumer reads from N shards round-robin and dual-reads the legacy stream during cutover; per-org consume-time quota filter (XACK+drop, never whole-shard stop) — the consume-time-filter refinement of spec §7.2.
• api: central ScopedClickHouse auto-injects and org_id = {org_id:UUID} into every CH WHERE; canary router runs_query migrated; workspace switcher (/v1/me/workspaces); audit egress dual-writes (postgres + CH); admin endpoints (/v1/admin/orgs/{id}/quotas, /audit, /audit/gaps).
• Cutover doc docs/multi-tenancy-cutover.md — sequencing, verification XLEN commands, the backlog-skew alert that's the trigger for the deferred weighted_map TODO.
• Property test statically asserts no router emits a tenant-aware CH query without org_id in WHERE (or ScopedClickHouse in scope, which auto-injects). 14 routers still on the raw ch.query() path are tracked in _PENDING_ROUTERS as remaining work; they all scope by project_id (FK-bound to org_id) so they're not security holes today, just no partition pruning.

Phase ↔ commit map

Phase	Commit
1: postgres 0024	7dd133d
2: clickhouse 0006	283bb70
3: clickhouse 0007	a59b07e
4: shared tenant module (+ phase 8 eval semaphore seam)	e9593fb
5: ingest-api	03ebd14
6: ingest-worker	b1de7c2
7+10: api + admin endpoints	ce927cb
9: cutover doc	306bb00
11: reconcilers	5b367bd
12: integration + property tests	6db1cdb

Test plan

• uv run pytest services/ — 46 passed against the live docker-compose stack (postgres, redis, clickhouse).
• Apply migrations to staging (0024_multitenancy_seam.sql first, then CH 0006, then CH 0007). Both CH migrations were validated against the local container during development.
• Deploy ingest-worker first (dual-reads legacy + sharded streams), then ingest-api, per docs/multi-tenancy-cutover.md.
• Watch XLEN tracebility:ingest:v1 decay to 0; flip TRACEBILITY_INGEST_DUAL_READ_LEGACY=false and roll the worker.
• Smoke-test: send a run through POST /v1/runs with a real api_key, confirm row in tracebility.run has the right org_id + workspace_id.
• Smoke-test: hit GET /v1/admin/orgs/{id}/quotas from an org-admin session, confirm meter bars render.
• Schedule the quota reconciler (60s) and audit reconciler (24h) in the operator service.

Deliberately deferred (each has a reactivation trigger in the spec)

• 14 routers still on the raw ch.query() path. Each is a 5-line _assert_project_access → resolve_project_scope swap; tracked in services/api/tests/test_property_org_id_in_where.py::_PENDING_ROUTERS.
• Web UI for /v1/admin/orgs/{id}/quotas and /audit (frontend not in this branch).
• eval-orchestrator service (the seam exists in tracebility_tenant.eval_concurrency; the service itself doesn't exist yet on main).
• Weighted shard map — TODO documented in services/_shared/tenant/tracebility_tenant/shard.py and the cutover runbook. Triggered by sustained backlog skew on a single shard.
• License JWT / on-prem self_hosted mode — collapsed into a future focused project per office-hours decision fix(migrator): version-track ClickHouse migrations (unblock helm-deploy) #7.

🤖 Generated with Claude Code