feat(006): API datastore abstraction with memdb and ScyllaDB backends#138
Merged
Conversation
Introduces a pluggable Datastore interface decoupling the GraphQL API from its persistence layer. Backend is selected at runtime via GITSTORE_DATASTORE_BACKEND (default: memdb). ScyllaDB backend auto-migrates on first start using a distributed LWT lock to prevent concurrent migration races. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d2d55da557
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- ScyllaDB backend: open a keyspace-free session for migrations then a keyspaced session for data ops; fixes "Keyspace does not exist" on first start because gocql refused to connect before CREATE KEYSPACE ran - ScyllaDB tests: switch to TestMain/shared-container pattern; eliminates per-test container spawning that exhausted aio-max-nr on GitHub Actions - Integration test: skip TestTagPushPublishesToGraphQL with an explanatory message; the git→datastore ingestion path was removed in feature 006 and is tracked as a future pipeline task Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…it step The application now assumes the keyspace already exists (operator's responsibility, enforced by the compose scylla-init service). Changes: - Remove CREATE KEYSPACE from migration CQL files; use unqualified table names so migrations run in the session keyspace - RunMigrations creates schema_migrations_lock itself (migration runner infrastructure, like Flyway's own schema table) - backend.New uses a single keyspace-scoped session; no bootstrap logic - compose.scylla.yml adds a scylla-init service that runs cqlsh to create the keyspace before the api service starts - testcontainers TestMain provisions the keyspace via a no-keyspace session before any test opens a store, mirroring the compose init pattern Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…kend
- Switch gocql fork from v1.18.0 to v1.15.3 to fix shard-aware port
issues with Docker mapped ports on macOS (Rancher Desktop)
- Use MapScanCAS instead of ScanCAS for LWT INSERT/DELETE so variable
column counts on conflict do not cause unmarshal errors
- Embed migrations as a flat FS via fs.Sub so gocqlx/migrate can locate
*.cql files; merge 002_add_indexes.cql into 001_initial_schema.cql
- Add `-- CALL await_tables;` callback to poll tables after CREATE TABLE
before running CREATE INDEX — on single-node --developer-mode=1 ScyllaDB
schema agreement returns immediately but storage may not be initialised
- Place a semicolon in the header comment block so ReadString(';') splits
it off before CREATE TABLE, preventing isComment from swallowing the
first CREATE TABLE statement
- Pass explicit non-key column list to table.Update() to avoid empty SET
clause (CQL parse error) in UpdateProduct/Category/Collection
- Add DisableShardAwarePort config field and parse host:port in parseHosts;
set cluster.Port explicitly (gocql.NewCluster does not parse ports)
- Use wait.ForExec(cqlsh) and 30 s provisionKeyspace retry loop to handle
ScyllaDB accepting connections slightly after its startup log message
- price column is decimal (CQL) / *inf.Dec (Go) not double/float64
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ackend errors - Move table metadata from package-level vars into the scyllaDatastore struct, built at New() time with the configured keyspace prefix so deployments using a non-default keyspace are routed correctly - Thread keyspace into RunMigrations and awaitTablesCallback so the await-poll queries also target the right keyspace - Distinguish gocql.ErrNotFound from other errors in all Get* methods: only ErrNotFound maps to datastore.ErrNotFound; timeouts and network failures are surfaced as-is instead of masking them as missing entities - Update* and Delete* now propagate the error from their existence check directly rather than double-wrapping it Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This was
linked to
issues
May 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
datastore.Datastoreinterface (18 CRUD methods +Close) decoupling the GraphQL API from its persistence layer; backend selected at startup viaGITSTORE_DATASTORE_BACKEND(default:memdb)go-memdbbackend for zero-dependency local development andScyllaDBbackend for production, with automatic schema migrations on first start (distributed LWT lock prevents concurrent races)InstrumentedDatastore— per-operation Prometheus histogram + error counter + structured zap error logRunContractSuite(30 sub-tests); moves gRPC tests totests/contract/grpc/; adds//go:build scyllaintegration tests via testcontainersWhat changed
internal/datastore/Datastoreinterface,entities.go,InstrumentedDatastore, metrics, memdb + ScyllaDB backends, factoryinternal/graph/Servicerewritten to useDatastore;cache.Managerremoved from the service layerinternal/loader/*catalog.CatalogtoDatastoreinternal/health/ListProductsinstead of cache pingcmd/server/main.gofactory.NewDatastore+InstrumentedDatastoreat startuptests/contract/compose.scylla.ymlcompose.admin.ymlpattern).github/workflows/ci-integration.ymldatastore-contract-test-scyllajob; restoredintegration-testjobTest plan
go test -race ./...— 13 packages, all passgo fmt,go vet,staticcheck— clean--alland--diff-base origin/main) — passgo test -tags scylla -timeout 10m ./tests/contract/datastore/... ./internal/datastore/scylla/...— runs in CI viadatastore-contract-test-scyllajob (requires Docker)go test -tags grpc -timeout 5m ./tests/contract/grpc/...— runs in CI viagrpc-contract-testjob🤖 Generated with Claude Code