[bot] Migrate direct raw-S3 bbop-sqlite consumers to the semanticsql CDN (tracking)

[kltm]

[bot] Opened by a Claude Code agent on behalf of @kltm.

Companion to #112 (remove raw-bucket access). Some downstream projects hardcode the raw bucket URL (https://s3.amazonaws.com/bbop-sqlite/… or bbop-sqlite.s3.amazonaws.com/…) directly, rather than going through OAK's sqlite:obo: selector, so they will not auto-migrate when OAK/ODK are bumped. These need to repoint to https://semanticsql.berkeleybop.io before raw-bucket access is removed.

Note: the CDN returns 403 to the default Python-urllib User-Agent (Browser Integrity Check), so a naive URL swap must also set a non-default User-Agent.

Candidate direct-hardcoder consumers (from a GitHub code search — verify before repointing; external repos, tracked here for coordination, their maintainers to action):

• biobricks-ai/semsql — stages/01_download.sh (loops the whole catalog on push; heaviest single consumer)
• Knowledge-Graph-Hub/kg-alzheimers — src/kg_alzheimers/download.yaml
• Knowledge-Graph-Hub/kg-microbe — download.yaml
• Knowledge-Graph-Hub/universalizer — universalizer/oak_utils.py
• microbiomedata/ontology-loader — src/ontology_loader/ontology_processor.py
• ccb-hms/NHANES-metadata — code/generate_ontology_tables.py
• vjcitn/ontoProc2 — R/semsql_url.R, R/bbop_sqlite_db_gz.R
• monarch-initiative/monarch-ingest — src/monarch_ingest/download.yaml
• monarch-initiative/dictybase-ingest — download.yaml

Owned repos handled via their own PRs (not this list): monarch-initiative/monarch-app (HP_DB_URL), monarch-initiative/dismech (scripts/fetch_ontology_dbs.sh).

— Posted by Claude Code agent on behalf of @kltm.

[github-actions]

Summary + live status check (2026-07-14)

This is a tracking issue in a chain:

• Provide sqlite via a different URL #110 (closed) — established the vendor-neutral semanticsql.berkeleybop.io CDN URL.
• Eliminate S3 bucket access #112 (open) — plan to eliminate raw S3 (bbop-sqlite) bucket access once traffic has shifted to the CDN.
• [bot] Cut a semsql release including the semanticsql.berkeleybop.io default #114 (open) — blocker: the CDN-default change landed on master but hasn't been released to PyPI yet (semsql is still 0.4.0, from 2025-02-05, as of this check). Anyone relying on semsql download / the library default (rather than a hardcoded URL) won't get the CDN switch until that release ships.
• [bot] Migrate direct raw-S3 bbop-sqlite consumers to the semanticsql CDN (tracking) #115 (this issue) — the subset of consumers that bypass both OAK's selector and the semsql package entirely by hardcoding the raw S3 URL, so neither Eliminate S3 bucket access #112's traffic-shift nor [bot] Cut a semsql release including the semanticsql.berkeleybop.io default #114's release will fix them.

Verified current state of the checklist

I fetched each listed file directly (as of 2026-07-14) to confirm none have migrated yet — all 9 repos / 10 files are still unmigrated:

Repo	File	Status
biobricks-ai/semsql	stages/01_download.sh	❌ still https://s3.amazonaws.com/bbop-sqlite/${onto}.db.gz (loops full catalog — highest-traffic consumer)
Knowledge-Graph-Hub/kg-alzheimers	src/kg_alzheimers/download.yaml	❌ still S3 (e.g. ddpheno.db, phenio.db.gz)
Knowledge-Graph-Hub/kg-microbe	download.yaml (default branch is master, not main)	❌ still S3 (ncit.db.gz, bto.db.gz, po.db.gz, ...)
Knowledge-Graph-Hub/universalizer	universalizer/oak_utils.py	❌ still f"https://s3.amazonaws.com/bbop-sqlite/{db}.db"
microbiomedata/ontology-loader	src/ontology_loader/ontology_processor.py	❌ still ontology_db_url_prefix = "https://s3.amazonaws.com/bbop-sqlite/"
ccb-hms/NHANES-metadata	code/generate_ontology_tables.py	❌ still "https://s3.amazonaws.com/bbop-sqlite/%s.db.gz"
vjcitn/ontoProc2	R/semsql_url.R	❌ still "https://s3.amazonaws.com/bbop-sqlite/%s.db.gz" (default ontology "efo")
vjcitn/ontoProc2	R/bbop_sqlite_db_gz.R	❌ still url = "https://s3.amazonaws.com/bbop-sqlite" default arg
monarch-initiative/monarch-ingest	src/monarch_ingest/download.yaml	❌ still S3 (phenio.db.gz)
monarch-initiative/dictybase-ingest	download.yaml	❌ still S3 (ddpheno.db)

I also searched all 9 downstream repos for any existing issue/PR mentioning semanticsql.berkeleybop.io or bbop-sqlite — none found. So outreach to these external maintainers hasn't started yet; this checklist accurately reflects zero progress so far.

Notes for whoever actions this

• The CDN's 403-to-default-User-Agent behavior (noted in the issue body) will bite non-Python clients too — worth flagging explicitly for vjcitn/ontoProc2 (R's download.file/httr also send a default UA that could be blocked) and biobricks-ai/semsql (shell curl/wget).
• biobricks-ai/semsql is worth prioritizing given it "loops the whole catalog on push" — it's the single heaviest hardcoded consumer of the raw bucket by request volume.
• Since none of these 9 repos yet have an issue opened against them, and this tracking issue only coordinates (doesn't action), consider whether outreach issues should be filed in each downstream repo — e.g. via @dragon-ai-agent please open an issue in each of the 9 downstream repos listed here, linking back to INCATools/semantic-sql#115, asking maintainers to repoint their hardcoded bbop-sqlite S3 URLs to https://semanticsql.berkeleybop.io (with a non-default User-Agent) before the raw bucket is retired per #112.
• Recommend not closing raw-bucket access (Eliminate S3 bucket access #112) until (a) [bot] Cut a semsql release including the semanticsql.berkeleybop.io default #114's release ships and (b) at least the high-traffic items here (biobricks-ai/semsql, kg-microbe, monarch-ingest) have migrated — the S3 access logs mentioned in Eliminate S3 bucket access #112 should make it easy to confirm before flipping the switch.

[caufieldjh]

Downstream outreach filed (2026-07-14)

Opened repointing issues in the downstream repos listed above. Each links back here and to #112, cites the specific hardcoded file(s), gives the s3.amazonaws.com/bbop-sqlite/ → semanticsql.berkeleybop.io/ swap, and flags the 403-to-default-User-Agent gotcha (tailored per client: shell curl/wget, Python urllib/requests, R download.file/httr).

Repo	File(s)	Outreach issue
biobricks-ai/semsql	stages/01_download.sh	biobricks-ai/semsql#1
Knowledge-Graph-Hub/kg-alzheimers	src/kg_alzheimers/download.yaml	Knowledge-Graph-Hub/kg-alzheimers#78
Knowledge-Graph-Hub/kg-microbe	download.yaml (master)	Knowledge-Graph-Hub/kg-microbe#595
Knowledge-Graph-Hub/universalizer	universalizer/oak_utils.py	Knowledge-Graph-Hub/universalizer#41
ccb-hms/NHANES-metadata	code/generate_ontology_tables.py	ccb-hms/NHANES-metadata#37
vjcitn/ontoProc2	R/semsql_url.R, R/bbop_sqlite_db_gz.R	vjcitn/ontoProc2#11
monarch-initiative/monarch-ingest	src/monarch_ingest/download.yaml	monarch-initiative/monarch-ingest#711
monarch-initiative/dictybase-ingest	download.yaml	monarch-initiative/dictybase-ingest#3
microbiomedata/ontology-loader	src/ontology_loader/ontology_processor.py	⚠️ not filed — see below

⚠️ microbiomedata/ontology-loader could not be filed automatically

The microbiomedata org rejects classic personal access tokens for API writes:

microbiomedata forbids access via a personal access token (classic). Please use a GitHub App, OAuth App, or a personal access token with fine-grained permissions.

So this one needs to be opened manually (or with a fine-grained token / GitHub App that the org allows). The target is src/ontology_loader/ontology_processor.py — ontology_db_url_prefix = "https://s3.amazonaws.com/bbop-sqlite/" → "https://semanticsql.berkeleybop.io/", plus the non-default User-Agent requirement.

[caufieldjh]

Opened the issue in microbiomedata/ontology-loader manually: microbiomedata/ontology-loader#59