Skip to content

Add catalog discovery (browse, crawl) and deepen the records#4

Merged
tamnd merged 2 commits into
mainfrom
discover-and-deepen-records
Jun 19, 2026
Merged

Add catalog discovery (browse, crawl) and deepen the records#4
tamnd merged 2 commits into
mainfrom
discover-and-deepen-records

Conversation

@tamnd

@tamnd tamnd commented Jun 19, 2026

Copy link
Copy Markdown
Owner

This adds two ways to discover the catalog and fills the gaps in what each record captures, after a deep audit of every data model, field, page, and piece of metadata against the live endpoints.

Discovery: browse and crawl

st browse reads the same paginated search/results list the storefront calls as it scrolls, so it enumerates the whole catalog (a total around 260,000 apps) rather than the ten-hit cap of storesearch.
Each row is a full app reference, so a hit feeds st app, st reviews, or st crawl.
Filters: --query, --sort, --maxprice, --start.

st crawl walks the typed kit:link edges breadth-first from an app, package, or profile.
An app reaches its DLC, demos, base game, and packages; a package reaches its apps; a profile reaches its most-played apps.
It emits one CrawlNode per visited record (keyed kind:id), never revisits a node, and is best-effort past the seed.
It uses a lightweight appCore read so following an app's edges does not pay for the store-page and review-summary enrichment.
Flag: --depth.

A deeper app record

st app now captures the appdetails fields earlier versions read past:

  • Per-board content ratings (ESRB, PEGI, USK, ...).
  • Per-platform system requirements (pc_requirements, mac_requirements, linux_requirements), parsed leniently since the store returns an object when supported and an empty array when not.
  • Buy options from package_groups, with any package they name folded into the edges.
  • Highlighted achievements with their localized names and icon URLs.
  • Extra store art (capsule_image, capsule_imagev5, background_raw) and the legal, DRM, and external-account notices.
  • Demos, as records and crawlable edges.

It also folds in the canonical review summary from the appreviews query_summary: review_score_desc (the "Overwhelmingly Positive" label), total_reviews, total_positive, total_negative.

A fixed package record

st package read page_content and header_image, keys the endpoint does not return, so its description and image were always empty.
It now reads page_image, small_logo, the controller support, and the individual price.

Smaller additions

st search carries the controller support a hit reports, and st news carries each item's feed_type.

Upgrade notes

The package record dropped the always-empty description and header_image fields and replaced them with page_image and small_logo.

Tests and gate

New tests cover the new app fields, the package fix, the catalog row parser (including the whitespace-wrapped responsive_secondrow release date), and the crawl breadth-first walk.
gofmt, go vet, go test ./..., golangci-lint, and CGO_ENABLED=0 go build all pass.

tamnd added 2 commits June 19, 2026 11:16
browse pages the store search/results list the storefront scrolls, so it
enumerates the whole catalog (around 260k apps) rather than the ten-hit cap
of storesearch. Each row is a full app reference, so a hit feeds app,
reviews, or crawl. crawl walks the typed kit:link edges breadth-first from
an app, package, or profile, emitting one CrawlNode per visited record,
keyed kind:id, best-effort past the seed. It uses a light appCore read so
following an app's edges does not pay for the store-page and review-summary
enrichment.

The app record now captures the appdetails fields earlier versions read
past: the per-board content ratings, the per-platform system requirements,
the buy options from package_groups (folding any new package into the
edges), the highlighted achievements with their localized names and icons,
the extra store art and the legal/drm/external-account notices, and the
demos as records and crawlable edges. It also folds in the canonical review
summary from the appreviews query_summary: the review_score_desc label and
the positive/negative/total counts.

The package parser read page_content and header_image, keys the endpoint
does not return, so its description and image were always empty. It now
reads page_image, small_logo, the controller support, and the individual
price. search now carries controller support and news carries feed_type.

Tests cover the new app fields, the package fix, the catalog row parser
(including the whitespace-wrapped responsive_secondrow release date), and
the crawl breadth-first walk.
Add browse and crawl to the README usage block and the graph paragraph, a
discovery section to the quick start, the two commands and their flags to
the CLI reference, and a v0.3.0 release notes page covering the catalog
discovery pair and the record audit.
@tamnd
tamnd merged commit 0c0337d into main Jun 19, 2026
7 checks passed
@tamnd
tamnd deleted the discover-and-deepen-records branch June 19, 2026 04:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant