Add catalog discovery (browse, crawl) and deepen the records#4
Merged
Conversation
browse pages the store search/results list the storefront scrolls, so it enumerates the whole catalog (around 260k apps) rather than the ten-hit cap of storesearch. Each row is a full app reference, so a hit feeds app, reviews, or crawl. crawl walks the typed kit:link edges breadth-first from an app, package, or profile, emitting one CrawlNode per visited record, keyed kind:id, best-effort past the seed. It uses a light appCore read so following an app's edges does not pay for the store-page and review-summary enrichment. The app record now captures the appdetails fields earlier versions read past: the per-board content ratings, the per-platform system requirements, the buy options from package_groups (folding any new package into the edges), the highlighted achievements with their localized names and icons, the extra store art and the legal/drm/external-account notices, and the demos as records and crawlable edges. It also folds in the canonical review summary from the appreviews query_summary: the review_score_desc label and the positive/negative/total counts. The package parser read page_content and header_image, keys the endpoint does not return, so its description and image were always empty. It now reads page_image, small_logo, the controller support, and the individual price. search now carries controller support and news carries feed_type. Tests cover the new app fields, the package fix, the catalog row parser (including the whitespace-wrapped responsive_secondrow release date), and the crawl breadth-first walk.
Add browse and crawl to the README usage block and the graph paragraph, a discovery section to the quick start, the two commands and their flags to the CLI reference, and a v0.3.0 release notes page covering the catalog discovery pair and the record audit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds two ways to discover the catalog and fills the gaps in what each record captures, after a deep audit of every data model, field, page, and piece of metadata against the live endpoints.
Discovery: browse and crawl
st browsereads the same paginatedsearch/resultslist the storefront calls as it scrolls, so it enumerates the whole catalog (a total around 260,000 apps) rather than the ten-hit cap ofstoresearch.Each row is a full app reference, so a hit feeds
st app,st reviews, orst crawl.Filters:
--query,--sort,--maxprice,--start.st crawlwalks the typedkit:linkedges breadth-first from an app, package, or profile.An app reaches its DLC, demos, base game, and packages; a package reaches its apps; a profile reaches its most-played apps.
It emits one
CrawlNodeper visited record (keyedkind:id), never revisits a node, and is best-effort past the seed.It uses a lightweight
appCoreread so following an app's edges does not pay for the store-page and review-summary enrichment.Flag:
--depth.A deeper app record
st appnow captures the appdetails fields earlier versions read past:pc_requirements,mac_requirements,linux_requirements), parsed leniently since the store returns an object when supported and an empty array when not.package_groups, with any package they name folded into the edges.capsule_image,capsule_imagev5,background_raw) and the legal, DRM, and external-account notices.It also folds in the canonical review summary from the appreviews
query_summary:review_score_desc(the "Overwhelmingly Positive" label),total_reviews,total_positive,total_negative.A fixed package record
st packagereadpage_contentandheader_image, keys the endpoint does not return, so its description and image were always empty.It now reads
page_image,small_logo, the controller support, and the individual price.Smaller additions
st searchcarries the controller support a hit reports, andst newscarries each item'sfeed_type.Upgrade notes
The
packagerecord dropped the always-emptydescriptionandheader_imagefields and replaced them withpage_imageandsmall_logo.Tests and gate
New tests cover the new app fields, the package fix, the catalog row parser (including the whitespace-wrapped
responsive_secondrowrelease date), and the crawl breadth-first walk.gofmt,go vet,go test ./...,golangci-lint, andCGO_ENABLED=0 go buildall pass.