Add discover: a breadth-first walk of the page graph#13
Merged
Conversation
Every structure command answers one question about one page: its links, its categories, a category's members. discover chains them. From a seed page or category it follows the object's edges, then each neighbor's, hop by hop, streaming one row per object it reaches. The walker is built against a six-method grapher interface, an exact subset of the client, so the BFS logic is tested over an in-memory fake with no network. Neighbors arrive pre-hydrated from the list they came in on, so a depth-1 walk costs one seed fetch plus the seed's list calls and nothing per neighbor. Five edges (links, backlinks, categories, members, subcats) across two node kinds (page, category), grouped into presets: content (the default), network, cats, all. --follow takes a preset or a bare edge list; the two vocabularies are disjoint so a preset can never shadow an edge. Wikipedia's API is uniformly open, so unlike the sibling discover commands there are no gated edges and no tier degradation. wiki keeps no local database, so the walk streams to stdout; pipe it to keep it. A bad seed is fatal like any single read; a flaky edge deeper in is a note and the walk continues.
The walk was documented inside the page-structure guide, where it sat awkwardly next to the single-edge reads. Pull it into a dedicated Discovering guide so it reads as the one command that chains the others, matching how the rest of the CLI tools document their graph walk. The structure guide keeps a short pointer, and the guide index now lists Discovering as its own entry.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A new
discovercommand (aliaseswalk,graph) that walks the Wikipedia graph breadth first. The structure commands each follow one edge of one page;discoverchains them, from a seed page or category outward, streaming one row per object it reaches.Shape
links,backlinks,categories,members,subcats) across two node kinds (page,category), grouped into presetscontent(default),network,cats,all.--followtakes a preset or a bare edge list; the two vocabularies are disjoint so a preset can never shadow an edge.grapherinterface that is an exact subset of*Client, so the BFS is tested over an in-memory fake with no network.--depth(default 1; 0 = seeds only),--fanoutper edge (default 25), and an-nstreaming budget (default 500) so a bare walk always terminates.Adapted to Wikipedia
Unlike the sibling
discovercommands in the other tamnd CLIs:wikikeeps no local crawl database, so the walk is streaming-only (no--store); pipe it to keep it.Error policy: a seed that cannot be fetched is fatal (like any single read); a flaky edge deeper in the walk becomes a one-line note on stderr and the walk continues.
Tests
wiki/discover_test.gois hermetic over afakeGraph: edge/preset parsing, edge source/target, seed parsing, BFS order andviatagging from page and category seeds, thenetworkbacklink path, cycle dedup, theMaxbudget,--fanoutcapping, depth-0 seeds-only, fatal missing seed, deeper-error-degrades-to-note, and multiple seeds.Docs & spec
8002_wikipedia.md(graph taxonomy, BFS contract, the open-API / no-store contrasts).discoversection in the Page structure guide, a guides-index mention, and a reference CLI row.CGO_ENABLED=0 go build/vet/test ./...green,gofmtclean, no em-dashes in code or docs.