Skip to content

Keep search, geo and metrics output lossless#8

Open
tamnd wants to merge 2 commits into
mainfrom
search-geo-metrics-lossless
Open

Keep search, geo and metrics output lossless#8
tamnd wants to merge 2 commits into
mainfrom
search-geo-metrics-lossless

Conversation

@tamnd

@tamnd tamnd commented Jun 13, 2026

Copy link
Copy Markdown
Owner

What and why

The search, geo and metrics commands decoded only a subset of what their
APIs return, so -o json/-o jsonl could not reconstruct the original
record. This is the last domain in the losslessness sweep.

Each struct is widened to the full property set, and the request asks for it
(gsprop, clprop, rvprop, llprop, wider siprop). Table and download
views are unchanged; where a column needs a flat string, an accessor method
covers it (e.g. Revision.TagsLine()). Presence-only keys (primary,
hidden, writeapi) and the number-or-string dim decode through
json.RawMessage.

Coverage

  • search / related / random keep page id, key, matched_title, thumbnail
  • geosearch keeps page id, ns, primary, type, name, dim, country, region, globe
  • revisions keep user id, parsed comment, sha1; tags become an array
  • categories keep sort key, prefix, timestamp, hidden; members keep page id
  • langlinks keep langname; backlinks keep page id
  • pageviews keep per-item project/article/access/agent/granularity; top keeps article key/project/access/date
  • stats widens siteinfo with server, paths, software versions, wiki id, server time, jobs

Before / after

Before, search emitted only title, description, snippet, url. After:

{"title":"Alan Turing","pageid":1208,"key":"Alan_Turing","description":"English computer scientist (1912–1954)","snippet":"...","thumbnail":{"url":"//upload.wikimedia.org/.../60px-Alan_turing_header.jpg","mimetype":"image/jpeg","width":60,"height":80},"url":"https://en.wikipedia.org/wiki/Alan_Turing"}

Before, geosearch emitted only title, lat, lon, dist, url. After:

{"pageid":9232,"title":"Eiffel Tower","lat":48.85822,"lon":2.2945,"dist":19.8,"primary":true,"dim":1000,"country":"FR","region":"75","url":"https://en.wikipedia.org/wiki/Eiffel_Tower"}

revisions now carries userid, sha1 and a tags array; stats reports
the server, paths, versions, wiki id and server time.

Tests

wiki/search_geo_metrics_test.go decodes representative API blobs and asserts
the new fields survive (thumbnail/key, geo props with presence primary and
string dim, revision user id + tags array, category sortkey/hidden,
langname, wider siteinfo, pageview context). go test ./... green, golangci-lint clean. Verified live against the API.

tamnd added 2 commits June 13, 2026 22:27
The search, geosearch, revisions, categories, langlinks, backlinks,
pageviews, top and stats commands dropped fields the API returns, so
their JSON could not reconstruct the original record.

Widen each struct to the full property set and request it:

- search/related/random keep the page id, key, matched_title and
  thumbnail
- geosearch sets gsprop and keeps page id, namespace, primary,
  type, name, dim, country, region and globe
- revisions keep the user id, parsed comment and sha1, and carry
  change tags as an array
- categories keep the sort key, prefix, timestamp and hidden flag;
  members keep their page id
- langlinks keep the langname alongside the autonym
- backlinks keep the page id
- pageviews keep the per-item project, article, access, agent and
  granularity; top keeps the article key, project, access and date
- stats widens siteinfo with the server, paths, software versions,
  wiki id, server time and job count

Table and download views are unchanged; new accessor methods cover
the cases where a column needs a flat string. Adds decode tests and
updates the guides.
The output reference listed the formats but never said the structured
ones carry every field the API returns while the table shows a curated
subset. Spell that out so readers know to reach for -o json when they
want a field the table does not show.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant