Keep search, geo and metrics output lossless#8
Open
tamnd wants to merge 2 commits into
Open
Conversation
The search, geosearch, revisions, categories, langlinks, backlinks, pageviews, top and stats commands dropped fields the API returns, so their JSON could not reconstruct the original record. Widen each struct to the full property set and request it: - search/related/random keep the page id, key, matched_title and thumbnail - geosearch sets gsprop and keeps page id, namespace, primary, type, name, dim, country, region and globe - revisions keep the user id, parsed comment and sha1, and carry change tags as an array - categories keep the sort key, prefix, timestamp and hidden flag; members keep their page id - langlinks keep the langname alongside the autonym - backlinks keep the page id - pageviews keep the per-item project, article, access, agent and granularity; top keeps the article key, project, access and date - stats widens siteinfo with the server, paths, software versions, wiki id, server time and job count Table and download views are unchanged; new accessor methods cover the cases where a column needs a flat string. Adds decode tests and updates the guides.
The output reference listed the formats but never said the structured ones carry every field the API returns while the table shows a curated subset. Spell that out so readers know to reach for -o json when they want a field the table does not show.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
The search, geo and metrics commands decoded only a subset of what their
APIs return, so
-o json/-o jsonlcould not reconstruct the originalrecord. This is the last domain in the losslessness sweep.
Each struct is widened to the full property set, and the request asks for it
(
gsprop,clprop,rvprop,llprop, widersiprop). Table and downloadviews are unchanged; where a column needs a flat string, an accessor method
covers it (e.g.
Revision.TagsLine()). Presence-only keys (primary,hidden,writeapi) and the number-or-stringdimdecode throughjson.RawMessage.Coverage
key,matched_title,thumbnaillangname; backlinks keep page idBefore / after
Before,
searchemitted onlytitle,description,snippet,url. After:{"title":"Alan Turing","pageid":1208,"key":"Alan_Turing","description":"English computer scientist (1912–1954)","snippet":"...","thumbnail":{"url":"//upload.wikimedia.org/.../60px-Alan_turing_header.jpg","mimetype":"image/jpeg","width":60,"height":80},"url":"https://en.wikipedia.org/wiki/Alan_Turing"}Before,
geosearchemitted onlytitle,lat,lon,dist,url. After:{"pageid":9232,"title":"Eiffel Tower","lat":48.85822,"lon":2.2945,"dist":19.8,"primary":true,"dim":1000,"country":"FR","region":"75","url":"https://en.wikipedia.org/wiki/Eiffel_Tower"}revisionsnow carriesuserid,sha1and atagsarray;statsreportsthe server, paths, versions, wiki id and server time.
Tests
wiki/search_geo_metrics_test.godecodes representative API blobs and assertsthe new fields survive (thumbnail/key, geo props with presence
primaryandstring
dim, revision user id + tags array, category sortkey/hidden,langname, wider siteinfo, pageview context).
go test ./...green,golangci-lintclean. Verified live against the API.