Keep every revision and its metadata when streaming a dump#4
Open
tamnd wants to merge 1 commit into
Open
Conversation
A page record used to flatten to a single revision: one id, one timestamp, the text, and a redirect bool. That threw away most of what the XML schema carries, and it could not represent a meta-history dump at all, where a page holds its whole revision list. DumpPage now keeps a Revisions slice. Each revision carries its id, parent id, timestamp, contributor (a registered user's name and id, or an anonymous edit's IP, or a deleted marker), minor flag, comment, content model and format, origin, sha1, and byte length. The page keeps its redirect target title and edit/move restrictions. Latest() and LatestText() give callers the current revision, which is what the pages/grep/export commands want for a pages-articles dump. The dump list output also gained md5, the job status, and the job's updated time, so -o json mirrors dumpstatus.json.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A streamed dump page used to flatten to a single revision: one id, one timestamp, the text, and a redirect bool. That dropped most of what the MediaWiki XML schema carries, and it could not represent a
pages-meta-historydump at all, where a page holds its full revision list.This makes the dump output lossless.
Changes
DumpPagenow keeps aRevisionsslice plus the page'sRedirectTitleandRestrictions.DumpRevisioncarries id, parent id, timestamp, contributor, minor flag, comment, content model and format, origin, sha1, and byte length.Contributordistinguishes a registered user (username + id), an anonymous edit (IP), and a deleted author.Latest()/LatestText()return the current revision, which is whatpages,grep, andexportuse on a pages-articles dump.dump listoutput gainedmd5, the jobstatus, and the jobupdatedtime, so-o jsonmirrorsdumpstatus.json.Before / after
A history page now round-trips in full:
```json
{
"id": 25202,
"ns": 0,
"title": "Quantum mechanics",
"restrictions": "edit=sysop",
"revisions": [
{"id": 100, "timestamp": "2020-01-01T00:00:00Z",
"contributor": {"username": "Alice", "id": 42},
"minor": true, "comment": "first", "model": "wikitext",
"format": "text/x-wiki", "origin": 100, "sha1": "abc",
"textbytes": 9, "text": "old text."},
{"id": 101, "parentid": 100, "timestamp": "2021-02-03T04:05:06Z",
"contributor": {"ip": "10.0.0.1"},
"comment": "second", "model": "wikitext", "format": "text/x-wiki",
"origin": 101, "sha1": "def", "textbytes": 9, "text": "new text."}
]
}
```
Before, that page emitted only
{id, ns, title, revid: 100, timestamp, redirect}and could not show the second revision, the contributors, or the restrictions.Tests
TestStreamPagesFullRevisionMetadataasserts multiple revisions, registered and anonymous contributors, redirect target, restrictions, and the per-revision fields all survive the parse.go build,go vet,go test ./..., andgolangci-lint runall clean.