Decode several months in parallel on a roomy host#12
Merged
Conversation
The zstd decoder was gated to one month at a time by a global mutex, to keep a small box from holding two 2 GB windows at once. That gate is the throughput wall for a backlog of huge months: even with four process slots, giant dumps decode strictly one after another. Make the gate a counting semaphore sized from the budget. ComputeBudget grants a second or third decode slot only when RAM can hold the extra windows plus their conversions (about 8 GB per slot), never more than the process slots. ARCTIC_MAX_DECODES overrides it for hand tuning. A small box still decodes one at a time.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The zstd decoder was serialized to one month at a time by a global mutex, a guard against a small box holding two 2 GB decode windows at once. For a backlog of huge recent-year months that gate is the throughput wall: even with four process slots, giant dumps decode strictly one after another while the other slots idle.
This turns the gate into a counting semaphore sized from the budget:
ComputeBudgetgrants a second or third decode slot only when RAM can hold the extra 2 GB windows plus their conversions (budgeted at ~8 GB per decode slot), and never more than the process-slot count.ARCTIC_MAX_DECODESoverrides it for hand tuning and monitoring.acquireDecodercaptures the current gate so a later resize never routes a release to the wrong channel.On a 23 GB / 8-core host this allows 2 concurrent decodes, roughly doubling throughput on a backlog of large months.
Tested:
decode_gate_test.gocovers the concurrency cap (peak == 2 at size 2), strict serialization at size 1, the zero-size floor, and the budget/override math. Full arctic package passes with-race.