perf: replace subprocess zstd with in-process klauspost/compress#217
perf: replace subprocess zstd with in-process klauspost/compress#217joshfriend wants to merge 2 commits intomainfrom
Conversation
Both compress and decompress now use the klauspost Go library instead of shelling out to zstd. The practical benefit isn't raw speed — zstd at 2 GB/s was never the bottleneck. It's fewer moving parts: no subprocess spawning, no kernel pipes for IPC, no scheduling jitter from coordinating across process boundaries. It also removes the runtime dependency on zstd being installed, making the binary more portable. Benchmarked on r8id.metal-48xlarge with a 2.4 GB compressed / 4.5 GB on-disk cache bundle (334K files).
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4e8abce973
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
joshfriend
force-pushed
the
jfriend/in-process-zstd
branch
2 times, most recently
from
d03e465 to
6a22063
Compare
If io.Copy returns early (e.g. upload interrupted), tar can block forever writing to its stdout pipe. Close the read end so tar gets SIGPIPE and exits. Also toned down the comment — we still have one kernel pipe (tar→Go), we eliminated the second one (tar→zstd subprocess).
joshfriend
force-pushed
the
jfriend/in-process-zstd
branch
from
6a22063 to
65df4f6
Compare
|
Have you benchmarked this? |
|
FYI the main reason for using system zstd/tar wasn't speed, it was compatibility. I'd be pretty surprised if scheduling jitter was a concern for cachew too... |
|
I discovered during playpen testing that Go klauspost/compress zstd decompression is catastrophically slow for git mirror snapshots that contain multi-GB packfiles. a very large repo mirror snapshot has a single 6.8 GB pack file. The Go zstd decoder feeds through Go's For cachew's use cases (git mirror snapshots with large packfiles), the 15% small-file improvement doesn't justify a 30x regression on large files. Keeping the subprocess pipeline. |
2 participants
Both compress and decompress now use the klauspost Go library instead of shelling out to the zstd binary. The practical benefit isn't raw speed — zstd at 2 GB/s was never the bottleneck. It's fewer moving parts: no subprocess spawning, no kernel pipes for IPC, no scheduling jitter from coordinating across process boundaries. It also removes the runtime dependency on zstd being installed, making the binary more portable.
Benchmarked on r8id.metal-48xlarge with a 2.4 GB compressed / 4.5 GB on-disk cache bundle (334K files).