fix: run block simulation on dedicated tokio runtime#233
fix: run block simulation on dedicated tokio runtime#233
Conversation
The simulator was running CPU-heavy EVM work on the main tokio runtime, starving the healthcheck handler and causing liveness probe kills (exit code 137) in Kubernetes. This creates a dedicated multi-thread runtime sized to CONCURRENCY_LIMIT for simulation, keeping the main runtime free for healthchecks, cache I/O, and channel operations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
So, this technically would fix the main issue: The main tokio thread's getting blocked by the simulation work, which is blocking and CPU heavy. The problem is that we're spawning an entirely new tokio runtime for this which increases resource usage just for isolating the sim task. On top of this, assigning CONCURRENCY_LIMIT worker threads is a tad wasteful since the sim runtime only runs one task at a time, which is the simulation work.
What we actually want is to isolate the single task, CPU heavy work from blocking the main tokio thread so that it can schedule handling healthchecks and other tasks while also continuing the sim work. Tokio can do this with spawn_blocking, which shifts the spawned task into its blocking task thread pool, avoiding the contention issue. I've implemented the changes on a PR on top of this: see #234
This stack of pull requests is managed by Graphite. Learn more about stacking. |
|
This should be closed now, yeah? |
|
yep |

Summary
BlockBuild::build()) on the main tokio runtime, which also serves the/healthcheckendpoint. WithCONCURRENCY_LIMIT=10and a 2-CPU pod, all tokio worker threads were saturated with EVM work, preventing the healthcheck from responding within the 1s liveness probe timeout — causing repeated SIGKILL (exit code 137) and a crash loop.tokio::runtime::Runtimefor simulation, sized toCONCURRENCY_LIMITworker threads. The main runtime stays free for healthchecks, cache I/O, and channel operations regardless of simulation load.Test plan
make clippyandmake testpass (confirmed locally)🤖 Generated with Claude Code