Skip to content

Dump stack trace during consumer shutdown#100857

Merged
fpacifici merged 1 commit into
masterfrom
fpacifici/dump_stack_trace
Oct 3, 2025
Merged

Dump stack trace during consumer shutdown#100857
fpacifici merged 1 commit into
masterfrom
fpacifici/dump_stack_trace

Conversation

@fpacifici

Copy link
Copy Markdown
Contributor

We have been seeing a number of cases where, on small workloads, upon consumer restart,
a parallel Kafka Consuemr may hang right after partitions are assigned.
In this scenario the consumer does not pick up any work and, after 30 seconds is kicked
out of the consumer group.

The consumer is then restarted by the liveness probe 5 minutes later.

This PR dumps the stack trace of all threads upon SIGINT if the consumer
is listed in a specific option consumer.dump_stacktrace_on_shutdown.

This way we should be able to see where the consumer gets stuck, thus find a fix.

@github-actions github-actions Bot added the Scope: Backend Automatically applied to PRs that change backend components label Oct 3, 2025
@fpacifici
fpacifici requested a review from a team October 3, 2025 00:59
@codecov

codecov Bot commented Oct 3, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 50.00000% with 4 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/sentry/utils/kafka.py 42.85% 4 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           master   #100857      +/-   ##
===========================================
+ Coverage   79.82%    81.14%   +1.31%     
===========================================
  Files        8612      8613       +1     
  Lines      382248    382309      +61     
  Branches    24045     24045              
===========================================
+ Hits       305133    310207    +5074     
+ Misses      76787     71774    -5013     
  Partials      328       328              

@fpacifici
fpacifici merged commit 0b9316b into master Oct 3, 2025
95 of 97 checks passed
@fpacifici
fpacifici deleted the fpacifici/dump_stack_trace branch October 3, 2025 16:14
@sentry

sentry Bot commented Oct 3, 2025

Copy link
Copy Markdown
Contributor

Issues attributed to commits in this pull request

This pull request was merged and Sentry observed the following issues:

@github-actions github-actions Bot locked and limited conversation to collaborators Oct 31, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Scope: Backend Automatically applied to PRs that change backend components

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants