Skip to content

conn-pool: fix use-after-free crash during connectivity grid teardown - #45324

Merged
RyanTheOptimist merged 3 commits into
envoyproxy:mainfrom
danzh2010:fixconnectivitygrid
Jun 1, 2026
Merged

conn-pool: fix use-after-free crash during connectivity grid teardown#45324
RyanTheOptimist merged 3 commits into
envoyproxy:mainfrom
danzh2010:fixconnectivitygrid

Conversation

@danzh2010

@danzh2010 danzh2010 commented May 27, 2026

Copy link
Copy Markdown
Contributor

Description

This PR resolves a use-after-free (UAF) / pure virtual method call crash that occurs during ConnectivityGrid teardown when connection pools (HTTP/3, HTTP/2, or HTTP/1) are destroyed while active connection attempts are still ongoing in the background.


💥 The Bug & Crash Callstack

When the ConnectivityGrid is deleted, its destructor destroys its underlying connection pools in reverse order. If a pool has active connection attempts, deleting the pool synchronously cancels those attempts. In C++, once a derived class destructor finishes, the object's vtable pointer transitions to point to the base class vtable.

Because destructAllConnections() is invoked in ~HttpConnPoolImplBase() (the base class destructor), the connection cancellation callback wrappers execute onConnectionAttemptFailed() after the derived pool object has already been destructed.

When onConnectionAttemptFailed() attempts to write a log trace calling describePool(attempt->pool()) (which executes attempt->pool().protocolDescription()), virtual dispatch invokes a pure virtual function call on the base class HttpConnPoolImplBase, crashing Envoy instantly:

Program received signal SIGSEGV, Segmentation fault.
0x000056453663656e in Envoy::Http::ConnectivityGrid::WrapperCallbacks::onConnectionAttemptFailed()
Backtrace:
  #0  0x000056453663656e in Envoy::Http::ConnectivityGrid::WrapperCallbacks::onConnectionAttemptFailed()
  #1  0x000056453663606d in Envoy::Http::ConnectivityGrid::WrapperCallbacks::ConnectionAttemptCallbacks::onPoolFailure()
  #2  0x000056453664e7f1 in Envoy::Http::HttpConnPoolImplBase::onPoolFailure()
  #3  0x00005645366cb447 in Envoy::ConnectionPool::ConnPoolImplBase::purgePendingStreams()
  #4  0x00005645366c9d6e in Envoy::ConnectionPool::ConnPoolImplBase::onConnectionEvent()
  #5  0x000056453666394d in Envoy::Tcp::ActiveTcpClient::onEvent()
  #6  0x000056453709b0e9 in Envoy::Network::MockConnectionBase::raiseEvent()
  ...
  #24 0x00005645366be287 in Envoy::ConnectionPool::ConnPoolImplBase::destructAllConnections()
  #25 0x00005645366766b8 in Envoy::Http::HttpConnPoolImplBase::~HttpConnPoolImplBase()
  #26 0x000056453664e70f in Envoy::Http::HttpConnPoolImplMixed::~HttpConnPoolImplMixed()
  #27 0x000056453664e739 in Envoy::Http::HttpConnPoolImplMixed::~HttpConnPoolImplMixed()
  #28 0x000056453647099c in std::__1::default_delete<>::operator()()
  #29 0x000056453647091c in std::__1::unique_ptr<>::reset()
  #30 0x000056453663a3ad in Envoy::Http::ConnectivityGrid::~ConnectivityGrid()
  #31 0x000056453663a4f9 in Envoy::Http::ConnectivityGrid::~ConnectivityGrid()
  #32 0x00005645365de65c in std::__1::default_delete<>::operator()()
  #33 0x000056453656b55c in std::__1::unique_ptr<>::reset()
  #34 0x00005645363174fa in ConnectivityGridTest_DestroyGridWithActiveH2Attempts_Test::TestBody()

🛠️ The Fix

Introduced a reloadable feature flag guard envoy.reloadable_features.conn_pool_grid_early_return_on_teardown (enabled by default) to shield both H3 and H2/H1 pools during destruction:

  • Early Return: Inside WrapperCallbacks::onConnectionAttemptFailed(), we check if delete_started_ is true. If the flag is enabled and the wrapper is already being deleted/torn down, we return immediately, completely skipping the unsafe virtual method lookups on the half-destructed pool object.
  • ALPN & Protocol Safety: This safely protects the system regardless of which pool is being torn down (H3 or H2/H1 mixed pools).

🧪 Testing & Verification

  • Added the DestroyGridWithActiveH2Attempts unit test to verify that tearing down the ConnectivityGrid while an active H2 connection attempt is running does crash without the fix and not with the fix.

Risk Level: low, only exposed with log level trace
Testing: new unit test
Docs Changes: N/A
Release Notes: Y
Platform Specific Features: N/A
Runtime guard: envoy.reloadable_features.conn_pool_grid_early_return_on_teardown

@repokitteh-read-only

Copy link
Copy Markdown

CC @envoyproxy/runtime-guard-changes: FYI only for changes made to (source/common/runtime/runtime_features.cc).

🐱

Caused by: #45324 was opened by danzh2010.

see: more, trace.

@danzh2010
danzh2010 force-pushed the fixconnectivitygrid branch 2 times, most recently from e364d44 to cd23ba8 Compare May 28, 2026 13:38
Signed-off-by: Dan Zhang <danzh@google.com>
@danzh2010
danzh2010 force-pushed the fixconnectivitygrid branch from cd23ba8 to 7b70ddd Compare May 28, 2026 14:37
@danzh2010

Copy link
Copy Markdown
Contributor Author

/assign @RyanTheOptimist

@danzh2010 danzh2010 changed the title http: fix use-after-free crash during connectivity grid teardown conn-pool: fix use-after-free crash during connectivity grid teardown May 29, 2026

@RyanTheOptimist RyanTheOptimist left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@RyanTheOptimist
RyanTheOptimist enabled auto-merge (squash) May 31, 2026 23:41
@RyanTheOptimist

Copy link
Copy Markdown
Contributor

Looks like this needs a main merge.

Signed-off-by: danzh <danzh2010@users.noreply.github.com>
@danzh2010

Copy link
Copy Markdown
Contributor Author

merged main. PTAL!

Signed-off-by: Dan Zhang <danzh@google.com>
@RyanTheOptimist
RyanTheOptimist merged commit 2729a01 into envoyproxy:main Jun 1, 2026
28 checks passed
nezdolik pushed a commit to nezdolik/envoy that referenced this pull request Jun 16, 2026
…envoyproxy#45324)

### Description
This PR resolves a use-after-free (UAF) / pure virtual method call crash
that occurs during `ConnectivityGrid` teardown when connection pools
(HTTP/3, HTTP/2, or HTTP/1) are destroyed while active connection
attempts are still ongoing in the background.

---

### 💥 The Bug & Crash Callstack

When the `ConnectivityGrid` is deleted, its destructor destroys its
underlying connection pools in reverse order. If a pool has active
connection attempts, deleting the pool synchronously cancels those
attempts. In C++, once a derived class destructor finishes, the object's
vtable pointer transitions to point to the base class vtable.

Because `destructAllConnections()` is invoked in
`~HttpConnPoolImplBase()` (the base class destructor), the connection
cancellation callback wrappers execute `onConnectionAttemptFailed()`
after the derived pool object has already been destructed.

When `onConnectionAttemptFailed()` attempts to write a log trace calling
`describePool(attempt->pool())` (which executes
`attempt->pool().protocolDescription()`), virtual dispatch invokes a
**pure virtual function call** on the base class `HttpConnPoolImplBase`,
crashing Envoy instantly:

```log
Program received signal SIGSEGV, Segmentation fault.
0x000056453663656e in Envoy::Http::ConnectivityGrid::WrapperCallbacks::onConnectionAttemptFailed()
Backtrace:
  #0  0x000056453663656e in Envoy::Http::ConnectivityGrid::WrapperCallbacks::onConnectionAttemptFailed()
  #1  0x000056453663606d in Envoy::Http::ConnectivityGrid::WrapperCallbacks::ConnectionAttemptCallbacks::onPoolFailure()
  #2  0x000056453664e7f1 in Envoy::Http::HttpConnPoolImplBase::onPoolFailure()
  #3  0x00005645366cb447 in Envoy::ConnectionPool::ConnPoolImplBase::purgePendingStreams()
  envoyproxy#4  0x00005645366c9d6e in Envoy::ConnectionPool::ConnPoolImplBase::onConnectionEvent()
  envoyproxy#5  0x000056453666394d in Envoy::Tcp::ActiveTcpClient::onEvent()
  envoyproxy#6  0x000056453709b0e9 in Envoy::Network::MockConnectionBase::raiseEvent()
  ...
  envoyproxy#24 0x00005645366be287 in Envoy::ConnectionPool::ConnPoolImplBase::destructAllConnections()
  envoyproxy#25 0x00005645366766b8 in Envoy::Http::HttpConnPoolImplBase::~HttpConnPoolImplBase()
  envoyproxy#26 0x000056453664e70f in Envoy::Http::HttpConnPoolImplMixed::~HttpConnPoolImplMixed()
  envoyproxy#27 0x000056453664e739 in Envoy::Http::HttpConnPoolImplMixed::~HttpConnPoolImplMixed()
  envoyproxy#28 0x000056453647099c in std::__1::default_delete<>::operator()()
  envoyproxy#29 0x000056453647091c in std::__1::unique_ptr<>::reset()
  envoyproxy#30 0x000056453663a3ad in Envoy::Http::ConnectivityGrid::~ConnectivityGrid()
  envoyproxy#31 0x000056453663a4f9 in Envoy::Http::ConnectivityGrid::~ConnectivityGrid()
  envoyproxy#32 0x00005645365de65c in std::__1::default_delete<>::operator()()
  envoyproxy#33 0x000056453656b55c in std::__1::unique_ptr<>::reset()
  envoyproxy#34 0x00005645363174fa in ConnectivityGridTest_DestroyGridWithActiveH2Attempts_Test::TestBody()
```

---

### 🛠️ The Fix

Introduced a reloadable feature flag guard
`envoy.reloadable_features.conn_pool_grid_early_return_on_teardown`
(enabled by default) to shield both H3 and H2/H1 pools during
destruction:
* **Early Return**: Inside
`WrapperCallbacks::onConnectionAttemptFailed()`, we check if
`delete_started_` is `true`. If the flag is enabled and the wrapper is
already being deleted/torn down, we return immediately, completely
skipping the unsafe virtual method lookups on the half-destructed pool
object.
* **ALPN & Protocol Safety**: This safely protects the system regardless
of which pool is being torn down (H3 or H2/H1 mixed pools).

---

### 🧪 Testing & Verification

* Added the `DestroyGridWithActiveH2Attempts` unit test to verify that
tearing down the `ConnectivityGrid` while an active H2 connection
attempt is running does crash without the fix and not with the fix.

Risk Level: low, only exposed with log level trace
Testing: new unit test
Docs Changes: N/A 
Release Notes: Y
Platform Specific Features: N/A
Runtime guard:
envoy.reloadable_features.conn_pool_grid_early_return_on_teardown

Signed-off-by: Dan Zhang <danzh@google.com>
Co-authored-by: Dan Zhang <danzh@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants