Skip to content

fix(metrics): resolve thread-safety race condition in MetricsInterceptor#1491

Open
sinhasubham wants to merge 1 commit intomainfrom
fix/already_exists_409
Open

fix(metrics): resolve thread-safety race condition in MetricsInterceptor#1491
sinhasubham wants to merge 1 commit intomainfrom
fix/already_exists_409

Conversation

@sinhasubham
Copy link
Contributor

Title:
fix(metrics): resolve thread-safety race condition in MetricsInterceptor

Description: This PR fixes a thread-safety regression introduced in v3.60.0 that caused AlreadyExists: 409 errors during parallel batch operations.

The Bug: The
MetricsInterceptor was accessing the global SpannerMetricsTracerFactory.current_metrics_tracer after the gRPC call completed. In concurrent scenarios, this global variable could be overwritten by another thread starting a new request, leading to the interceptor attempting to record completion on the wrong (or uninitialized) tracer. This resulted in an AttributeError crash, which the client library would catch and retry, causing a duplicate INSERT attempt and the 409 error.

The Fix:

Thread Safety: The intercept method now captures the current_metrics_tracer into a local variable before making the gRPC call. This guarantees that record_attempt_completion uses the same tracer instance that started the attempt, regardless of what other threads do.

Testing: Added test_intercept_thread_safety which uses threading and mocks to deterministically reproduce the race condition and verify the fix.

@sinhasubham sinhasubham requested review from a team as code owners February 3, 2026 17:52
@product-auto-label product-auto-label bot added size: m Pull request size is medium. api: spanner Issues related to the googleapis/python-spanner API. labels Feb 3, 2026
@gemini-code-assist
Copy link

Summary of Changes

Hello @sinhasubham, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical thread-safety regression in the MetricsInterceptor component, which was causing data integrity issues and retry-related errors in concurrent operations. By ensuring that the correct metrics tracer is consistently used for each gRPC call, the changes prevent erroneous metric reporting and eliminate the root cause of 409 errors during parallel batch processing.

Highlights

  • Thread-Safety Fix: Resolved a thread-safety race condition in MetricsInterceptor where the global current_metrics_tracer could be overwritten by concurrent requests, leading to AttributeError crashes and subsequent 409 errors during retries.
  • Tracer Capture: The intercept method now captures the current_metrics_tracer into a local variable before the gRPC call, ensuring the correct tracer instance is used throughout the call's lifecycle, regardless of other concurrent threads.
  • Regression Testing: A new unit test, test_intercept_thread_safety, has been added to deterministically reproduce the race condition and verify the effectiveness of the thread-safety fix.
Changelog
  • google/cloud/spanner_v1/metrics/metrics_interceptor.py
    • Introduced a local variable tracer to capture SpannerMetricsTracerFactory.current_metrics_tracer at the beginning of the intercept method.
    • Updated all subsequent references within the intercept method to use the local tracer variable instead of the global SpannerMetricsTracerFactory.current_metrics_tracer.
  • tests/unit/test_metrics_interceptor.py
    • Added threading and time imports to support concurrency testing.
    • Implemented test_intercept_thread_safety to specifically test and confirm the fix for the MetricsInterceptor race condition.
Activity
  • No specific review comments or discussions have been recorded yet for this pull request.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copy link

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request effectively resolves the thread-safety race condition in MetricsInterceptor by capturing the current_metrics_tracer in a local variable within the intercept method. This is a solid fix for the described bug. The addition of the test_intercept_thread_safety is excellent, as it deterministically reproduces the race condition and verifies the fix, preventing future regressions. The existing comment regarding a potential copy-paste error in the test code has been retained as it does not conflict with any provided rules.

@sinhasubham sinhasubham force-pushed the fix/already_exists_409 branch from 1d5bdcb to fd16d8a Compare February 4, 2026 07:18
@sinhasubham sinhasubham force-pushed the fix/already_exists_409 branch from fd16d8a to eb2635d Compare February 4, 2026 09:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: spanner Issues related to the googleapis/python-spanner API. size: m Pull request size is medium.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants