Fix pytest collection failure for classes decorated with context managers - #55915
Merged
Conversation
kaxil
requested review from
ashb,
gopidesupavan,
hussein-awala and
potiuk
as code owners
September 20, 2025 01:54
jedcunningham
approved these changes
Sep 20, 2025
jedcunningham
added a commit
to astronomer/airflow
that referenced
this pull request
Sep 22, 2025
It'll actually run when apache#55915 is merged, but it will pass when it does start running.
…gers
Classes decorated with `@conf_vars` and other context managers were disappearing
during pytest collection, causing tests to be silently skipped. This affected
several test classes including `TestWorkerStart` in the Celery provider tests.
Root cause: `ContextDecorator` transforms decorated classes into callable wrappers.
Since pytest only collects actual type objects as test classes, these wrapped
classes are ignored during collection.
Simple reproduction (no Airflow needed):
```py
import contextlib
import inspect
@contextlib.contextmanager
def simple_cm():
yield
@simple_cm()
class TestExample:
def test_method(self):
pass
print(f'Is class? {inspect.isclass(TestExample)}') # False - pytest won't collect
```
and then run
```shell
pytest test_example.py --collect-only
```
Airflow reproduction:
```shell
breeze run pytest providers/celery/tests/unit/celery/cli/test_celery_command.py --collect-only -v
breeze run pytest providers/celery/tests/unit/celery/cli/test_celery_command.py --collect-only -v
```
Solution:
1. Fixed affected test files by replacing class-level `@conf_vars` decorators
with pytest fixtures
2. Created pytest fixtures to apply configuration changes
3. Used `@pytest.mark.usefixtures` to apply configuration to test classes
4. Added custom linter to prevent future occurrences and integrated it
into pre-commit hooks
Files changed:
- Fixed 3 test files with problematic class decorators
- Added custom linter with pre-commit integration
This ensures pytest properly collects all test classes and prevents similar
issues in the future through automated detection.
- Fix linter to consistently check only test files when processing individual files - Add mock_supervisor_comms fixture to Kafka integration tests to resolve SUPERVISOR_COMMS import errors The Kafka tests were failing because they execute operators directly via operator.execute() which requires the task execution environment to be properly mocked.
kaxil
force-pushed
the
fix-celery-test
branch
from
September 23, 2025 03:11
5144a97 to
227eb45
Compare
kaxil
pushed a commit
to astronomer/airflow
that referenced
this pull request
Sep 23, 2025
It'll actually run when apache#55915 is merged, but it will pass when it does start running.
kaxil
pushed a commit
to astronomer/airflow
that referenced
this pull request
Sep 23, 2025
It'll actually run when apache#55915 is merged, but it will pass when it does start running.
kaxil
added a commit
that referenced
this pull request
Sep 23, 2025
…gers (#55915) Classes decorated with `@conf_vars` and other context managers were disappearing during pytest collection, causing tests to be silently skipped. This affected several test classes including `TestWorkerStart` in the Celery provider tests. Root cause: `ContextDecorator` transforms decorated classes into callable wrappers. Since pytest only collects actual type objects as test classes, these wrapped classes are ignored during collection. Simple reproduction (no Airflow needed): ```py import contextlib import inspect @contextlib.contextmanager def simple_cm(): yield @simple_cm() class TestExample: def test_method(self): pass print(f'Is class? {inspect.isclass(TestExample)}') # False - pytest won't collect ``` and then run ```shell pytest test_example.py --collect-only ``` Airflow reproduction: ```shell breeze run pytest providers/celery/tests/unit/celery/cli/test_celery_command.py --collect-only -v breeze run pytest providers/celery/tests/unit/celery/cli/test_celery_command.py --collect-only -v ``` Solution: 1. Fixed affected test files by replacing class-level `@conf_vars` decorators with pytest fixtures 2. Created pytest fixtures to apply configuration changes 3. Used `@pytest.mark.usefixtures` to apply configuration to test classes 4. Added custom linter to prevent future occurrences and integrated it into pre-commit hooks Files changed: - Fixed 3 test files with problematic class decorators - Added custom linter with pre-commit integration This ensures pytest properly collects all test classes and prevents similar issues in the future through automated detection. (cherry picked from commit e95f24f)
kaxil
pushed a commit
that referenced
this pull request
Sep 23, 2025
* Only send hostname to celery worker if passed in cli We were setting the hostname in the celery worker, even if the user did not specify one. This meant we were passing on None. Historically, that has still resulted in the default value being used instead of None, but that has changed in click 8.3.0 (which celery uses in its cli). Either way, no need for us to pass it! * Fix test It'll actually run when #55915 is merged, but it will pass when it does start running.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Classes decorated with
@conf_varsand other context managers were disappearing during pytest collection, causing tests to be silently skipped. This affected several test classes includingTestWorkerStartin the Celery provider tests.Root cause:
ContextDecoratortransforms decorated classes into callable wrappers. Since pytest only collects actual type objects as test classes, these wrapped classes are ignored during collection.Simple reproduction (no Airflow needed):
and then run
Airflow reproduction:
Before
After:
Solution:
@conf_varsdecorators with pytest fixtures@pytest.mark.usefixturesto apply configuration to test classesFiles changed:
This ensures pytest properly collects all test classes and prevents similar issues in the future through automated detection.