Skip to content

Fix pytest collection failure for classes decorated with context managers - #55915

Merged
kaxil merged 2 commits into
apache:mainfrom
astronomer:fix-celery-test
Sep 23, 2025
Merged

Fix pytest collection failure for classes decorated with context managers#55915
kaxil merged 2 commits into
apache:mainfrom
astronomer:fix-celery-test

Conversation

@kaxil

@kaxil kaxil commented Sep 20, 2025

Copy link
Copy Markdown
Member

Classes decorated with @conf_vars and other context managers were disappearing during pytest collection, causing tests to be silently skipped. This affected several test classes including TestWorkerStart in the Celery provider tests.

Root cause: ContextDecorator transforms decorated classes into callable wrappers. Since pytest only collects actual type objects as test classes, these wrapped classes are ignored during collection.

Simple reproduction (no Airflow needed):

import contextlib
import inspect

@contextlib.contextmanager
def simple_cm():
    yield

@simple_cm()
class TestExample:
    def test_method(self):
        pass

print(f'Is class? {inspect.isclass(TestExample)}')  # False - pytest won't collect

and then run

pytest test_example.py --collect-only

Airflow reproduction:

Before

> breeze run pytest providers/celery/tests/unit/celery/cli/test_celery_command.py --collect-only -v

collected 6 items

<Dir airflow>
  <Dir providers>
    <Dir celery>
      <Dir tests>
        <Package unit>
          <Package celery>
            <Package cli>
              <Module test_celery_command.py>
                <Class TestRemoteCeleryControlCommands>
                  <Function test_list_celery_workers>
                  <Function test_shutdown_worker>
                  <Function test_shutdown_all_workers>
                  <Function test_add_queue>
                  <Function test_remove_queue>
                <Function test_stale_bundle_cleanup>

After:

> breeze run pytest providers/celery/tests/unit/celery/cli/test_celery_command.py --collect-only -v

collected 14 items

<Dir airflow>
  <Dir providers>
    <Dir celery>
      <Dir tests>
        <Package unit>
          <Package celery>
            <Package cli>
              <Module test_celery_command.py>
                <Class TestCeleryStopCommand>
                  <Function test_if_right_pid_is_read>
                  <Function test_same_pid_file_is_used_in_start_and_stop>
                  <Function test_custom_pid_file_is_used_in_start_and_stop>
                <Class TestWorkerStart>
                  <Function test_worker_started_with_required_arguments>
                <Class TestWorkerFailure>
                  <Function test_worker_failure_gracefull_shutdown>
                <Class TestFlowerCommand>
                  <Function test_run_command>
                  <Function test_run_command_daemon_v_3_below>
                  <Function test_run_command_daemon_v3_above>
                <Class TestRemoteCeleryControlCommands>
                  <Function test_list_celery_workers>
                  <Function test_shutdown_worker>
                  <Function test_shutdown_all_workers>
                  <Function test_add_queue>
                  <Function test_remove_queue>
                <Function test_stale_bundle_cleanup>

Solution:

  1. Fixed affected test files by replacing class-level @conf_vars decorators with pytest fixtures
  2. Created pytest fixtures to apply configuration changes
  3. Used @pytest.mark.usefixtures to apply configuration to test classes
  4. Added custom linter to prevent future occurrences and integrated it into pre-commit hooks

Files changed:

  • Fixed 3 test files with problematic class decorators
  • Added custom linter with pre-commit integration

This ensures pytest properly collects all test classes and prevents similar issues in the future through automated detection.

Comment thread scripts/ci/prek/check_contextmanager_class_decorators.py
Comment thread scripts/ci/prek/check_contextmanager_class_decorators.py Outdated
Comment thread scripts/ci/prek/check_contextmanager_class_decorators.py
Comment thread scripts/ci/prek/check_contextmanager_class_decorators.py
jedcunningham added a commit to astronomer/airflow that referenced this pull request Sep 22, 2025
It'll actually run when apache#55915 is merged, but it will pass when it does
start running.
…gers

Classes decorated with `@conf_vars` and other context managers were disappearing
during pytest collection, causing tests to be silently skipped. This affected
several test classes including `TestWorkerStart` in the Celery provider tests.

Root cause: `ContextDecorator` transforms decorated classes into callable wrappers.
Since pytest only collects actual type objects as test classes, these wrapped
classes are ignored during collection.

Simple reproduction (no Airflow needed):

```py
import contextlib
import inspect

@contextlib.contextmanager
def simple_cm():
    yield

@simple_cm()
class TestExample:
    def test_method(self):
        pass

print(f'Is class? {inspect.isclass(TestExample)}')  # False - pytest won't collect
```

and then run

```shell
pytest test_example.py --collect-only
```

Airflow reproduction:

```shell
breeze run pytest providers/celery/tests/unit/celery/cli/test_celery_command.py --collect-only -v

breeze run pytest providers/celery/tests/unit/celery/cli/test_celery_command.py --collect-only -v
```

Solution:
1. Fixed affected test files by replacing class-level `@conf_vars` decorators
   with pytest fixtures
2. Created pytest fixtures to apply configuration changes
3. Used `@pytest.mark.usefixtures` to apply configuration to test classes
4. Added custom linter to prevent future occurrences and integrated it
   into pre-commit hooks

Files changed:
- Fixed 3 test files with problematic class decorators
- Added custom linter with pre-commit integration

This ensures pytest properly collects all test classes and prevents similar
issues in the future through automated detection.
- Fix linter to consistently check only test files when processing individual files
- Add mock_supervisor_comms fixture to Kafka integration tests to resolve SUPERVISOR_COMMS import errors

The Kafka tests were failing because they execute operators directly via operator.execute()
which requires the task execution environment to be properly mocked.
@kaxil
kaxil merged commit e95f24f into apache:main Sep 23, 2025
74 checks passed
@kaxil
kaxil deleted the fix-celery-test branch September 23, 2025 03:42
kaxil pushed a commit to astronomer/airflow that referenced this pull request Sep 23, 2025
It'll actually run when apache#55915 is merged, but it will pass when it does
start running.
kaxil pushed a commit to astronomer/airflow that referenced this pull request Sep 23, 2025
It'll actually run when apache#55915 is merged, but it will pass when it does
start running.
kaxil added a commit that referenced this pull request Sep 23, 2025
…gers (#55915)

Classes decorated with `@conf_vars` and other context managers were disappearing
during pytest collection, causing tests to be silently skipped. This affected
several test classes including `TestWorkerStart` in the Celery provider tests.

Root cause: `ContextDecorator` transforms decorated classes into callable wrappers.
Since pytest only collects actual type objects as test classes, these wrapped
classes are ignored during collection.

Simple reproduction (no Airflow needed):

```py
import contextlib
import inspect

@contextlib.contextmanager
def simple_cm():
    yield

@simple_cm()
class TestExample:
    def test_method(self):
        pass

print(f'Is class? {inspect.isclass(TestExample)}')  # False - pytest won't collect
```

and then run

```shell
pytest test_example.py --collect-only
```

Airflow reproduction:

```shell
breeze run pytest providers/celery/tests/unit/celery/cli/test_celery_command.py --collect-only -v

breeze run pytest providers/celery/tests/unit/celery/cli/test_celery_command.py --collect-only -v
```

Solution:
1. Fixed affected test files by replacing class-level `@conf_vars` decorators
   with pytest fixtures
2. Created pytest fixtures to apply configuration changes
3. Used `@pytest.mark.usefixtures` to apply configuration to test classes
4. Added custom linter to prevent future occurrences and integrated it
   into pre-commit hooks

Files changed:
- Fixed 3 test files with problematic class decorators
- Added custom linter with pre-commit integration

This ensures pytest properly collects all test classes and prevents similar
issues in the future through automated detection.

(cherry picked from commit e95f24f)
kaxil pushed a commit that referenced this pull request Sep 23, 2025
* Only send hostname to celery worker if passed in cli

We were setting the hostname in the celery worker, even if the user did
not specify one. This meant we were passing on None. Historically, that
has still resulted in the default value being used instead of None, but
that has changed in click 8.3.0 (which celery uses in its cli). Either
way, no need for us to pass it!

* Fix test

It'll actually run when #55915 is merged, but it will pass when it does
start running.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants