Clarify logging_config_class contract and document REMOTE_TASK_LOG

[jason810496]

• This PR separate the core changes from Clarify logging_config_class contract and document REMOTE_TASK_LOG #67104 .

• related: Decouple remote logging configuration from core airflow #65281
• closes: DEFAULT_LOGGING_CONFIG After the migration to structlog #61175
• closes: Airflow Logging Configs are ignored / not used in Airflow 3.x #53442
• closes: Custom logging handler not sending logs to ClickHouse when set via LOGGING_CONFIG_CLASS #56340

Why

[logging] logging_config_class is documented as a "Logging class" but actually resolves to a logging.config.dictConfig dict, and the REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID side channel that powers remote log read-back was undocumented.

So custom configs silently lost UI log read-back, and ElasticsearchTaskHandler / OpensearchTaskHandler papered over it by self-registering from inside __init__. The provider-side deprecation of that self-registration is split into follow-up PRs (one for elasticsearch, one for opensearch); this PR is the core/SDK/shared-library piece they depend on.

How

• Document the real contract for logging_config_class (dict, not class) and the REMOTE_TASK_LOG / DEFAULT_REMOTE_CONN_ID module-level attributes in the config option help, advanced-logging-configuration.rst, and the discover_remote_log_handler docstring.
• Add a startup WARNING for the case that "user module is missing REMOTE_TASK_LOG while remote_logging is on". The warning is emitted from configure_logging after dictConfig runs, so the check sees the final state — important because the deprecated ES/OS self-registration path populates _ActiveLoggingConfig.remote_task_log from inside the handler __init__.
• Drop the remote_logging_enabled parameter from discover_remote_log_handler (no longer needed now that the warning lives in configure_logging).
• Add unit tests for _warn_if_missing_remote_task_log.

---

Was generative AI tooling used to co-author this PR?

• Yes, with help of Claude Code Opus 4.7 following the guidelines

[amoghrajesh]

@jason810496 I wanna take a look at this one but do not have b/w atm, will do soon

[eladkal]

I think we will need to bump

airflow/scripts/ci/prek/update_airflow_pyproject_toml.py

Lines 91 to 92 in 6082d6a

	"elasticsearch": parse_version("6.5.0"),
	"opensearch": parse_version("1.9.0"),

as part of this PR but we will need to wait for providers to be released

[eladkal]

@jason810496 can you fix the problem above?

[jason810496]

Thanks for the review Elad, Phani. I addressed both comments in last rebase.

[jason810496]

Hi @eladkal,
I some help to resolve the static check CI failure when you have a moment. Thanks.

If I don't update the root pyproject.toml I will encounter:

Update Airflow's meta-package pyproject.toml............................................................Failed
  - hook id: update-pyproject-toml
  - files were modified by this hook

All changes made by hooks:
diff --git a/pyproject.toml b/pyproject.toml
index bae4c06..5a6f250 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -219,7 +219,7 @@ apache-airflow = "airflow.__main__:main"
     "apache-airflow-providers-edge3>=1.0.0"
 ]
 "elasticsearch" = [
-    "apache-airflow-providers-elasticsearch>=6.5.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+    "apache-airflow-providers-elasticsearch>=6.6.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
 ]
 "exasol" = [
     "apache-airflow-providers-exasol>=4.6.1"
@@ -306,7 +306,7 @@ apache-airflow = "airflow.__main__:main"
     "apache-airflow-providers-openlineage>=2.3.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
 ]
 "opensearch" = [
-    "apache-airflow-providers-opensearch>=1.9.0" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+    "apache-airflow-providers-opensearch>=1.9.3" # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
 ]
 "opsgenie" = [
     "apache-airflow-providers-opsgenie>=5.8.0"
@@ -447,7 +447,7 @@ apache-airflow = "airflow.__main__:main"
     "apache-airflow-providers-discord>=3.9.0",
     "apache-airflow-providers-docker>=3.14.1",
     "apache-airflow-providers-edge3>=1.0.0",
-    "apache-airflow-providers-elasticsearch>=6.5.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+    "apache-airflow-providers-elasticsearch>=6.6.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
     "apache-airflow-providers-exasol>=4.6.1",
     "apache-airflow-providers-fab>=3.6.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
     "apache-airflow-providers-facebook>=3.7.0",
@@ -476,7 +476,7 @@ apache-airflow = "airflow.__main__:main"
     "apache-airflow-providers-openai>=1.5.0",
     "apache-airflow-providers-openfaas>=3.7.0",
     "apache-airflow-providers-openlineage>=2.3.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
-    "apache-airflow-providers-opensearch>=1.9.0", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
+    "apache-airflow-providers-opensearch>=1.9.3", # Set from MIN_VERSION_OVERRIDE in update_airflow_pyproject_toml.py
     "apache-airflow-providers-opsgenie>=5.8.0",
     "apache-airflow-providers-oracle>=3.12.0",
     "apache-airflow-providers-pagerduty>=3.8.1",

Fail run: https://github.com/apache/airflow/actions/runs/27183757151/job/80249517727

However, if I update the pyproject.toml. I will encounter the following error from breeze ci selective-check for the CI Image check:

Provider dependency version bumps detected that should only be performed by Release Managers!

  - pyproject.toml : apache-airflow-providers-elasticsearch >= version changed from 6.5.0 to 6.6.0
  - pyproject.toml : apache-airflow-providers-opensearch >= version changed from 1.9.0 to 1.9.3
  - pyproject.toml : apache-airflow-providers-elasticsearch >= version changed from 6.5.0 to 6.6.0
  - pyproject.toml : apache-airflow-providers-opensearch >= version changed from 1.9.0 to 1.9.3

Fail run: https://github.com/apache/airflow/actions/runs/27185728163/job/80254150717

[eladkal]

I think you just need to add allow provider dependency bump label and rebase the PR. That will make breeze ci selective-check pass.
I assume release manager @vatsrahul1001 is OK with this

[eladkal]

@jason810496 CI seems to be green. I guess the PR is ready for review?

[jason810496]

@jason810496 CI seems to be green. I guess the PR is ready for review?

Yes, it's ready for review now. Thanks for the reminder.

[ashb]

I'm fairly sure that the logging.config.dictConfig from the file is not 99% ignored?