AIP-51 - Local Executor Compatibility

[o-nikolas]

Overview

Executors that run locally have certain limitations and requirements, many of which are currently hardcoded in core Airflow code. To add a new Executor, that would run locally, one would be required to change this core Airflow code.

Examples

• 1a) Whether or not to start the serve_logs sub-process, currently hardcoded to LocalExecutor and SequentialExecutor:

  airflow/airflow/cli/commands/scheduler_command.py

  Line 93 in 27e2101

  if conf.get("core", "executor") in ["LocalExecutor", "SequentialExecutor"]:

• 1b) When running in standalone mode, asserts a local Executor is being used, currently hardcoded to LocalExecutor and SequentialExecutor:

  airflow/airflow/cli/commands/standalone_command.py

  Lines 156 to 158 in 26f94c5

  	if conf.get("core", "executor") not in [
  	executor_constants.LOCAL_EXECUTOR,
  	executor_constants.SEQUENTIAL_EXECUTOR,

• 1c) Local Executors need a copy of Airflow configuration created in tmp, currently hardcoded to LocalExecutor and SequentialExecutor:

  airflow/airflow/jobs/backfill_job.py

  Lines 505 to 509 in 26f94c5

  	if self.executor_class in (
  	executor_constants.LOCAL_EXECUTOR,
  	executor_constants.SEQUENTIAL_EXECUTOR,
  	):
  	cfg_path = tmp_configuration_copy()

Proposal

A static method or attribute on the Executor class which can be checked by core code.

There is a precedent already set with the supports_ad_hoc_ti_run attribute, see:

airflow/airflow/executors/kubernetes_executor.py

Line 435 in fb741fd

supports_ad_hoc_ti_run: bool = True

airflow/airflow/www/views.py

Lines 1735 to 1737 in 26f94c5

	if not getattr(executor, "supports_ad_hoc_ti_run", False):
	msg = "Only works with the Celery, CeleryKubernetes or Kubernetes executors"
	return redirect_or_json(origin, msg, "error", 400)

[pierrejeambrun]

Would like to help on this one, assigning myself :)

[snjypl]

@pierrejeambrun , i faced an issue with the LocalKubernetesExecutor #28638
the scheduler command is not starting the serve_logs process for LocalKubernetesExecutor.

i think, is_local should be set to True for LocalKubernetesExecutor.

if you get a chance, can you please review the PR.