Version the worker-bound TaskInstance fields in the execution API schema#68390
Merged
Conversation
The supervisor's task routing (added in apache#65958) reads queue, pool_slots and priority_weight from the TaskInstance it receives, but those fields lived in a TaskInstanceDTO duplicated between airflow-core and task-sdk and kept in sync by an AST-comparison prek hook, outside the versioned execution API schema the Task SDK datamodels are generated from. Add the three fields to the execution API TaskInstance schema (with a Cadwyn version change) and regenerate the SDK datamodels, so the generated TaskInstance carries everything the supervisor needs. Re-point StartupDetails and the coordinator interfaces at the generated model, fold the core TaskInstanceDTO into a subclass of the schema model that only adds the executor-side fields, and drop the duplicated task-sdk DTO and the sync hook. The unused parent_context_carrier field (no readers or writers) is removed.
Member
The last two being required on the worker seems like a code smell. If it makes it to the worker it should just executed. That is the scheduler's job, not the workers |
ashb
reviewed
Jun 11, 2026
…he schema The supervisor only routes on queue; pool_slots and priority_weight are executor concerns (queued-workload priority ordering and edge concurrency slot accounting) that the worker never reads. Keep them on the executor DTO, serialized as before, and add only queue to the worker-facing schema.
Member
Author
|
Agreed and fixed in 78a6b82 -- the worker-facing schema now only gains |
ashb
reviewed
Jun 11, 2026
ashb
reviewed
Jun 11, 2026
ashb
reviewed
Jun 11, 2026
jason810496
approved these changes
Jun 11, 2026
jason810496
left a comment
Member
There was a problem hiding this comment.
Really nice catch, thanks. Do we need full tests before merge Btw?
586b49e to
3d48d40
Compare
The generated TaskInstance no longer accepts pool_slots/priority_weight kwargs, its queue is Optional in the generated typing, and the DTO now validates the inherited hostname field that the ECS adoption test's mock did not set.
imrichardwu
pushed a commit
to imrichardwu/airflow
that referenced
this pull request
Jun 16, 2026
…ema (apache#68390) The supervisor's task routing (added in apache#65958) reads queue, pool_slots and priority_weight from the TaskInstance it receives, but those fields lived in a TaskInstanceDTO duplicated between airflow-core and task-sdk and kept in sync by an AST-comparison prek hook, outside the versioned execution API schema the Task SDK datamodels are generated from. Add the three fields to the execution API TaskInstance schema (with a Cadwyn version change) and regenerate the SDK datamodels, so the generated TaskInstance carries everything the supervisor needs. Re-point StartupDetails and the coordinator interfaces at the generated model, fold the core TaskInstanceDTO into a subclass of the schema model that only adds the executor-side fields, and drop the duplicated task-sdk DTO and the sync hook. The unused parent_context_carrier field (no readers or writers) is removed. * Keep pool_slots and priority_weight executor-side; only queue joins the schema The supervisor only routes on queue; pool_slots and priority_weight are executor concerns (queued-workload priority ordering and edge concurrency slot accounting) that the worker never reads. Keep them on the executor DTO, serialized as before, and add only queue to the worker-facing schema. The generated TaskInstance no longer accepts pool_slots/priority_weight kwargs, its queue is Optional in the generated typing, and the DTO now validates the inherited hostname field that the ECS adoption test's mock did not set.
dingo4dev
pushed a commit
to dingo4dev/airflow
that referenced
this pull request
Jun 16, 2026
…ema (apache#68390) The supervisor's task routing (added in apache#65958) reads queue, pool_slots and priority_weight from the TaskInstance it receives, but those fields lived in a TaskInstanceDTO duplicated between airflow-core and task-sdk and kept in sync by an AST-comparison prek hook, outside the versioned execution API schema the Task SDK datamodels are generated from. Add the three fields to the execution API TaskInstance schema (with a Cadwyn version change) and regenerate the SDK datamodels, so the generated TaskInstance carries everything the supervisor needs. Re-point StartupDetails and the coordinator interfaces at the generated model, fold the core TaskInstanceDTO into a subclass of the schema model that only adds the executor-side fields, and drop the duplicated task-sdk DTO and the sync hook. The unused parent_context_carrier field (no readers or writers) is removed. * Keep pool_slots and priority_weight executor-side; only queue joins the schema The supervisor only routes on queue; pool_slots and priority_weight are executor concerns (queued-workload priority ordering and edge concurrency slot accounting) that the worker never reads. Keep them on the executor DTO, serialized as before, and add only queue to the worker-facing schema. The generated TaskInstance no longer accepts pool_slots/priority_weight kwargs, its queue is Optional in the generated typing, and the DTO now validates the inherited hostname field that the ECS adoption test's mock did not set.
This was referenced Jun 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Since #65958 the supervisor routes every task through
get_coordinator_manager().for_queue(ti.queue), so it depends onqueuebeing on the task instance it receives. That field (along withpool_slotsandpriority_weight) lived in aBaseTaskInstanceDTOduplicated between airflow-core and task-sdk (#67174), kept aligned by an AST-comparison prek hook, while theTaskInstanceschema in the versioned execution API spec lacked it.That spec is already the designated home for this vocabulary:
get_extra_schemas()injectsTaskInstance(andBundleInfo, imported fromairflow.executors.workloads) into the OpenAPI schema precisely so client SDKs generate these types from a versioned source. Adding the missing field there gives one source of truth instead of two hand-synced copies.What
queueto the execution APITaskInstanceschema, with anAddTaskInstanceQueueFieldCadwyn version change in the 2026-06-30 bundle, and regenerate the task-sdk datamodels.StartupDetails.ti,supervise_task()and the coordinator interfaces at the generatedTaskInstance.TaskInstanceDTOinto a subclass of the schema model that adds only the executor-side fields:pool_slotsandpriority_weight(read by the executor's priority sort and the edge executor's concurrency accounting, never by the worker), plusexternal_executor_idandexecutor_config(excluded from serialization).execution_time.workloadsmodule and thecheck-task-instance-dto-synchook. A round-trip test (executor DTO JSON validated by the SDK's generated model) covers the contract the hook used to enforce.parent_context_carrierfrom the DTO: it has no Python readers or writers anywhere in core, task-sdk or providers.Design notes
queueis worker-facing: the supervisor routes on it.pool_slotsandpriority_weightare executor concerns and stay on the executor DTO, still serialized in the workload (required, exactly the pre-PR wire shape) so older workers that deserialize the workload with the required-field model keep working.queuegets a default ("default") instead of being required. The executor always sends the real value (ExecuteTask.makevalidates from the ORM task instance, where the column is non-null); the default keeps hand-built instances such as tests anddry_runruns valid, and matches howmap_indexandhostnameare already modelled in this schema.VersionChangeentry is possible.plugins/www/openapi-genandgo-sdk/pkg/edgeapiare not regenerated here: their codegen is not wired into prek and already lags the spec on main (the Go client was last regenerated in Simple sending of return value from Go tasks to XCom #56481), and the affected fields are optional in both.