Add SnowparkContainerJobOperator

[justinpakzad]

---

Description

Snowpark Container Services lets users run containerized workloads directly within their Snowflake account. Things like ML training jobs, data processing pipelines, or any custom application packaged as a Docker image. Jobs are submitted via EXECUTE JOB SERVICE, which runs containers on a compute pool and terminates once the code exits. Currently there's no operator for orchestrating these jobs. Users would have to fall back to raw SQL via SQLExecuteQueryOperator and manually handle polling, log retrieval, error handling, and service teardown.

This PR adds SnowparkContainerJobOperator which:

• Submits the job asynchronously via EXECUTE JOB SERVICE
• Polls DESCRIBE SERVICE until the job reaches a terminal state (DONE, FAILED, etc.)
• Fetches container logs via SYSTEM$GET_SERVICE_LOGS for all replicas and surfaces them in the Airflow task logs
• Supports both stage-based specs (spec + spec_stage) and inline YAML (spec_text)
• Supports fire-and-forget mode (wait_for_completion=False) for long-running jobs as well as other optional parameters (drop_on_completion, replicas, query_warehouse, etc.)

Testing:

• Unit tests covering parameter validation, SQL generation, polling logic, log retrieval, error handling for unexpected statuses, on_kill cleanup, and execution paths.
• Manually tested end-to-end against a real Snowflake account with an active compute pool. Verified the success path (job completes, logs retrieved) and failure scenarios (job fails, error logs surfaced, operator raises appropriately). Also tested with various parameters such as replicas, spec_text vs spec/spec_stage, and wait_for_completion variations.

Added an example DAG and operator documentation. Running the example DAG requires some infrastructure to be provisioned beforehand (compute pool, image repository, pushed container image, etc.) those steps are covered in the operator's how-to guide. If approved, I plan on opening a follow-up PR to implement deferrable mode.

Was generative AI tooling used to co-author this PR?

• Yes (Claude Opus 4.6)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

---

Note

✅ Ready for review · @justinpakzad → @potiuk · 2026-06-22 06:34 UTC

Thanks @justinpakzad — the earlier triage items look addressed and this PR is now ready for review. The ball is back with the maintainers; a maintainer will take the next look.

_{Automated triage — may be imperfect.}

[justinpakzad]

Could I get a re-trigger of the failed check please? Seems there was an error building the image. Thanks.

[justinpakzad]

Here are a couple of screenshots that show a successful run and a failed run against my Snowflake instance. Happy to provide more if needed.

[SameerMesiah97]

Good PR. I have left some comments.

[justinpakzad]

Good PR. I have left some comments.

I appreciate the detailed review. I addressed all of your feedback. I left one comment unresolved so you can see my response to your first question but your second point was addressed.

[justinpakzad]

Would appreciate a review on this. Thanks.

[justinpakzad]

@potiuk would you mind taking a look at this when you have a moment. Thanks.