Skip to content

Remove unused [datalib] extra from OpenAI provider base install#69408

Merged
kaxil merged 1 commit into
apache:mainfrom
YAshhh29:remove-openai-datalib-extra
Jul 13, 2026
Merged

Remove unused [datalib] extra from OpenAI provider base install#69408
kaxil merged 1 commit into
apache:mainfrom
YAshhh29:remove-openai-datalib-extra

Conversation

@YAshhh29

@YAshhh29 YAshhh29 commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

The apache-airflow-providers-openai package pulls in the [datalib]
extra of the openai SDK — which transitively installs numpy, pandas,
and pandas-stubs — for every user of this provider. That extra has no
functional use inside this provider today, so it is pure bloat and CVE
noise for anyone installing the provider.

Investigation

I audited every code path that could depend on numpy/pandas reaching
users through this provider:

Check Location Result
Provider source providers/openai/src/**/*.py 0 references to numpy/pandas/DataFrame
Provider unit tests providers/openai/tests/unit/** 0 references
Provider example DAGs / system tests providers/openai/tests/system/openai/example_*.py 0 references — only pendulum + Airflow imports
Provider docs providers/openai/docs/** Only auto-generated dep tables
Operator return types operators/openai.py list[float], str, str | None — no DataFrames
Cross-provider consumers (RAG examples) providers/{pgvector,pinecone,weaviate}/tests/system/**/example_*_openai.py pgvector & pinecone: 0 references. weaviate has its own pandas dep declared in providers/weaviate/pyproject.toml
Historical usage git log providers/openai/pyproject.toml [datalib] was carried forward across every major version bump (openai 0.x → 1.x → 2.x); recent PR #69068 just bumped openai[datalib]>=1.66.0openai[datalib]>=2.37.0 without justifying the extra
Upstream SDK docs openai/openai-python README (SDK 2.x) [datalib] is not mentioned anywhere in the modern SDK docs ([aiohttp], [bedrock], [realtime] are — [datalib] is not)

Why it was there in the first place

[datalib] was the extra used by OpenAI SDK 0.x's legacy CLI
(openai tools fine_tunes.prepare_data), which processed JSONL
fine-tuning files via pandas. That CLI was removed when OpenAI SDK 1.x
shipped in 2023, but the extra remained in the package for backward
compatibility. This Airflow provider carried the extra forward across
every subsequent version bump without re-evaluating whether it was
still needed — a classic transitive fossil dependency.

What changes

  • providers/openai/pyproject.toml: base dep is now openai>=2.37.0 (no [datalib]); inline comment records the rationale so it doesn't get re-added by the next version bump.
  • providers/openai/README.rst and providers/openai/docs/index.rst: dep table updated to reflect the base dep.
  • providers/openai/docs/changelog.rst: behavior note under the Changelog header explaining the change and pointing users at pip install 'openai[datalib]' if they had unknowingly relied on the transitive numpy/pandas install for their DAG code.

Impact for users

  • Everyone who installs the OpenAI provider stops pulling in numpy, pandas, and pandas-stubs transitively — smaller Docker images, smaller pip-audit surfaces.
  • Users of OpenAIHook, OpenAIEmbeddingOperator, OpenAIResponseOperator, and the batch operators are unaffected — none of their code paths use pandas/numpy.
  • The one edge case is a user whose DAG code (not the provider) does import pandas and was relying on this provider to bring pandas along transitively. The changelog note points those users at the exact one-line fix.

@potiuk
potiuk force-pushed the remove-openai-datalib-extra branch from 7e5cde2 to 197b432 Compare July 12, 2026 21:44
@kaxil
kaxil merged commit ef1a92b into apache:main Jul 13, 2026
72 checks passed
@boring-cyborg

boring-cyborg Bot commented Jul 13, 2026

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

joshuabvarghese pushed a commit to joshuabvarghese/airflow that referenced this pull request Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants