feat(embedder): support native OpenAI embedding dimensions instead of truncating (#1087)

[Anai-Guo]

What

Adds an opt-in dimensions field to OpenAIEmbedderConfig so OpenAIEmbedder can request natively reduced embeddings via the OpenAI dimensions parameter, instead of naively slicing the full vector.

Related to #1087.

Why

Today create/create_batch return result.data[0].embedding[: self.config.embedding_dim]. Slicing a text-embedding-3-* vector drops the trailing components without renormalizing, so the resulting vector is no longer unit-length and cosine similarity degrades — the exact retrieval-quality issue raised in #1087 (most severe for text-embedding-3-large, 3072 → 1024 = 66% dropped).

The OpenAI models support generating the embedding at a target dimensionality server-side, which returns a properly normalized reduced vector. As @danielchalef noted on the issue, truncation is intentional for storage cost, so this is added as an opt-in rather than a behavior change.

Behavior

• dimensions=None (default): unchanged — full vector requested and truncated to embedding_dim. Safe for models/endpoints that don't accept the parameter (e.g. text-embedding-ada-002, custom base_url deployments).
• dimensions=N: N is forwarded to the API and the natively sized, renormalized vector is returned without truncation. Users setting this should match embedding_dim for downstream storage consistency (documented on the field).

Tests

Extends tests/embedder/test_openai.py:

• existing tests assert dimensions is not sent when unconfigured;
• new tests assert it is forwarded and that the full native vector is returned (no truncation) for both create and create_batch.

All four pass; the two new tests fail against the pre-change implementation, confirming they exercise the new path. ruff check and ruff format --check are clean.

🤖 Generated with Claude Code

[Anai-Guo]

Picking this up per @danielchalef's note on #1087 (the original reporter offered a PR ~8 months ago but none was filed). Kept it strictly opt-in so the default truncation behavior — and support for endpoints that don't accept dimensions — is unchanged. Happy to adjust the API shape (e.g. tie it to embedding_dim instead of a separate field) if you'd prefer.