feat(jobs): first-class /v1/files + /v1/batches + /v1/fine_tuning surface (openai-family + azure) with batch cost attribution - #724
Conversation
…face Adds the OpenAI-compatible Files / Batches / Fine-tuning routes as typed handlers (previously only reachable via the opaque /passthrough/:provider/* tunnel), for openai-family and Azure OpenAI providers. Routing follows the LiteLLM file-id encoding baseline: an upload names its routing Model once (multipart field / query / x-aisix-model header); the returned id embeds the Model (aisix-<base64url>) so every later call that references it routes automatically. Explicit model/query/header hints and raw provider ids keep working. Batch cost attribution (LiteLLM _handle_completed_batch parity): the first retrieve that observes status=completed downloads the output JSONL, aggregates per-line usage grouped by the provider-billed model, and emits UsageEvents with a deterministic request_id (batch-<id>) for cp-side idempotency. Management calls emit zero-token events like /passthrough (#699). The full auth envelope applies: caller-key auth + model ACL, IP allowlist, budget + rate-limit reservation, input/output guardrail blob scan (#911 [6] passthrough precedent), model E2E timeout, and cooldown accounting on transport failures. Fixes #720
|
Warning Review limit reachedYou’ve reached a temporary PR review limit under our Fair Usage Limits Policy. Next review available in: 16 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (6)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
1 participant
Summary
First-class
/v1/files+/v1/batches+/v1/fine_tuning/jobson the data plane (matrix row "Batch/Files/Fine-tuning 透传" in AISIX-Cloud#873 §⑤). Until now these provider APIs were only reachable through the raw/passthrough/:provider/*tunnel — single-provider, opaque, zero token/cost attribution.Fixes #720
Routes
POST/GET /v1/files,GET/DELETE /v1/files/{id},GET /v1/files/{id}/contentPOST/GET /v1/batches,GET /v1/batches/{id},POST /v1/batches/{id}/cancelPOST/GET /v1/fine_tuning/jobs,GET /v1/fine_tuning/jobs/{id},POST …/cancelProvider coverage v1: adapter
openai(any OpenAI-compatibleapi_base) +azure-openai(resource-scoped/openai/*+api-keyheader). Vertex (GCS-staged) / Bedrock (S3-staged) / Anthropic native/v1/messages/batchesare different wire+storage flows — deliberately out of scope here, rejected with a clear 400.Routing: gateway-encoded resource ids (LiteLLM baseline)
The core problem: a batch create references a file id — no
modelin the body. Following LiteLLM's file-id encoding (litellm/proxy/openai_files_endpoints/common_utils.py), an upload names its routing Model once (multipartmodelfield,?model=, orx-aisix-modelheader) and the returned id is re-encoded asaisix-<base64url("<raw>;model,<name>")>. Every later reference (batch createinput_file_id, retrieve/cancel ids, fine-tuningtraining_file) decodes and routes automatically. Precedence mirrors LiteLLM's scenario order: id-embedded → explicit body/query/header → first accessible OpenAI-compatible model. Raw provider ids keep working (fall back to explicit/default routing). Decoded ids are charset-guarded before touching the upstream URL path.One deliberate extension over LiteLLM: fine-tuning job ids are also encoded (LiteLLM routes FT retrieve/cancel via config-side
custom_llm_providerinstead — we have no equivalent config object, and the id mechanism is already client-visible on files/batches there).Batch usage/cost attribution (LiteLLM
_handle_completed_batchparity)On the first retrieve observing
status=completed, the DP downloads the output JSONL, aggregates per-lineusagegrouped by the provider-billedmodel, and emits UsageEvent(s) with real token counts (inbound_protocol="batch",cached_prompt_tokensincluded). Dedup: process-local guard + deterministicrequest_id = "batch-<id>"so re-emission after a DP restart stays idempotent CP-side. Failed downloads release the guard and retry on the next retrieve. Management calls emit zero-token events (same contract as/passthrough, #699).Governance envelope
Caller-key auth + model ACL, per-model IP allowlist (#697), budget + rate-limit reservation after the input guardrail (passthrough parity), input/output guardrail whole-body scan (#911 [6]), model E2E timeout, cooldown accounting on transport failures (#701), bounded metric labels (#451).
Tests
batch-files-finetuning-e2e.test.ts, real binary + etcd + mock provider, driven through the official OpenAI Node SDK): upload→batch→retrieve→content→cancel lifecycle, attribution download observed on the mock, FT lifecycle, raw-multipart routing variant, 401 baseline.No new CP config surface (reuses existing Model/ProviderKey resources). Docs PR to api7/docs follows separately.