Conversation
…o feature/stt-evaluation
…o feature/stt-evaluation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds end-to-end Speech-to-Text (STT) evaluation: API endpoints, models, CRUD, services, Gemini batch provider, batch polling/processing, storage/file handling, DB migration, and comprehensive tests and docs. Changes
Sequence Diagram(s)sequenceDiagram
participant Client
participant API as STT API
participant Service
participant Storage
participant DB as Database
Client->>API: POST /files/audio (multipart)
API->>Service: validate_audio_file(file)
Service->>Service: check extension & size
Service->>Storage: upload file to object store
Storage-->>Service: object_store_url
Service->>DB: create_file record
DB-->>API: file metadata
API-->>Client: AudioUploadResponse (s3_url, file_id)
sequenceDiagram
participant Client
participant API as STT API
participant CRUD
participant BatchSvc as Batch Service
participant Storage
participant Gemini
Client->>API: POST /runs (start evaluation)
API->>CRUD: validate dataset & list samples
CRUD-->>API: samples
API->>CRUD: create run & create result records
API->>BatchSvc: start_stt_evaluation_batch(run, samples)
BatchSvc->>Storage: generate signed URLs for sample files
BatchSvc->>BatchSvc: build JSONL requests
BatchSvc->>Gemini: submit batch job(s)
Gemini-->>BatchSvc: provider batch id/status
BatchSvc->>CRUD: update_stt_run(status=processing, batch ids)
API-->>Client: STTEvaluationRunPublic (processing)
sequenceDiagram
participant Cron
participant CRUD as Run CRUD
participant Gemini
participant Results as Result CRUD
participant DB as Database
Cron->>CRUD: get_pending_stt_runs()
CRUD-->>Cron: pending runs with batch_job_id
loop each run
Cron->>Gemini: get_batch_status(batch_id)
Gemini-->>Cron: state
alt terminal
Cron->>Gemini: download_batch_results(batch_output_id)
Gemini-->>Cron: results JSONL
Cron->>Results: update_stt_result entries
Results->>DB: persist updates
Cron->>CRUD: update_stt_run(status=completed/failed)
else
Cron-->>Cron: continue polling
end
end
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@backend/app/api/routes/stt_evaluations/evaluation.py`:
- Around line 116-127: The except block in start_stt_evaluation currently
returns str(e) to the client, leaking internal error details; instead keep
logging the full error and saving the detailed message via
update_stt_run(session=_session, run_id=run.id, status="failed",
error_message=str(e)) but change the HTTPException detail to a generic message
(optionally include a non-sensitive identifier like run.id or an internal error
code) so the client receives no internal stack/secret data; locate the except
block that references logger.error, update_stt_run, and raise HTTPException and
replace the HTTPException detail payload accordingly.
- Around line 62-80: The run's total_items is being computed from
dataset.dataset_metadata.sample_count which can be stale; instead fetch the
actual samples first (the variable named samples used later) and compute
total_items = len(samples) * len(run_create.providers) before calling
create_stt_run (update the create_stt_run call to pass that computed value).
Ensure you still fall back to dataset.dataset_metadata.get("sample_count", 0)
only if samples is empty or the samples fetch fails, and keep using language_id
= dataset.language_id and other create_stt_run parameters unchanged.
🧹 Nitpick comments (7)
backend/app/models/stt_evaluation.py (5)
74-82:default_factory=dictproduces{}but column isnullable=True— pick one semantic.The Python default is
{}(viadefault_factory=dict), sosample_metadatawill never beNonewhen created through the ORM without an explicitNoneassignment, yet the DB column allowsNULL. This may cause confusion for downstream consumers checkingis Nonevs== {}. Consider aligning: either usedefault=None(matchingnullable=True) or setnullable=False, server_default=text("'{}'").
136-147:providerandstatusonSTTResultare barestr— consider enum validation or constraints.
statushas anSTTResultStatusenum defined but the column is typed asstrwith no DB-level check constraint. Similarly,providercould drift fromSUPPORTED_STT_PROVIDERS. At minimum, adding aCheckConstraintonstatusensures DB-level integrity.
273-277:STTFeedbackUpdateallows an empty payload (both fieldsNone) — this is a no-op.If both
is_correctandcommentareNone, the update request does nothing meaningful. Consider a model-level validator to require at least one field.Example validator
class STTFeedbackUpdate(BaseModel): """Request model for updating human feedback on a result.""" is_correct: bool | None = Field(None, description="Is the transcription correct?") comment: str | None = Field(None, description="Feedback comment") + + `@model_validator`(mode="after") + def check_at_least_one_field(self) -> "STTFeedbackUpdate": + if self.is_correct is None and self.comment is None: + raise ValueError("At least one of 'is_correct' or 'comment' must be provided") + return self
108-112:updated_atusesdefault_factory=nowwhich only fires on INSERT—but CRUD updates explicitly set it, making this a best-practice refactoring suggestion.Both
STTSample.updated_atandSTTResult.updated_atusedefault_factory=now, which only executes on INSERT. However, inspection of the CRUD layer shows that all update operations explicitly setupdated_at = now()(e.g.,backend/app/crud/stt_evaluations/run.py:222,backend/app/crud/stt_evaluations/result.py:238). While this explicit management works, consider adding SQLAlchemy'sonupdateto the column definition for additional safety against future updates that might miss manual assignment.Example using sa_column with onupdate
- updated_at: datetime = SQLField( - default_factory=now, - nullable=False, - sa_column_kwargs={"comment": "Timestamp when the sample was last updated"}, - ) + updated_at: datetime = SQLField( + default_factory=now, + nullable=False, + sa_column_kwargs={ + "comment": "Timestamp when the sample was last updated", + "onupdate": now, + }, + )Also applies to: 216-219
232-265: Consider addingmodel_config = ConfigDict(from_attributes=True)toSTTSamplePublicandSTTResultPublicfor idiomatic Pydantic usage.These models could benefit from Pydantic v2's
from_attributes=Trueconfiguration. While the current explicit keyword argument construction (e.g.,STTSamplePublic(id=sample.id, file_id=sample.file_id, ...)) works correctly, adoptingfrom_attributes=Truewould enable the idiomatic patternmodel_validate(orm_instance)and reduce duplication across construction sites.backend/app/api/routes/stt_evaluations/evaluation.py (2)
137-155: Verbose manual field-by-field model construction — usemodel_validateorfrom_orm.Both
STTEvaluationRunPublicandSTTEvaluationRunWithResultsare constructed by manually mapping every field from the ORM object. This is error-prone (easy to miss a field when the model evolves) and verbose. If you addmodel_config = ConfigDict(from_attributes=True)to the Pydantic models, you can replace this with:STTEvaluationRunPublic.model_validate(run)This would also simplify
STTEvaluationRunWithResultsconstruction.Also applies to: 233-253
190-206: Response model mismatch wheninclude_results=False.The endpoint declares
response_model=APIResponse[STTEvaluationRunWithResults], but wheninclude_results=False,resultswill be an empty list andresults_totalwill be 0. While this technically validates, it's semantically misleading — the response schema always advertises results. Consider using a union type or separate endpoint, or at minimum document this behavior clearly.
| sample_count = (dataset.dataset_metadata or {}).get("sample_count", 0) | ||
|
|
||
| if sample_count == 0: | ||
| raise HTTPException(status_code=400, detail="Dataset has no samples") | ||
|
|
||
| # Use language_id from the dataset | ||
| language_id = dataset.language_id | ||
|
|
||
| # Create run record | ||
| run = create_stt_run( | ||
| session=_session, | ||
| run_name=run_create.run_name, | ||
| dataset_id=run_create.dataset_id, | ||
| dataset_name=dataset.name, | ||
| org_id=auth_context.organization_.id, | ||
| project_id=auth_context.project_.id, | ||
| providers=run_create.providers, | ||
| language_id=language_id, | ||
| total_items=sample_count * len(run_create.providers), |
There was a problem hiding this comment.
total_items derived from metadata may diverge from actual sample count.
sample_count is read from dataset.dataset_metadata (line 62), but the actual samples are fetched separately on line 84. If metadata becomes stale (e.g., samples added/removed without metadata update), total_items stored on the run will be incorrect. Consider computing total_items from len(samples) after fetching them.
Proposed fix
+ # Get samples for the dataset
+ samples = get_samples_by_dataset_id(
+ session=_session,
+ dataset_id=run_create.dataset_id,
+ org_id=auth_context.organization_.id,
+ project_id=auth_context.project_.id,
+ )
+
+ if not samples:
+ raise HTTPException(status_code=400, detail="Dataset has no samples")
+
# Create run record
run = create_stt_run(
session=_session,
run_name=run_create.run_name,
dataset_id=run_create.dataset_id,
dataset_name=dataset.name,
org_id=auth_context.organization_.id,
project_id=auth_context.project_.id,
providers=run_create.providers,
language_id=language_id,
- total_items=sample_count * len(run_create.providers),
+ total_items=len(samples) * len(run_create.providers),
)
-
- # Get samples for the dataset
- samples = get_samples_by_dataset_id(
- session=_session,
- dataset_id=run_create.dataset_id,
- org_id=auth_context.organization_.id,
- project_id=auth_context.project_.id,
- )🤖 Prompt for AI Agents
In `@backend/app/api/routes/stt_evaluations/evaluation.py` around lines 62 - 80,
The run's total_items is being computed from
dataset.dataset_metadata.sample_count which can be stale; instead fetch the
actual samples first (the variable named samples used later) and compute
total_items = len(samples) * len(run_create.providers) before calling
create_stt_run (update the create_stt_run call to pass that computed value).
Ensure you still fall back to dataset.dataset_metadata.get("sample_count", 0)
only if samples is empty or the samples fetch fails, and keep using language_id
= dataset.language_id and other create_stt_run parameters unchanged.
| except Exception as e: | ||
| logger.error( | ||
| f"[start_stt_evaluation] Batch submission failed | " | ||
| f"run_id: {run.id}, error: {str(e)}" | ||
| ) | ||
| update_stt_run( | ||
| session=_session, | ||
| run_id=run.id, | ||
| status="failed", | ||
| error_message=str(e), | ||
| ) | ||
| raise HTTPException(status_code=500, detail=f"Batch submission failed: {e}") |
There was a problem hiding this comment.
Internal error details leaked to the client.
str(e) is included in the HTTP 500 response detail, which may expose internal implementation details (stack traces, service URLs, credentials in connection strings, etc.) to API consumers.
Proposed fix
- raise HTTPException(status_code=500, detail=f"Batch submission failed: {e}")
+ raise HTTPException(status_code=500, detail="Batch submission failed")🤖 Prompt for AI Agents
In `@backend/app/api/routes/stt_evaluations/evaluation.py` around lines 116 - 127,
The except block in start_stt_evaluation currently returns str(e) to the client,
leaking internal error details; instead keep logging the full error and saving
the detailed message via update_stt_run(session=_session, run_id=run.id,
status="failed", error_message=str(e)) but change the HTTPException detail to a
generic message (optionally include a non-sensitive identifier like run.id or an
internal error code) so the client receives no internal stack/secret data;
locate the except block that references logger.error, update_stt_run, and raise
HTTPException and replace the HTTPException detail payload accordingly.
| ) | ||
| return cls(api_key=api_key) | ||
|
|
||
| def validate_connection(self) -> bool: |
There was a problem hiding this comment.
I think once the client instantiation is successful this check is rendered redundant unless there is a wifi outage.
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Fix all issues with AI agents
In `@backend/app/api/docs/stt_evaluation/start_evaluation.md`:
- Around line 1-8: The docs refer to "providers" but the API expects "models";
update the wording in this doc so it matches the STTEvaluationRunCreate contract
by replacing mentions of "providers" with "models" and noting the field name
`models: list[str]` where examples or supported options are listed (e.g., change
"**Supported providers:** gemini-2.5-pro" to "**Supported models:**
gemini-2.5-pro") and ensure any descriptive text referencing "providers" (lines
describing processing through providers) instead mentions processing through the
specified `models`.
In `@backend/app/crud/stt_evaluations/batch.py`:
- Around line 133-198: The code only saves first_batch_job_id
(first_batch_job_id) to the EvaluationRun so only the first model's batch job is
discoverable by poll_all_pending_stt_evaluations (which uses
EvaluationRun.batch_job_id); fix by persisting all created batch job IDs instead
of only the first: collect batch_job.id values from start_batch_job into
batch_jobs (or a list batch_job_ids) and call update_stt_run with that
collection (e.g., update_stt_run(session, run_id=run.id, status="processing",
batch_job_id=None, metadata={"batch_job_ids": batch_job_ids}) or add a new
parameter/field to store batch_job_ids); alternatively, if multi-model runs are
not supported, add an explicit assertion or comment near the loop and avoid
creating multiple jobs — reference functions/vars: start_batch_job, batch_jobs,
first_batch_job_id, update_stt_run, and poll_all_pending_stt_evaluations when
making the change.
In `@backend/app/crud/stt_evaluations/cron.py`:
- Around line 282-284: Remove the unused boolean any_succeeded (set at
any_succeeded = False and later modified) or else use it in the final-status
calculation; specifically, edit the block that defines/updates all_terminal,
any_failed, and any_succeeded (search for any_succeeded and any_failed in
cron.py) and either delete all references and assignments to any_succeeded, or
integrate it into the terminal-state logic (e.g., use any_succeeded to
distinguish "all failed" vs "partially succeeded" when computing the overall
status) and remove any linter warnings. Ensure no other code relies on
any_succeeded before removing.
- Around line 435-452: The loop currently skips when result_record is None which
hides failures; update the branch handling to log a warning (including
identifiers like the batch_result payload or an id/metadata if available) and
increment failed_count so missing DB rows are visible; specifically, in the
block where you check result_record (the variable result_record inside the loop
that later calls update_stt_result), add a processLogger.warning (or existing
logger) call describing the missing result_record and the batch_result context,
and increment failed_count (not processed_count) so metrics reflect the skipped
result instead of silently continuing.
- Around line 346-363: The run finalization currently leaves runs stuck in
"processing" when all batches are terminal but there remain pending results;
modify the finalization logic in the block using count_results_by_status,
STTResultStatus, any_failed, errors and update_stt_run so that if all_terminal
is True and pending > 0 you treat the run as terminal (set status to "completed"
or "failed") and populate error_message accordingly (e.g., include failed_count
and a message about unresolved pending results or list errors); ensure
update_stt_run is called with the terminal status and a clear error_message so
subsequent cron cycles won't reprocess the same run.
In `@backend/app/models/collection.py`:
- Around line 42-48: The provider Field's description mentions 'google' but the
ProviderType enum (ProviderType) does not include a GOOGLE/GEMINI member; fix by
either adding a GOOGLE (or GEMINI) enum value to ProviderType (ensure downstream
serialization/name matches existing enum style) or update the provider Field
description in collection.py to only list supported providers (e.g., 'openai')
so OpenAPI/docs match the actual ProviderType; modify the ProviderType
definition or the description string in the provider Field accordingly and run
tests/type checks to confirm no regressions.
🧹 Nitpick comments (15)
backend/app/core/providers.py (1)
1-4: Consider modernizing type hints to Python 3.11+ builtins.Pre-existing code uses
Dict,List,Optionalfromtyping(lines 2, 22, 26, 62, 86). Since the coding guidelines require Python 3.11+, these can be replaced with the built-indict,list, andX | Nonesyntax. Not blocking, but worth a follow-up cleanup. As per coding guidelines: "Use Python 3.11+ with type hints throughout the codebase."backend/app/models/stt_evaluation.py (4)
15-17: Remove commented-out code with duplicate entries.The commented-out list on line 16 contains
"gemini-2.5-pro"twice (likely intended to be"gemini-2.5-flash"). Remove the dead comment to avoid confusion.Proposed fix
-# Supported STT models for evaluation -# SUPPORTED_STT_MODELS = ["gemini-2.5-pro", "gemini-2.5-pro", "gemini-2.0-flash"] -SUPPORTED_STT_MODELS = ["gemini-2.5-pro"] +# Supported STT models for evaluation +SUPPORTED_STT_MODELS = ["gemini-2.5-pro"]
327-341: Confusing variable names invalidate_models.The parameter
valid_modelreads as a singular validated model, but it's actually the full list being validated. The loop variablemodelsshadows the field name conceptually and reads as plural when it represents a single model string.Proposed fix
`@field_validator`("models") `@classmethod` - def validate_models(cls, valid_model: list[str]) -> list[str]: + def validate_models(cls, models: list[str]) -> list[str]: """Validate that all models are supported.""" - if not valid_model: + if not models: raise ValueError("At least one model must be specified") unsupported = [ - models for models in valid_model if models not in SUPPORTED_STT_MODELS + m for m in models if m not in SUPPORTED_STT_MODELS ] if unsupported: raise ValueError( f"Unsupported model(s): {', '.join(unsupported)}. " f"Supported models are: {', '.join(SUPPORTED_STT_MODELS)}" ) - return valid_model + return models
36-113: Consider adding database indexes on frequently queried foreign keys.
STTSamplewill be queried bydataset_id(listing samples for a dataset) andSTTResultbyevaluation_run_idandstt_sample_id(listing results for a run, looking up results for a sample). Without indexes on these columns, those queries will degrade as data grows.Example: adding table-level indexes
from sqlalchemy import Index class STTSample(SQLModel, table=True): __tablename__ = "stt_sample" __table_args__ = ( Index("idx_stt_sample_dataset", "dataset_id"), ) # ... fields ... class STTResult(SQLModel, table=True): __tablename__ = "stt_result" __table_args__ = ( Index("idx_stt_result_run", "evaluation_run_id"), Index("idx_stt_result_sample", "stt_sample_id"), ) # ... fields ...Also applies to: 116-221
274-278: No-op feedback update is silently accepted.Both
is_correctandcommentdefault toNone. A request body{}passes validation but performs a no-op update. Consider adding a model-level validator to require at least one field.Proposed fix
class STTFeedbackUpdate(BaseModel): """Request model for updating human feedback on a result.""" is_correct: bool | None = Field(None, description="Is the transcription correct?") comment: str | None = Field(None, description="Feedback comment") + + `@model_validator`(mode="after") + def check_at_least_one_field(self) -> "STTFeedbackUpdate": + if self.is_correct is None and self.comment is None: + raise ValueError("At least one of 'is_correct' or 'comment' must be provided") + return selfbackend/app/crud/evaluations/processing.py (1)
669-674: Use theEvaluationTypeenum for consistency.The STT polling in
cron.pyusesEvaluationType.STT.value, but here the type filter uses a raw string"text". Using the enum constant is safer against typos and keeps the codebase consistent.Proposed fix
+from app.models.stt_evaluation import EvaluationType + # Single query to fetch all processing text evaluation runs # STT/TTS evaluations have their own polling statement = select(EvaluationRun).where( EvaluationRun.status == "processing", - EvaluationRun.type == "text", + EvaluationRun.type == EvaluationType.TEXT.value, )backend/app/crud/stt_evaluations/batch.py (1)
123-124: UseRuntimeError(or a domain exception) instead of bareException.Raising bare
Exceptionmakes it harder for callers to catch specific errors. This also applies to line 190.Proposed fix
if not signed_urls: - raise Exception("Failed to generate signed URLs for any audio files") + raise RuntimeError("Failed to generate signed URLs for any audio files") ... if not batch_jobs: - raise Exception("Batch submission failed for all models") + raise RuntimeError("Batch submission failed for all models")Also applies to: 189-190
backend/app/tests/api/routes/test_stt_evaluation.py (1)
14-99: Consider moving factory helpers to a shared conftest or utility module.The
create_test_file,create_test_stt_dataset, andcreate_test_stt_samplehelpers are well-structured factory functions. Per the project's testing guidelines, these could be extracted to a shared test utilities module (e.g.,backend/app/tests/utils/stt_factories.py) so they can be reused by other test files that may need STT test data. As per coding guidelines, "Use factory pattern for test fixtures inbackend/app/tests/".backend/app/core/storage_utils.py (1)
139-206:upload_jsonl_to_object_storeduplicates the upload logic instead of delegating toupload_to_object_store.The new
upload_to_object_storewas introduced as the generic upload function, andupload_csv_to_object_storecorrectly delegates to it. However,upload_jsonl_to_object_storestill manually createsUploadFile, callsstorage.put, and duplicates the same error handling. This defeats the purpose of the generic function.Proposed fix
def upload_jsonl_to_object_store( storage: CloudStorage, results: list[dict], filename: str, subdirectory: str, format: Literal["json", "jsonl"] = "jsonl", ) -> str | None: ... try: - file_path = Path(subdirectory) / filename - if format == "jsonl": jsonl_content = ( "\n".join(json.dumps(result, ensure_ascii=False) for result in results) + "\n" ) - content_type = {"content-type": "application/jsonl"} + ct = "application/jsonl" else: jsonl_content = json.dumps(results, ensure_ascii=False) - content_type = {"content-type": "application/json"} + ct = "application/json" content_bytes = jsonl_content.encode("utf-8") - headers = Headers(content_type) - upload_file = UploadFile( - filename=filename, - file=BytesIO(content_bytes), - headers=headers, + return upload_to_object_store( + storage=storage, + content=content_bytes, + filename=filename, + subdirectory=subdirectory, + content_type=ct, ) - - destination = storage.put(source=upload_file, file_path=file_path) - object_store_url = str(destination) - ... except CloudStorageError as e: ...backend/app/tests/core/test_storage_utils.py (2)
359-369: Uniqueness test doesn't assert uniqueness.The test is named
test_unique_filenamesbut only asserts prefix format — it never comparesfilename1 != filename2. Combined with the comment acknowledging same-second collisions, this test adds no value beyondtest_timestamp_format.Either assert inequality or remove the test to avoid misleading coverage.
Proposed fix
def test_unique_filenames(self): """Test that consecutive calls produce different filenames.""" import time filename1 = generate_timestamped_filename("test") - time.sleep(0.01) # Small delay to ensure different timestamp + time.sleep(1.1) # Ensure different second-level timestamp filename2 = generate_timestamped_filename("test") - # They may be the same if called in the same second - # but the format should be correct - assert filename1.startswith("test_") - assert filename2.startswith("test_") + assert filename1 != filename2, "Consecutive filenames should differ"
24-27: Add return type hints to test methods for consistency.Several test methods lack
-> Nonereturn type annotations (e.g.,test_mp3_url,test_wav_url,test_successful_upload, etc.), while others liketest_upload_json_file_successhave them. As per coding guidelines, all functions should have type hints on return values.Also applies to: 29-32, 97-109, 166-176, 197-219, 327-331, 333-337, 353-357
backend/app/crud/stt_evaluations/run.py (1)
218-220: Cannot explicitly clear a field toNone(e.g.,error_message).The
if value is not Noneguard means callers can never reset a field — for instance, clearingerror_messageon a retry. If intentional, document it; otherwise, consider a sentinel or explicit include-list.Example using a sentinel
+_UNSET = object() + def update_stt_run( *, session: Session, run_id: int, - status: str | None = None, - score: dict[str, Any] | None = None, - error_message: str | None = None, - object_store_url: str | None = None, - batch_job_id: int | None = None, + status: str | None = _UNSET, + score: dict[str, Any] | None = _UNSET, + error_message: str | None = _UNSET, + object_store_url: str | None = _UNSET, + batch_job_id: int | None = _UNSET, ) -> EvaluationRun | None: ... for field, value in updates.items(): - if value is not None: + if value is not _UNSET: setattr(run, field, value)backend/app/crud/stt_evaluations/result.py (3)
22-77:session.flush()beforesession.commit()is redundant.
commit()already callsflush()internally. The extraflush()on line 69 adds no value.Proposed fix
session.add_all(results) - session.flush() session.commit()
247-299: CRUD layer raisesHTTPException— mixing concerns.
update_human_feedbackraisesHTTPException(line 280), coupling the data-access layer to HTTP semantics. If this function is called from a non-HTTP context (e.g., a script, background job), the HTTP exception is inappropriate.Return
Noneand let the API route handler raise the 404.Proposed fix
-from app.core.exception_handlers import HTTPException ... def update_human_feedback( ... ) -> STTResult | None: ... result = get_stt_result_by_id(...) if not result: - raise HTTPException(status_code=404, detail="Result not found") + return None ...Then in the API route, check for
Noneand raiseHTTPException(status_code=404, ...)there.
350-358: Simplify dict comprehension per Ruff C416.Proposed fix
- return {status: count for status, count in rows} + return dict(rows)
| Start an STT evaluation run on a dataset. | ||
|
|
||
| The evaluation will: | ||
| 1. Process each audio sample through the specified providers | ||
| 2. Generate transcriptions using Gemini Batch API | ||
| 3. Store results for human review | ||
|
|
||
| **Supported providers:** gemini-2.5-pro |
There was a problem hiding this comment.
Documentation uses "providers" but the API model uses "models".
STTEvaluationRunCreate defines the field as models: list[str], but this doc references "providers" (lines 4 and 8). Update the terminology to match the API contract to avoid confusing consumers.
Proposed fix
Start an STT evaluation run on a dataset.
The evaluation will:
-1. Process each audio sample through the specified providers
+1. Process each audio sample through the specified models
2. Generate transcriptions using Gemini Batch API
3. Store results for human review
-**Supported providers:** gemini-2.5-pro
+**Supported models:** gemini-2.5-pro📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| Start an STT evaluation run on a dataset. | |
| The evaluation will: | |
| 1. Process each audio sample through the specified providers | |
| 2. Generate transcriptions using Gemini Batch API | |
| 3. Store results for human review | |
| **Supported providers:** gemini-2.5-pro | |
| Start an STT evaluation run on a dataset. | |
| The evaluation will: | |
| 1. Process each audio sample through the specified models | |
| 2. Generate transcriptions using Gemini Batch API | |
| 3. Store results for human review | |
| **Supported models:** gemini-2.5-pro |
🤖 Prompt for AI Agents
In `@backend/app/api/docs/stt_evaluation/start_evaluation.md` around lines 1 - 8,
The docs refer to "providers" but the API expects "models"; update the wording
in this doc so it matches the STTEvaluationRunCreate contract by replacing
mentions of "providers" with "models" and noting the field name `models:
list[str]` where examples or supported options are listed (e.g., change
"**Supported providers:** gemini-2.5-pro" to "**Supported models:**
gemini-2.5-pro") and ensure any descriptive text referencing "providers" (lines
describing processing through providers) instead mentions processing through the
specified `models`.
| # Submit one batch job per model | ||
| batch_jobs: dict[str, Any] = {} | ||
| first_batch_job_id: int | None = None | ||
|
|
||
| for model in models: | ||
| model_path = f"models/{model}" | ||
| batch_provider = GeminiBatchProvider( | ||
| client=gemini_client.client, model=model_path | ||
| ) | ||
|
|
||
| try: | ||
| batch_job = start_batch_job( | ||
| session=session, | ||
| provider=batch_provider, | ||
| provider_name="google", | ||
| job_type="stt_evaluation", | ||
| organization_id=org_id, | ||
| project_id=project_id, | ||
| jsonl_data=jsonl_data, | ||
| config={ | ||
| "model": model, | ||
| "stt_provider": model, | ||
| "evaluation_run_id": run.id, | ||
| }, | ||
| ) | ||
|
|
||
| batch_jobs[model] = { | ||
| "batch_job_id": batch_job.id, | ||
| "provider_batch_id": batch_job.provider_batch_id, | ||
| } | ||
|
|
||
| if first_batch_job_id is None: | ||
| first_batch_job_id = batch_job.id | ||
|
|
||
| logger.info( | ||
| f"[start_stt_evaluation_batch] Batch job created | " | ||
| f"run_id: {run.id}, model: {model}, " | ||
| f"batch_job_id: {batch_job.id}" | ||
| ) | ||
|
|
||
| except Exception as e: | ||
| logger.error( | ||
| f"[start_stt_evaluation_batch] Failed to submit batch | " | ||
| f"model: {model}, error: {str(e)}" | ||
| ) | ||
| pending = get_pending_results_for_run( | ||
| session=session, run_id=run.id, provider=model | ||
| ) | ||
| for result in pending: | ||
| update_stt_result( | ||
| session=session, | ||
| result_id=result.id, | ||
| status=STTResultStatus.FAILED.value, | ||
| error_message=f"Batch submission failed for {model}: {str(e)}", | ||
| ) | ||
|
|
||
| if not batch_jobs: | ||
| raise Exception("Batch submission failed for all models") | ||
|
|
||
| # Link first batch job to the evaluation run (for pending run detection) | ||
| update_stt_run( | ||
| session=session, | ||
| run_id=run.id, | ||
| status="processing", | ||
| batch_job_id=first_batch_job_id, | ||
| ) |
There was a problem hiding this comment.
Only the first batch job is linked to the evaluation run — multi-model runs will lose track of subsequent jobs.
first_batch_job_id (line 164-165) records only the first model's batch job. The cron poller in poll_all_pending_stt_evaluations discovers runs via EvaluationRun.batch_job_id, so batch jobs for any additional models won't be polled or processed. While currently only one model is supported, this code path explicitly loops over multiple models and would silently drop results for all but the first.
Consider either: (a) creating one EvaluationRun per model, (b) storing all batch job IDs (e.g., in the run's metadata/config), or (c) adding a comment/assertion that multi-model is not yet fully supported.
🤖 Prompt for AI Agents
In `@backend/app/crud/stt_evaluations/batch.py` around lines 133 - 198, The code
only saves first_batch_job_id (first_batch_job_id) to the EvaluationRun so only
the first model's batch job is discoverable by poll_all_pending_stt_evaluations
(which uses EvaluationRun.batch_job_id); fix by persisting all created batch job
IDs instead of only the first: collect batch_job.id values from start_batch_job
into batch_jobs (or a list batch_job_ids) and call update_stt_run with that
collection (e.g., update_stt_run(session, run_id=run.id, status="processing",
batch_job_id=None, metadata={"batch_job_ids": batch_job_ids}) or add a new
parameter/field to store batch_job_ids); alternatively, if multi-model runs are
not supported, add an explicit assertion or comment near the loop and avoid
creating multiple jobs — reference functions/vars: start_batch_job, batch_jobs,
first_batch_job_id, update_stt_run, and poll_all_pending_stt_evaluations when
making the change.
| all_terminal = True | ||
| any_succeeded = False | ||
| any_failed = False |
There was a problem hiding this comment.
any_succeeded is assigned but never used.
The variable is set on lines 283 and 329 but never read. Static analysis (Ruff F841) confirms this. Either remove it or incorporate it into the final status logic if the intent was to distinguish "all failed" from "partially succeeded."
Proposed fix — remove unused variable
all_terminal = True
- any_succeeded = False
any_failed = False
errors: list[str] = []
for batch_job in batch_jobs:
provider_name = batch_job.config.get("stt_provider", "unknown")
# Skip batch jobs already in terminal state that have been processed
if batch_job.provider_status in TERMINAL_STATES:
if batch_job.provider_status == BatchJobState.SUCCEEDED.value:
- any_succeeded = True
+ pass
else:
any_failed = True
...
if provider_status == BatchJobState.SUCCEEDED.value:
await process_completed_stt_batch(
session=session,
run=run,
batch_job=batch_job,
batch_provider=batch_provider,
)
- any_succeeded = True
else:Also applies to: 329-329
🤖 Prompt for AI Agents
In `@backend/app/crud/stt_evaluations/cron.py` around lines 282 - 284, Remove the
unused boolean any_succeeded (set at any_succeeded = False and later modified)
or else use it in the final-status calculation; specifically, edit the block
that defines/updates all_terminal, any_failed, and any_succeeded (search for
any_succeeded and any_failed in cron.py) and either delete all references and
assignments to any_succeeded, or integrate it into the terminal-state logic
(e.g., use any_succeeded to distinguish "all failed" vs "partially succeeded"
when computing the overall status) and remove any linter warnings. Ensure no
other code relies on any_succeeded before removing.
| # All batch jobs are done - finalize the run | ||
| status_counts = count_results_by_status(session=session, run_id=run.id) | ||
| pending = status_counts.get(STTResultStatus.PENDING.value, 0) | ||
| failed_count = status_counts.get(STTResultStatus.FAILED.value, 0) | ||
|
|
||
| final_status = "completed" if pending == 0 else "processing" | ||
| error_message = None | ||
| if any_failed: | ||
| error_message = "; ".join(errors) | ||
| elif failed_count > 0: | ||
| error_message = f"{failed_count} transcription(s) failed" | ||
|
|
||
| update_stt_run( | ||
| session=session, | ||
| run_id=run.id, | ||
| status=final_status, | ||
| error_message=error_message, | ||
| ) |
There was a problem hiding this comment.
Run stuck in "processing" forever when pending results remain after all batches are terminal.
When all_terminal is True but pending > 0 (e.g., unmatched custom_ids or missing result records), final_status is set to "processing" (line 351). On the next cron cycle, all batch jobs are already terminal and skipped (line 291-299), so the same finalization logic re-runs and sets "processing" again — creating an infinite loop.
If all batches have completed but results are still pending, those results will never be resolved. Mark the run as "completed" (or "failed") with an appropriate error message instead.
Proposed fix
- final_status = "completed" if pending == 0 else "processing"
+ if pending > 0:
+ # All batches are terminal but some results weren't matched/processed
+ final_status = "completed"
+ if error_message:
+ error_message += f"; {pending} result(s) still unresolved"
+ else:
+ error_message = f"{pending} result(s) still unresolved after all batches completed"
+ else:
+ final_status = "completed"🤖 Prompt for AI Agents
In `@backend/app/crud/stt_evaluations/cron.py` around lines 346 - 363, The run
finalization currently leaves runs stuck in "processing" when all batches are
terminal but there remain pending results; modify the finalization logic in the
block using count_results_by_status, STTResultStatus, any_failed, errors and
update_stt_run so that if all_terminal is True and pending > 0 you treat the run
as terminal (set status to "completed" or "failed") and populate error_message
accordingly (e.g., include failed_count and a message about unresolved pending
results or list errors); ensure update_stt_run is called with the terminal
status and a clear error_message so subsequent cron cycles won't reprocess the
same run.
| if result_record: | ||
| if batch_result.get("response"): | ||
| text = batch_result["response"].get("text", "") | ||
| update_stt_result( | ||
| session=session, | ||
| result_id=result_record.id, | ||
| transcription=text, | ||
| status=STTResultStatus.COMPLETED.value, | ||
| ) | ||
| processed_count += 1 | ||
| else: | ||
| update_stt_result( | ||
| session=session, | ||
| result_id=result_record.id, | ||
| status=STTResultStatus.FAILED.value, | ||
| error_message=batch_result.get("error", "Unknown error"), | ||
| ) | ||
| failed_count += 1 |
There was a problem hiding this comment.
Silently skips results when result_record is not found — no logging or error counting.
If result_record is None (line 433), the loop continues implicitly without any warning. This makes it impossible to diagnose why results are stuck in PENDING after batch completion. At minimum, log a warning and increment failed_count.
Proposed fix
result_record = session.exec(stmt).one_or_none()
if result_record:
if batch_result.get("response"):
...
else:
...
+ else:
+ logger.warning(
+ f"[process_completed_stt_batch] No result record found | "
+ f"run_id={run.id}, sample_id={sample_id}, provider={stt_provider}"
+ )
+ failed_count += 1📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if result_record: | |
| if batch_result.get("response"): | |
| text = batch_result["response"].get("text", "") | |
| update_stt_result( | |
| session=session, | |
| result_id=result_record.id, | |
| transcription=text, | |
| status=STTResultStatus.COMPLETED.value, | |
| ) | |
| processed_count += 1 | |
| else: | |
| update_stt_result( | |
| session=session, | |
| result_id=result_record.id, | |
| status=STTResultStatus.FAILED.value, | |
| error_message=batch_result.get("error", "Unknown error"), | |
| ) | |
| failed_count += 1 | |
| if result_record: | |
| if batch_result.get("response"): | |
| text = batch_result["response"].get("text", "") | |
| update_stt_result( | |
| session=session, | |
| result_id=result_record.id, | |
| transcription=text, | |
| status=STTResultStatus.COMPLETED.value, | |
| ) | |
| processed_count += 1 | |
| else: | |
| update_stt_result( | |
| session=session, | |
| result_id=result_record.id, | |
| status=STTResultStatus.FAILED.value, | |
| error_message=batch_result.get("error", "Unknown error"), | |
| ) | |
| failed_count += 1 | |
| else: | |
| logger.warning( | |
| f"[process_completed_stt_batch] No result record found | " | |
| f"run_id={run.id}, sample_id={sample_id}, provider={stt_provider}" | |
| ) | |
| failed_count += 1 |
🤖 Prompt for AI Agents
In `@backend/app/crud/stt_evaluations/cron.py` around lines 435 - 452, The loop
currently skips when result_record is None which hides failures; update the
branch handling to log a warning (including identifiers like the batch_result
payload or an id/metadata if available) and increment failed_count so missing DB
rows are visible; specifically, in the block where you check result_record (the
variable result_record inside the loop that later calls update_stt_result), add
a processLogger.warning (or existing logger) call describing the missing
result_record and the batch_result context, and increment failed_count (not
processed_count) so metrics reflect the skipped result instead of silently
continuing.
| provider: ProviderType = ( | ||
| Field( | ||
| nullable=False, | ||
| description="LLM provider used for this collection (e.g., 'openai', 'bedrock', 'gemini', etc)", | ||
| description="LLM provider used for this collection (e.g., 'openai', 'bedrock', 'google', etc)", | ||
| sa_column_kwargs={"comment": "LLM provider used for this collection"}, | ||
| ), | ||
| ) |
There was a problem hiding this comment.
Description references 'google' but ProviderType enum doesn't include it.
The description on line 45 lists 'google' as an example provider, but the ProviderType enum (lines 15–20) only defines openai—GEMINI is commented out and there's no GOOGLE member. This is misleading for API consumers who see the field description (e.g., in OpenAPI docs). Either add GOOGLE to ProviderType or keep the description aligned with what's actually accepted.
🤖 Prompt for AI Agents
In `@backend/app/models/collection.py` around lines 42 - 48, The provider Field's
description mentions 'google' but the ProviderType enum (ProviderType) does not
include a GOOGLE/GEMINI member; fix by either adding a GOOGLE (or GEMINI) enum
value to ProviderType (ensure downstream serialization/name matches existing
enum style) or update the provider Field description in collection.py to only
list supported providers (e.g., 'openai') so OpenAPI/docs match the actual
ProviderType; modify the ProviderType definition or the description string in
the provider Field accordingly and run tests/type checks to confirm no
regressions.
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
3 participants
Summary
Target issue is #533
Checklist
Before submitting a pull request, please ensure that you mark these task.
fastapi run --reload app/main.pyordocker compose upin the repository root and test.Notes
New Features
Functional Requirements Testing
Summary by CodeRabbit
New Features
Documentation
Tests