Skip to content

Batch Job: Improve error handling #723

@AkhileshNegi

Description

@AkhileshNegi

Is your feature request related to a problem?
When an OpenAI batch job completes with failed requests, the status is set to completed without an output_file_id, leading to a misleading success indication and an unhelpful generic error message that obscures the true cause of failure.

Describe the solution you'd like
Enhance the batch job error handling for cases with no output_file_id but an error_file_id:

  • Download the error file
  • Parse JSONL entries to extract response.body.error.message
  • Count unique errors and highlight the most common one
  • Update batch_job.error_message with the extracted message
  • Mark the evaluation_run as failed with the same error message

Error Batch results

Batch(id='batch_69c3c347282c81908ede48ecc95bf201', completion_window='24h', created_at=1774437191, endpoint='/v1/responses', input_file_id='file-2qvouj6sKUCGTbu3odouhk', object='batch', status='completed', cancelled_at=None, cancelling_at=None, completed_at=1774437324, error_file_id='file-QzTKe9srv1bbmpX5MwCTZx', errors=None, expired_at=None, expires_at=1774523591, failed_at=None, finalizing_at=1774437322, in_progress_at=1774437194, metadata={'description': 'Evaluation: test_run_new_1'}, model='gpt-5.4-pro-2026-03-05', output_file_id=None, request_counts=BatchRequestCounts(completed=0, failed=9, total=9), usage=BatchUsage(input_tokens=0, input_tokens_details=InputTokensDetails(cached_tokens=0), output_tokens=0, output_tokens_details=OutputTokensDetails(reasoning_tokens=0), total_tokens=0))

Success Batch results

Batch(id='batch_69c4f062ecfc8190b1d29752f95468e6', completion_window='24h', created_at=1774514274, endpoint='/v1/responses', input_file_id='file-DAyWeG7cwHwYLrjoZ61es7', object='batch', status='completed', cancelled_at=None, cancelling_at=None, completed_at=1774514642, error_file_id=None, errors=None, expired_at=None, expires_at=1774600674, failed_at=None, finalizing_at=1774514640, in_progress_at=1774514278, metadata={'description': 'Evaluation: gpt-5.4-pro'}, model='gpt-5.4-pro-2026-03-05', output_file_id='file-SDVYLXKe7iWoMERsPfVShQ', request_counts=BatchRequestCounts(completed=10, failed=0, total=10), usage=BatchUsage(input_tokens=420, input_tokens_details=InputTokensDetails(cached_tokens=0), output_tokens=6036, output_tokens_details=OutputTokensDetails(reasoning_tokens=4423), total_tokens=6456))

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Status

Closed

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions