-
Notifications
You must be signed in to change notification settings - Fork 10
Batch Job: Improve error handling #723
Copy link
Copy link
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
Is your feature request related to a problem?
When an OpenAI batch job completes with failed requests, the status is set to completed without an output_file_id, leading to a misleading success indication and an unhelpful generic error message that obscures the true cause of failure.
Describe the solution you'd like
Enhance the batch job error handling for cases with no output_file_id but an error_file_id:
- Download the error file
- Parse JSONL entries to extract
response.body.error.message - Count unique errors and highlight the most common one
- Update
batch_job.error_messagewith the extracted message - Mark the
evaluation_runas failed with the same error message
Error Batch results
Batch(id='batch_69c3c347282c81908ede48ecc95bf201', completion_window='24h', created_at=1774437191, endpoint='/v1/responses', input_file_id='file-2qvouj6sKUCGTbu3odouhk', object='batch', status='completed', cancelled_at=None, cancelling_at=None, completed_at=1774437324, error_file_id='file-QzTKe9srv1bbmpX5MwCTZx', errors=None, expired_at=None, expires_at=1774523591, failed_at=None, finalizing_at=1774437322, in_progress_at=1774437194, metadata={'description': 'Evaluation: test_run_new_1'}, model='gpt-5.4-pro-2026-03-05', output_file_id=None, request_counts=BatchRequestCounts(completed=0, failed=9, total=9), usage=BatchUsage(input_tokens=0, input_tokens_details=InputTokensDetails(cached_tokens=0), output_tokens=0, output_tokens_details=OutputTokensDetails(reasoning_tokens=0), total_tokens=0))
Success Batch results
Batch(id='batch_69c4f062ecfc8190b1d29752f95468e6', completion_window='24h', created_at=1774514274, endpoint='/v1/responses', input_file_id='file-DAyWeG7cwHwYLrjoZ61es7', object='batch', status='completed', cancelled_at=None, cancelling_at=None, completed_at=1774514642, error_file_id=None, errors=None, expired_at=None, expires_at=1774600674, failed_at=None, finalizing_at=1774514640, in_progress_at=1774514278, metadata={'description': 'Evaluation: gpt-5.4-pro'}, model='gpt-5.4-pro-2026-03-05', output_file_id='file-SDVYLXKe7iWoMERsPfVShQ', request_counts=BatchRequestCounts(completed=10, failed=0, total=10), usage=BatchUsage(input_tokens=420, input_tokens_details=InputTokensDetails(cached_tokens=0), output_tokens=6036, output_tokens_details=OutputTokensDetails(reasoning_tokens=4423), total_tokens=6456))
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Type
Projects
Status
Closed