-
Notifications
You must be signed in to change notification settings - Fork 2
LLM critic validator enhancement #76
Copy link
Copy link
Open
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
On calling LLM critic validator, sometimes we get responses like -
{
"success": false,
"data": {
"response_id": "11c2d6b7-6f2e-4d09-a78a-23b3d40aed6b",
"rephrase_needed": false
},
"error": "The response failed the following metrics: ['query']."
}
And sometimes like this -
{
"success": false,
"data": {
"response_id": "4985fe71-39fb-42d4-977c-d2d27a16f2d6",
"rephrase_needed": false
},
"error": "The response is missing or has invalid evaluations for the following metrics: ['query']."
}
Additional context
We are going to investigate why that happens and override the errors for the validator. The end goal is to provide proper exception to the end users.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Type
Projects
Status
To Do