Skip to content

LLM critic validator enhancement #76

@rkritika1508

Description

@rkritika1508

Describe the bug
On calling LLM critic validator, sometimes we get responses like -

{
  "success": false,
  "data": {
    "response_id": "11c2d6b7-6f2e-4d09-a78a-23b3d40aed6b",
    "rephrase_needed": false
  },
  "error": "The response failed the following metrics: ['query']."
}

And sometimes like this -

{
  "success": false,
  "data": {
    "response_id": "4985fe71-39fb-42d4-977c-d2d27a16f2d6",
    "rephrase_needed": false
  },
  "error": "The response is missing or has invalid evaluations for the following metrics: ['query']."
}

Additional context
We are going to investigate why that happens and override the errors for the validator. The end goal is to provide proper exception to the end users.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

Projects

Status

To Do

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions