Classification : Model evaluation of fine tuned models

[nishika26]

Summary

Target issue is #303

Checklist

Before submitting a pull request, please ensure that you mark these task.

• Ran fastapi run --reload app/main.py or docker compose up in the repository root and test.
• If you've fixed a bug or added code that is tested and has test cases.

Notes

Core Logic (app/core/finetune/evaluation.py)

• Init – Store model info, file ID, prompt, labels lists.
• Load Data – Read the OpenAI NDJSON file line-by-line, take the last user message as the prompt and last assistant message as the cleaned label, and store them with all unique labels
• Normalize – Clean prediction, match to allowed labels, or if no exact match, choose the most similar label based on a 0.7 similarity threshold (normalization_cutoff).
• Predict – Send prompts to model, retry on errors or timeouts, normalize output.
• Evaluate – Compare predictions to true labels, return MCC/Accuracy/F1.
• Run – Do all steps in sequence.

Router Logic (app/api/routers/model_evaluation.py)

• evaluate_model — Creates evaluation jobs for provided fine-tuning IDs, skipping those with active evaluations, and queues background processing.
• run_model_evaluation — Runs the evaluation via ModelEvaluator, stores metrics (MCC, accuracy, F1), and updates evaluation status in the database.
• get_top_model_by_doc_id — Retrieves the best-performing model for a given document based on MCC score.
• get_evals_by_doc_id — Lists all evaluations for a given document.
• Note: The metrics list (["mcc", "f1", "accuracy"]) is currently hardcoded but can be made configurable via user input in future iterations.

CRUD Logic (app/crud/model_evaluation.py)

• create_model_evaluation — Creates a new evaluation entry from a fine-tuning job, pre-filling model details and metrics.
• fetch_by_eval_id / fetch_eval_by_doc_id / fetch_top_model_by_doc_id — Retrieves evaluations by ID, document ID, or selects the highest-MCC model for a document.
• fetch_active_model_evals — Returns all active (non-failed, non-deleted) evaluations for a fine-tuning ID.
• update_model_eval — Updates an evaluation record’s status, score, and other fields.

Test data Logic (app/tests/utils/test_data.py)

Adds helper functions to create test fine-tuning jobs and corresponding model evaluation records for testing purposes:

• create_test_finetuning_job_with_extra_fields — Generates test fine-tuning jobs and assigns mock testing_file_id and fine_tuned_model values.
• create_test_model_evaluation — Uses these jobs to create associated Model_Evaluation entries with preset metrics (mcc, f1, accuracy) and pending status.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing Touches

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch feature/model_eval

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[kartpop]

similar to the fine_tuning comment in the previous PR, not sure if model_evaluation should be part of the project table - can we think of any good use case for this?

[nishika26]

model evaluation uses openai chat completion for evaluating predictions from the fine tuned model, same reason as before, keeping model_eval a part of project table will help with cost tracking and observability later, since it also relies on the OpenAI resource.