Classification: Unified API endpoint handling all finetuning - model evaluations functions internally

[nishika26]

Problem

Our current pipeline for fine-tuning a GPT model for classification and then evaluating all created models is split into five separate steps. While the pipeline works, the number of endpoints makes it cumbersome for users.

Proposed Solution

• Introduce a single initiation endpoint where users can:

  • Submit the request body required to start fine-tuning.

  • Upload the dataset file directly as part of the request.

• From there, the process will be handled internally:

  • The fine-tuning job is created.

  • A cron job periodically checks the job status until completion.

  • Once the fine-tuned model is ready, it is automatically passed to the model evaluation function.