🚀 **New Finetuning Script Released!** 🛠️

[urchade]

I'm excited to announce the release of & new finetuning script! It includes features such as gradient accumulation and mixed precision training. A big thanks to @MarcusLoppe for the contribution! 🙌

Link: https://github.com/urchade/GLiNER/tree/main/examples/finetuning

[andrea-mar]

i am trying to use this notebook for fine-tuning , but i fail to understand how loss is computed ..? i see the training logs , but no validation / test logs ...how can i visualize /access these?

[urchade]

Hi @andrea-mar, I invite you to read the section 2 of this paper: https://arxiv.org/pdf/2311.08526

[andrea-mar]

Thank you for the prompt reply @urchade ! I have read the article, it is amazing! I understand that the loss used was a binary cross entropy. My question was more regarding the code implementation for the fine-tuning example: when i run the notebook, only the training loss seems to be computed, or at least that is my understanding when i look at the output directory in the /trainer_state.json file. Here the only metric i see is 'loss' ... is this the evaluation set loss or the training set loss? also, the trainer output shows only 'training_loss'... i am interested to see training and validation loss together (i am trying to avoid overfitting ). What am I missing?

[urchade]

Hi @andrea-mar,
I have not implemented validation loss and think that it is not important. The main objective is to improve performance metrics such as the F1 score on validation, which does not correlate well with loss after some point in the training.
But of course, you can add it if you really need it for some reason

[andrea-mar]

ok... I am still unclear on where these metrics are logged while finetuning ... i see in the model class the colling model.evaluate(dataset) will return these metrics ... but not sure how to call this in training or where the predictions on the eval_dataset are logged ...?

[semerekiros]

Hi @andrea-mar, @urchade
I am trying to also calculate the F1 score on the eval_dataset every epoch in my finetuning. Were you able to compute the metrics in the Trainer directly? Can you provide a code snippet if you did?