SemEval 2025: classifying food hazards and extracting associated passages

Context

As part of the natural language processing course at Université de Sherbrooke, we worked on SemEval 2025 Task 9, a challenge focused on analyzing 6,644 annotated food incident reports.

The challenge consists of identifying the type of hazard present in a report and determining the category of the product concerned. We chose to go further: can a model also indicate the passages in the report that explain its prediction?

Approach

The project was conducted as an experimental study. We started by setting up baseline models, then compared different approaches before building our final solution.

The chosen idea is to train a single model to perform two tasks in parallel: classify the food hazard and identify passages in the text associated with that decision.

To train this second task, we built annotations using several language models. These annotations were then used as an additional signal during training.

Result

All results presented here were measured on the validation set, not on the challenge test set. The ST1 score combines hazard macro-F1 with product macro-F1 calculated when the hazard is predicted correctly.

After hyperparameter tuning, the final multi-task model reaches 0.7937 on this metric, compared with 0.7853 for the same architecture before tuning. In earlier experiments, the BERT baseline scored 0.7052; adding data cleaning, focal loss, and multi-task learning over hazards and products raised this score to 0.7838.

The metric combining ST1 classification with BIO extraction reaches 0.7606, compared with 0.7440 for the first architecture integrating both tasks.

For passage extraction, Dice scores reach 0.6972 for hazards and 0.7577 for products. A complementary evaluation on 562 validation examples measures 66.90% sufficiency and 57.12% comprehensiveness.

The interest of the project is therefore not solely classification performance: we also studied to what extent a model could produce an explanation directly extracted from the text.

My contribution

On the team, I set up the Hydra-configurable pipeline, data loaders, evaluation, and experiment tracking. I trained the TF-IDF, logistic regression, and BERT baselines, then contributed to comparing approaches and tuning their hyperparameters.

I also worked on the LLM-based BIO annotation strategy and its export, then extended the multi-task model to integrate passage extraction: token alignment, Dice loss, checkpointing, and logging.

Finally, I contributed to repository documentation and interpretation of the results presented in the final report.

Limits

The extracted passages constitute an approximation of the model's explanation. A passage may be relevant without precisely representing the elements that actually led the model to its decision.

Of the 562 evaluated examples, 36.65% satisfy both sufficiency and comprehensiveness. The model can therefore still rely on information outside the extracted passages.

Furthermore, the annotations used to train this part of the system were generated with language models and therefore do not replace human annotation.

Next steps

The most important next step would be to compare the explanations produced by the model with human annotations, to verify whether the extracted passages are truly relevant and faithful to the decisions made.