Benchmarking large language models for automated labeling: The case of issue report classification

This repository contains the replication material for the study described in the paper:

Giuseppe Colavito, Filippo Lanubile, Nicole Novielli. "Benchmarking large language models for automated labeling: The case of issue report classification, Information and Software Technology, 2025, 107758, ISSN 0950-5849, https://doi.org/10.1016/j.infsof.2025.107758

Datasets

All datasets are publicly available and distributed in the scope of previous work, as mentioned in the paper. In particular, we used two datasets of GitHub issues from open source software projects, which had been collected to support the issue-classification challenges of two editions of the Natural Language-Based Software Engineering (NLBSE) workshop series.

• Manually annotated dataset derived from the NLBSE '23 Tool Challenge dataset, available at: https://doi.org/10.5281/zenodo.7628150
• NLBSE '24 Tool Challenge dataset, available at: https://ieeexplore.ieee.org/document/10189143