A Comparative Study of Traditional and Ensemble Classifiers for Software Defect Prediction on the NASA PROMISE Repository

Abstract: Software defect prediction (SDP) directs limited testing effort toward the modules most likely to fail, and the NASA PROMISE datasets are its most common benchmark. Published accuracies on these datasets differ widely, from about 76% to more than 97%, and studies rarely report the majority-class baseline or state how resampling and data splitting were organised. This paper presents a controlled comparison of ten traditional and ensemble classifiers on CM1, KC1, KC2, PC1 and JM1, using the ADASYN-based study of Kabir et al. (2025) as the reference benchmark. We vary three factors: the model family (logistic regression, k-nearest neighbours, tuned SVM, random forest, extra trees, gradient boosting, histogram gradient boosting, AdaBoost, voting and stacking), the evaluation protocol (single split, oversampling before the split, oversampling inside training folds, repeated stratified cross-validation) and the data version (original versus deduplicated). Alongside accuracy, we report balanced accuracy, sensitivity and AUC. Three findings emerge. First, 11% to 43% of modules in the original datasets duplicate another module, and removing duplicate and inconsistent rows lowers the best accuracy by up to nine points. Second, oversampling the whole dataset before splitting raises accuracy by two to eight points over a raw split, but on a test set that is half synthetic; applied inside training folds, ADASYN lowers accuracy by roughly one to five points while raising sensitivity, and the area under the ROC curve is almost unchanged. Third, model choice matters little: top-ranked models differ from logistic regression by at most 1.1 accuracy points, and on CM1 and PC1 no model clearly exceeds the majority baseline. Under repeated stratified cross-validation our best accuracy exceeds the reference study on PC1 and JM1 for both data versions, while its CM1 result is not matched. We therefore recommend reporting accuracy with the baseline, balanced accuracy and sensitivity under a leakage-free protocol.

Keywords: Software defect prediction; NASA PROMISE repository; class imbalance; ADASYN; ensemble learning; data leakage; duplicate data; evaluation protocol