Beyond Contextual Embeddings: A Comparative Analysis of Feature-Engineered Classical Machine Learning and Transformer-Based Approaches for Deceptive Opinion Spam Detection

Abstract: We compare classical machine learning with a published fine-tuned BERT result on the Deceptive Opinion Spam Corpus (OpSpam), a public benchmark of 1,600 hotel reviews. Naive Bayes, logistic regression and linear SVM classifiers were trained on TF-IDF features under stratified 5-fold cross-validation, the protocol used by the BERT baseline we compare against. The strongest configuration joins word n-grams (1 to 3) with character n-grams (3 to 5). It reaches 91.12% accuracy on a fixed seed and 90.78% averaged over ten repeated runs, against 89.1% reported for BERT, and it trains in about two seconds per fold on a CPU. The margin needs careful reading. A plain unigram and bigram SVM already scores 89.38%, the fusion improves on the better of its two feature views by under one point, and its edge over the plain SVM is borderline (corrected resampled t-test, p = 0.053). When folds are grouped by hotel, so that test hotels never appear in training, every n-gram variant falls to about 89%, level with the BERT figure. We also audit the highest accuracy we found for this corpus, 96.03% for a RoBERTa and LSTM hybrid. It comes from an 80/20 split without cross-validation or variance estimates, and its precision, recall and accuracy are hard to reconcile with a single balanced test set. Under the BERT paper’s own protocol the n-gram fusion finishes 1.7 to 2.0 points ahead, but the lead disappears when hotels are held out.

Keywords: BERT, character n-grams, cross-validation, deceptive opinion spam, fake review detection, OpSpam, support vector machine, TF-IDF.