NUS · CS3264 Machine Learning · Jan – May 2025
Fake News Detection & the Fairness Cost of Metadata
A systematic ablation on 12,836 short political statements from PolitiFact (the LIAR benchmark), binarised into true/false. We swept classical models (Logistic Regression, SVC, Random Forest, XGBoost), an MLP, and a fine-tuned DistilBERT across TF-IDF, Word2Vec and metadata feature sets, with the fairness analysis done properly instead of gestured at. Framed around a real deployment context: Singapore's POFMA fact-checking ecosystem, where manual review doesn't scale.
01The ablation
Three feature variants per model family, statement-only, statement plus selected metadata (party, speaker title, subject), and full pipelines including the speaker's historical credibility counts, across sparse TF-IDF and dense pre-trained Word2Vec representations. Eighteen configurations in all, each with proper preprocessing: top-15 category encoding with rare values grouped, multi-label subject splitting, credibility counts as numeric features.
Patterns that held: dense embeddings beat TF-IDF for text-only inputs, SVMs beat logistic baselines in high dimensions, and the best configuration was XGBoost over Word2Vec plus party affiliation at 64.5% accuracy.
02The finding that mattered
Metadata consistently lifts accuracy by 1–4 points, and the ablations show part of that lift comes from learning speaker identity rather than statement content. A model fed only the speaker's credibility history, with no statement text at all, still reaches 56.7% accuracy, essentially a prior on the person. That is a measurable fairness problem: a claim's predicted truthfulness should not depend on who said it, and any system deployed into a fact-checking pipeline would need that trade-off surfaced, not buried.
For calibration: published binary-classification results on LIAR sit in a similar band, it is a genuinely hard benchmark, and the controlled comparison was the point.
next project
LectureAI →