NUS · CS4248 Natural Language Processing · Aug – Dec 2025
Margin-Triggered Reranking for Extractive QA
An error-analysis-first NLP project (team of five): instead of throwing a bigger model at SQuAD, we measured where the headroom actually was and spent compute only there. Fine-tuned RoBERTa-base to 84.28 EM / 90.93 F1, then added a bi-encoder reranking layer that activates only when the model's own confidence margin says it might be wrong.
01The headroom finding
The baseline model produces a full distribution over answer spans, but only the top one is ever used. We extracted the top-k candidates and computed oracle scores: with k=5, the gold span appears in the candidates for 95.1% of questions, and oracle exact match jumps from 84.28 to 95.11 (F1 from 90.93 to 97.08). The model wasn't failing to find answers, it was failing to rank them first.
The margin between the top two candidate scores turned out to be the signal. When the gold span is ranked first, the median margin is about 0.60; when the gold span sits at rank two, it collapses to about 0.11. Low margin means the model itself suspects it might be wrong.
02Spend compute only where the model is unsure
The reranking rule: if the top-two margin exceeds a threshold τ, keep the top answer untouched; below it, invoke a bi-encoder (all-MiniLM-L6-v2) that rescores both candidates by cosine similarity with the question, interpolated with the baseline scores at α = 0.5. Confident predictions cost nothing extra.
The final system changed just 55 of 10,570 dev predictions, 14 became correct, 1 broke, for +0.12 EM and +0.11 F1 at near-zero marginal compute. We also ran the controls that make the result meaningful: global reranking without a margin trigger consistently hurts, reranking over top-3 or top-5 candidates hurts (more noise, no more signal), and a cross-encoder cost more without consistently winning. Negative results you can explain are worth more than a clean-looking table.