fever (fact extraction and verification)

FEVER (Fact Extraction and VERification) is a large-scale benchmark dataset and shared task for evaluating automated fact-checking systems. It is the most widely used benchmark for systems that verify claims against textual evidence.

Dataset Structure

• 185,445 claims generated by altering sentences from Wikipedia, then manually verified by annotators.
• Evidence: The knowledge source is the full English Wikipedia (~5.4 million articles at time of creation).
• Labels: Each claim is labeled as:
• SUPPORTED: Evidence in Wikipedia confirms the claim.
• REFUTED: Evidence in Wikipedia contradicts the claim.
• NOT ENOUGH INFO (NEI): Wikipedia doesn't contain sufficient evidence to verify or refute.

The FEVER Task

• Step 1 — Document Retrieval: Given a claim, identify relevant Wikipedia documents.
• Step 2 — Sentence Selection: From retrieved documents, select the specific sentences that serve as evidence.
• Step 3 — Claim Verification: Using the selected evidence, classify the claim as SUPPORTED, REFUTED, or NEI.
• Evaluation Metric: FEVER Score — a claim is correctly verified only if both the label is correct AND the evidence sentences are correct (for SUPPORTED/REFUTED claims).

Why FEVER Matters

• Standard Benchmark: Nearly all automated fact-checking papers evaluate on FEVER, enabling direct comparison.
• Full Pipeline Evaluation: Tests the complete fact-checking pipeline, not just individual components.
• Research Impact: Has driven significant advances in evidence retrieval and natural language inference.

FEVER Shared Tasks

• FEVER 1.0 (2018): First shared task. Winning systems used TF-IDF retrieval + BERT-based NLI.
• FEVER 2.0 (2019): Added adversarial claim generation to test system robustness.
• Subsequent Work: Extensions like symmetric FEVER, multi-lingual FEVER, and FEVER with structured evidence.

State-of-the-Art Performance

• Top systems achieve ~80–85% FEVER Score, leaving significant room for improvement.
• The hardest cases involve multi-hop reasoning (requiring evidence from multiple sources) and NEI classification (distinguishing "not enough info" from "refuted").

Limitations

• Wikipedia Only: Real-world fact-checking requires evidence from diverse sources beyond Wikipedia.
• Synthetic Claims: Claims were generated by altering Wikipedia sentences, which may not reflect natural misinformation patterns.
• Temporal: Based on a Wikipedia snapshot — doesn't capture evolving knowledge.

FEVER is the foundational benchmark for automated fact-checking research — it established the standard evaluation framework that the field continues to build upon.