hellaswag

HellaSwag is a dataset for commonsense natural language inference (NLI) that asks the model to complete a sentence describing a physical situation or event — constructed using Adversarial Filtering to ensure the correct ending is difficult for BERT-like models to guess based on distribution alone.

Task

• Context: "A woman is outside with a bucket and a dog. The dog is running around trying to avoid a bath. She..."
• Ending A: "...rinses the bucket."
• Ending B: "...grabs the dog and washes it." (Correct).
• Ending C: "...gets in the bucket herself."

Why It Matters

• ActivityNet: Derived from video captions — focuses on grounded, temporal events.
• Adversarial: Specifically designed to break BERT; endings that "sounded right" to BERT but were nonsensical to humans were generated as distractors.
• LLM Benchmark: Remains a standard score reported for all new Foundation Models (GPT-3, LLaMA).

HellaSwag is predicting the next scene — testing if the model understands how physical events and human actions typically unfold.