winograd schema challenge

Winograd Schema Challenge (WSC) is a commonsense reasoning benchmark consisting of pairs of sentences that differ by only one or two words, containing an ambiguous pronoun whose resolution resorts to world knowledge — designed to be easy for humans but hard for machines.

Example

• Sentence A: "The trophy doesn't fit into the suitcase because it is too large." (It = Trophy).
• Sentence B: "The trophy doesn't fit into the suitcase because it is too small." (It = Suitcase).
• Logic: You must know physics (large things don't fit in small things) to resolve syntax.

Why It Matters

• Turing Test Alternative: Proposed by Hector Levesque as a better test of intelligence than the Turing Test.
• Evaluation: Standard benchmark for LLMs. GPT-4 scores >90%, effectively "solving" it, though smaller models struggle.
• Selectional Restrictions: Tests if models learn the physical/semantic constraints of verbs and adjectives.

Winograd Schema Challenge is the "It" test — a benchmark testing if AI has enough commonsense physics to resolve ambiguous pronouns.