Testing the limits of natural language models for predicting human language judgments.

Tal Golan, Matthew Siegelman, Nikolaus Kriegeskorte, Christopher Baldassano

Research output: Other contribution


Neural network language models can serve as computational hypotheses about how humans process language. We compared the model-human consistency of diverse language models using a novel experimental approach: controversial sentence pairs. For each controversial sentence pair, two language models disagree about which sentence is more likely to occur in natural text. Considering nine language models (including n-gram, recurrent neural networks, and transformer models), we created hundreds of such controversial sentence pairs by either selecting sentences from a corpus or synthetically optimizing sentence pairs to be highly controversial. Human subjects then provided judgments indicating for each pair which of the two sentences is more likely. Controversial sentence pairs proved highly effective at revealing model failures and identifying models that aligned most closely with human judgments. The most human-consistent model tested was GPT-2, although experiments also revealed significant shortcomings of its alignment with human perception.
Original languageEnglish
Number of pages25
StatePublished - 2022


Dive into the research topics of 'Testing the limits of natural language models for predicting human language judgments.'. Together they form a unique fingerprint.

Cite this