Test suites

View docs

Test suites lie at the heart of psycholinguistic evaluation. The items in a test suite are given as input to a language model, and the resulting surprisal values are used to assess the model's performance. Typically, test suites are designed in a way that probes a particular grammatical phenomenon.

Browse the available test suites in the table below, or add a new test suite by creating one interactively or uploading one as a .json file.

Available test suites
Name Reference Added by Models evaluated Average performance
Name Reference Added by Models evaluated Average performance
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
27.96%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
77.30%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
81.25%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
20.39%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
45.83%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
45.31%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
65.10%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
65.10%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
75.00%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
79.17%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
65.46%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
55.26%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
52.30%
James, Syntax. 1956. 'Sytnactic Strucgyms' Syntax James
8 / 8
66.25%
James, Syntax. 1956. 'Sytnactic Strucgyms' Syntax James
8 / 8
58.75%
James, Syntax. 1956. 'Sytnactic Strucgyms' Syntax James
8 / 8
57.50%
James, Syntax. 1956. 'Sytnactic Strucgyms' Syntax James
8 / 8
81.25%
James, Syntax. 1956. 'Sytnactic Strucgyms' Syntax James
8 / 8
78.75%
James, Syntax. 1956. 'Sytnactic Strucgyms' Syntax James
8 / 8
86.25%
James, Syntax. 1956. 'Sytnactic Strucgyms' Syntax James
8 / 8
65.48%
James, Syntax. 1956. 'Sytnactic Strucgyms' Syntax James
8 / 8
53.57%
James, Syntax. 1956. 'Sytnactic Strucgyms' Syntax James
8 / 8
60.12%
James, Syntax. 1956. 'Sytnactic Strucgyms' Syntax James
8 / 8
50.00%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
65.48%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
53.57%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
60.12%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
50.00%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
59.38%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
12.50%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
33.33%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
65.62%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
79.17%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
59.90%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
54.69%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
76.04%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
78.12%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
50.45%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
55.36%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
65.62%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
62.95%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
53.62%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
54.93%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
59.54%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
55.26%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
67.39%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
67.93%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
66.85%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
64.67%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
57.07%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
52.72%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
54.89%
James, Syntax. 1956. 'Syntactic Strucgyms' Syntax James
8 / 8
52.17%
ToDo ToDo
8 / 8
60.94%
ToDo ToDo
8 / 8
73.75%
ToDo ToDo
8 / 8
35.00%
ToDo ToDo
8 / 8
63.12%
ToDo ToDo
8 / 8
51.25%
ToDo ToDo
8 / 8
56.56%
ToDo ToDo
8 / 8
57.50%
ToDo ToDo
8 / 8
60.94%
ToDo ToDo
8 / 8
68.75%
ToDo ToDo
8 / 8
73.30%
ToDo ToDo
8 / 8
71.02%
ToDo ToDo
8 / 8
74.43%
ToDo ToDo
8 / 8
53.98%
ToDo ToDo
8 / 8
56.82%
ToDo ToDo
8 / 8
59.69%
ToDo ToDo
8 / 8
59.69%
ToDo ToDo
7 / 8
61.61%
ToDo ToDo
7 / 8
61.16%
http://www.glottopedia.org/index.php/That-trace_effect Roger Levy
0 / 8
nan%
Roger Levy
0 / 8
nan%
Roger Levy
0 / 8
nan%
Pickering, M. and Barry, G. (1991). Sentence processing without empty categories. Language and Cognitive Processes, 6(3):229–259 Roger Levy
0 / 8
nan%