COLLIE

reasoning

COLLIE is a grammar-based framework for systematic construction of constrained text generation tasks. It allows specification of rich, compositional constraints across diverse generation levels and modeling challenges including language understanding, logical reasoning, and semantic planning. The COLLIE-v1 dataset contains 2,080 instances across 13 constraint structures.

Leaderboard

Showing 10 of 10 results

GPT-5

99.0%

i
o3-mini

98.7%

i
o3

98.4%

i
Mistral Medium 3.5

95.8%

i
GPT-4.5

72.3%

i
GPT-4.1

65.8%

i
Mistral Small 4

62.9%

i
GPT-4o

61.0%

i
GPT-4.1 mini

54.6%

i
GPT-4.1 nano

42.5%

i