Synthetic visual-reasoning multiple-choice items across 3 reasoning categories. When evaluated, top models scored between 23–79% accuracy. Aligned to the VisuLogic benchmark.
Available on Request · Sample Available
Visual Reasoning
Synthetic visual-reasoning items engineered so language-only shortcuts fail.
Volume
0
Volume
Scalable Volume
Coverage
Visual Reasoning
23%Frontier Model · Pass@1
Targets
VisuLogic
Modality & Delivery
ImageText→Text
Formats
JSONImages
What You Receive
Coverage
Ground truth is known by construction (no manual labelling); every option panel is pixel-distinct. Several categories plant a chiral (mirror-image) distractor that captions identically to the true answer, so items resist language-only shortcuts and require genuine visual reasoning.
Package Structure
Differentiators
- Ground truth is known by construction and answer panels are pixel-distinct.
- Chiral distractors can caption identically to the correct answer, reducing language-only shortcuts.
Record Schema
answerstringThe concise final answer associated with the question.
image_refstringPath to the associated image file referenced by the record.
questionstringThe question or task presented in the dataset record.
solutionstringThe worked solution or supporting reasoning for the question.
Results
| Model | Metric | Result |
|---|---|---|
| Frontier Model | Pass@1 |
Methodology & Quality
Methodology
SME involvement
Domain-expert curated with up to L2 review
Stressors
Mirror-image distractors and visually similar option panels.
Design criteria
- Require visual reasoning that cannot be solved from captions alone.
Build assumptions
Quality approach
- Construction-derived ground truth and pixel-distinct option validation.
Known limitations
Customization & Scalability
Custom sets available. Additional volume in progress
Get Started
Request a Sample
Share your target domains and the OTS team will follow up with sample data and next steps.