Synthetic visual-reasoning multiple-choice items across 3 reasoning categories. When evaluated, top models scored between 23–79% accuracy. Aligned to the VisuLogic benchmark.
Available on Request · Sample Available
Visual Reasoning
Synthetic visual-reasoning items engineered so language-only shortcuts fail.
Volume
0
Volume
Scalable Volume
Coverage
Visual Reasoning
0.0
%
Frontier Model
Targets
VisuLogic
Modality & Delivery
ImageText→Text
Formats
JSONImages
What You Receive
Coverage
Ground truth is known by construction (no manual labelling); every option panel is pixel-distinct. Several categories plant a chiral (mirror-image) distractor that captions identically to the true answer, so items resist language-only shortcuts and require genuine visual reasoning.
Package Structure
Differentiators
- Ground truth is known by construction and answer panels are pixel-distinct.
- Chiral distractors can caption identically to the correct answer, reducing language-only shortcuts.
Record Schema
| Field | Description |
|---|---|
answer | The concise final answer associated with the question. |
image_ref | Path to the associated image file referenced by the record. |
question | The question or task presented in the dataset record. |
solution | The worked solution or supporting reasoning for the question. |
Results
| Model | Metric | Result |
|---|---|---|
| Frontier Model | Pass@1 |
Methodology & Quality
Methodology
SME involvement
Domain-expert curated with up to L2 review
Stressors
Mirror-image distractors and visually similar option panels.
Design criteria
- Require visual reasoning that cannot be solved from captions alone.
Build assumptions
Quality approach
- Construction-derived ground truth and pixel-distinct option validation.
Known limitations
Customization & Scalability
Custom sets available. Additional volume in progress