The question our customers keep asking
If your product depends on a model reading a document, inspecting a part, or interpreting a medical scan, you need to know: can it actually see the image, or is it guessing from context?
That question led us to VisuLogic, a benchmark published in April 2025 by researchers at the University of Science and Technology of China (Wu, et al., 2025). Across 1,000 visual-reasoning problems, leading multimodal models scored below 30%, barely above the 25% random-guess floor. Humans reached 51.4%.
The core finding: models often convert images into text descriptions and reason over them. That shortcut breaks when visual details can’t be captured in language, causing models to fail silently behind fluent, confident responses.
At Innodata, we took this as a design challenge and built our own testing harness: not another static benchmark, but a generator that lets us probe exactly where and why models fail.
Why we built a generator, not a benchmark
Static benchmarks provide consistent evaluation, but fixed problem sets are vulnerable to contamination. We built a seed-driven generator that produces unlimited, structurally equivalent problems across six categories: rotation, reflection, matrix, counting, spatial, and analogy. One command generates reproducible datasets at any scale and difficulty.
We also enforce integrity by construction. Correct answers are evenly distributed across positions to prevent placement bias. Analogy problems exclude trivial 180° rotations, reflection problems require visually distinct transformations, and counting answers are computed directly from the generated shapes to ensure they match the image.
The chirality trap: making the wrong answer sound right
The heart of the design is chirality: the property that makes an object different from its mirror image. Like your left hand, which can never be rotated to match your right. Our test images use an asymmetric flag-on-a-pole motif whose reflection is distinct from every rotation.
Both panels satisfy the same text description, yet they are visually distinct mirror images.
Both the original and its mirror satisfy the same text description: “a flag pointing up-left with a dot at the lower right.” A model reasoning from a caption rather than pixels cannot tell them apart. In other words, the same text description has multiple visual interpretations.
Each rotation, reflection, and analogy problem includes a mirror twin as its first and most plausible distractor. Here is a generated analogy item: A is correct, while B is its mirror.
A real analogy item from our generator. The correct answer (A) and its mirror twin (B) are indistinguishable by text description alone.
On reflection problems, these models performed remarkably well. Yet when those same mirror twins appeared in analogy or rotation problems, even frontier models struggled. The best model scored just 56% on analogy.
How today's models performed
We evaluated seven models on 150 items, 25 per category, with five choices each (20% chance). Five were frontier API models: Gemini 3.1 Pro, Claude Opus 4.8, Gemini 3 Flash, GPT-5.4, and GPT-4o. Two were open-weight: Qwen2.5-VL-3B and Nemotron-Nano-VL-8B.
Bar chart showing accuracy by model across 6 categories of 25 visual reasoning questions.
The good news: the latest frontier models are genuinely getting better at understanding visual information, not just describing. Gemini, Claude, and OpenAI’s latest models scored 0.70–0.79 overall, far above chance.
The bad news: analogy and rotation remain weak points. The best scores in each were 0.56, exactly where our chirality trap bites hardest. GPT-4o, a 2024-era model, scored near chance on several categories, highlighting the gains in newer models.
Per-category accuracy across frontier models. Green is strong; red is near or below chance.
The open-weight models both scored 0.28 overall, barely above chance and consistent with VisuLogic’s results (Wu, et al., 2025). Matrix problems revealed both a bright spot and a flaw: even the 3B model can infer the rule, with columns determining shape and rows determining count, but struggles to count the objects needed to distinguish A, C, and E in the figure below. One hypothesis is that reasoning through the pattern leaves less capacity for processing visual details.
Shape is fixed by column and count by row, so the missing cell must be four diamonds (A). The distractors are deliberately close: C is five and E is three, while B has the right count in the wrong shape.
One caveat: our results are not directly comparable to VisuLogic’s sub-30% scores. Our different chance floor and cleaner, constructed distractors may make our items easier. Both results measure different things: ours isolates specific failure modes at scale, while theirs captures aggregate difficulty on naturally messy problems.
Abstract Reasoning vs. Visual Perception: the key distinction
Benchmarking surfaced two distinct capabilities: can a model reason over the underlying structure in an image, and can it accurately perceive the visual details needed to apply that reasoning?
We gave Gemini 3.1 Pro, our best performing model, the following rotation problem:
Example of a rotation problem.
It correctly identified the pattern, noting that “Options A and E both point in the correct general direction.” Yet it chose E, arguing that A was “distorted” while E preserved the original symmetry. It failed to notice that A and E differed only in the flag’s orientation, selecting the distractor instead of A.
This wasn’t a one-off hallucination, but a consistent trend across top models. We analyzed each model’s errors on rotation and analogy problems and measured how often the model selected the distractor. Gemini 3.1 Pro chose the distractor in 85.7% of its errors, followed by Claude Opus 4.8 at 72%, Gemini 3 Flash at 61.3%, and GPT-5.4 at 58.1%. Two findings stand out. First, even models with perfect reflection scores struggle when reflection is not the underlying pattern. Second, models that fall for the trap more often are actually better at identifying the key pattern. They can recognize an abstract rule, such as a rotation or analogy, but struggle to incorporate the visual details needed to select the correct answer. Visual reasoning is therefore not a single capability, but a combination of skills that can fail independently.
Chain-of-thought (CoT) sharpened this distinction. On Qwen2.5-VL-3B, explicit CoT reduced accuracy from 0.267 to 0.162. The model reasons fluently but misperceives, giving it more room to reason confidently from an incorrect visual read.
What this means and what's next
Three things stood out. Counting, spatial composition, and reflection are largely solved for frontier models, with most scoring above 0.92. Chirality remains an open problem, while the gap between frontier API and smaller open-weight models is dramatic, even when smaller models show real abstract-reasoning ability.
The direction we find most promising is treating each failure mode as a dial. Chirality is currently binary: mirror or not. Make it continuous, and you can find the angular threshold where a model confuses rotation with reflection. Extend corruption across categories, and you can build a fuller perception-versus-reasoning profile for each capability.
That’s the thesis worth carrying forward: a benchmark tells you how much a model fails; a generator tells you why. If your team is evaluating whether your model can actually understand visual data, or trying to teach it to do so, we’d like to have that conversation.
Works Cited
Xu, Weiye, et al. “VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models.” arXiv, 2025, https://doi.org/10.48550/arxiv.2504.15279