2026 · Innodata Research
Long Context LLM Benchmark for Complex AI Interactions
Preliminary results
Innodata · June 22, 2026 · Caitlin Cisar & Jonathan Steuck
When a frontier model fails in the real world, it rarely fails on a single-turn prompt with a well-defined intent. It fails midway through a long conversation — forgetting an instruction from three turns ago, answering from memory when the document is right there.
Innodata’s Long Context & Complex Interaction (LCCI) Benchmark is a structured evaluation dataset built to surface those failure modes: hand-authored multi-turn cases, multi-modal attachments, and a multi-judge panel grading every response against a rubric written for that exact case.
Below: what’s in it, how it’s built, and what the results show.
Cases Compared Head-to-Head
scored by every evaluated model
Components Scored
individual binary rubric items judged
Models Evaluated
frontier models scored to date
Case Pass Rate
share of cases passing overall
01 · The Gap
We know a model failure when we see one. Benchmarks don't.
Model failure is famously hard to define — until you stop defining it from the model’s side and start defining it from the user’s. A user doesn’t experience an average score. They experience a response, and they experience it on a magnitude: some things in a response can be wrong and the user still walks away satisfied; other slips are egregious enough that the entire conversation is a bust.
The model side of the same picture looks symmetric. What the model is contending with is rarely a clean, single-turn prompt with a well-defined intent. It’s messy: user needs that evolve mid-conversation and arrive ambiguous or vague, critical details buried inside long documents or images, instructions revised, contradicted, or clarified turn over turn — often all at once. Against that input, “did the model mess up” is itself a magnitude question: a small slip on a tangential detail and a load-bearing miss on the user’s actual ask are not the same failure, even when a flat score treats them as such.
That’s the gap. Most benchmarks aren’t built around the experience of using the model — they’re built around what’s easy to score in a single shot. A composite number rolls a near-perfect response and a catastrophic one into the same average, and the magnitude disappears from both sides at once. LCCI is built the other way around: every case carries its own rubric of binary pass/fail components, so we can distinguish a response that missed one detail from a response that missed the point.
Where existing benchmarks stop
The interactions LCCI is designed for don’t fit cleanly inside any one existing benchmark. Long-context suites stress the token-window but stay single-turn and text-only. Conversational suites cover dialogue, but lightly — capped turn counts, no attachments, no controlled complexity gradient. Multi-modal suites add one extra input channel and stop there.
LCCI is the first benchmark we’re aware of that pressure-tests all three dimensions together — multi-turn, long-context, and multi-modal — under a controlled, case-specific rubric.
The matrix below shows where each one draws the line.
Two columns of the matrix matter most for the gap we just described. Case-specific rubrics is what makes magnitude legible: when criteria are written for the case in front of the model, a miss is a miss against that case’s own goals, not against a generic checklist that may not apply. Multi-judge with arbitration is what keeps magnitude trustworthy: two frontier-model judges score every response and a third arbitrates disagreement, so the binary verdicts the rest of this dashboard rests on aren’t a single judge’s idiosyncrasy.
The benchmark is structured so each case stands on its own — its conversation, attachments, target turn, and rubric all traceable back to a single case ID.
02 · A Look Inside the Data
Every case, hand-built. Every model, graded on the same set.
Sec 01 described what a case is. This section is about how the dataset is built so those cases are useful to compare across. Every LCCI case is authored by a subject-matter expert working from a structured collection guideline — target prompt word count, task type, artifact type, and modality are all assigned up front, so the dataset covers a broad range of interaction types rather than clustering around the easy combinations. There are no trick questions; the goal is to model authentic usage at a controlled difficulty.
The rubric for each case is generated directly from that case’s own conversation, target turn, and attached artifacts, which is what makes the benchmark hard to game: a model cannot pass by producing fluent, high-quality text that misses the point. And because each case — conversation, attachments, target turn, rubric — is auditable back to a single case ID, every number on this page can be traced to the specific cases that produced it. The figures below cover only the cases that have been scored by every evaluated model, so cross-model comparisons stay apples-to-apples.
Sample Conversations
Five real cases from the benchmark, picked to show how the three leaderboard models — Claude Opus 4.6 , GPT-5.4 mini , and Gemini 3.1 Pro Preview — can diverge on the same conversation. Each tab covers a different verdict pattern: one where only Claude Opus 4.6 passes, one where only GPT-5.4 mini passes, one where only Gemini 3.1 Pro Preview passes, one where all three pass, and one where all three fail. Turn lengths span Single, 2, 6, 8, and 10 so the case shape varies too. The shown thread is the response from the model the example highlights; the verdict badges at the top of each pane summarize how every model scored. The target turn(s) — the ones the rubric grades — are highlighted.
Sample Rubric
Each LCCI case has its own rubric, generated automatically from the case’s own conversation, target turn, and attached artifacts. A typical rubric has 10–20 turn-level components plus 11 standardized Overall Performance items, each one a binary pass/fail check.
Components are coded T1, T2, … for turn-level items and O1, O2, … for conversation-level Overall Performance items. Click below to open a few representative components from the rubrics for the same five sample cases shown above — each component shows how every model scored that single binary check, so you can see exactly where the case-level pattern came from.
03 · What We're Seeing
What's in a benchmark? It depends on what you're looking at.
“What’s in a name?” Shakespeare asked, and answered: nothing — a rose by any other name smells just as sweet. Benchmarks are not roses. What’s in a benchmark — what it asks, how it slices, where it looks — decides what we believe a model can do. The same — cases scored by the same — models tell three different stories depending on which lens you put in front of them. Below: by rubric category, by context length, and by token spend.
By Rubric Category
s5-lede
Two views of the same data, with two different scoring rules. The component view averages every binary check into a category-level miss rate. The case view is stricter for this chart only: a case clears a category when every one of its components in that category passed — a single miss within the category drops the case from that category’s pass count. (Note: this is tighter than the headline rule the leaderboard uses, where a case passes overall when at least 90% of its rubric components passed.) What you measure changes what you see: categories with component pass rates above 90% can still show case pass rates in the 60s. The aggregate hides the brittleness.
Both numbers matter, for different reasons. The case-level view is the deployment lens: a user does not experience an average component pass rate — they experience whether their specific request worked. The component-level view is the diagnostic lens: each rubric defines the steps a model needs to take, and when a step is missed, the failure’s trajectory is captured. Catastrophic failures can be traced back to the point of origin, step-by-step.
Case Pass Rate by Rubric Category
Each row is a high-level rubric category. Component fail rate is the share of rubric components in that category that the panel marked Fail across all evaluated responses to date. Higher percentages indicate categories where models are more likely to slip.
Same categories, different lens: a case clears a category only when every component in that category passed — a single miss within the category drops the case from that category's pass count. (Stricter than the overall ≥90% rule used by the leaderboard.) That's why component pass rates above 90% can still leave per-category case pass rates in the 60s. What you measure changes what you see.
By Context Length
As the word count of user turns and artifacts increases, our hypothesis is that case pass rates should fall. While we observe this tendency most markedly with the lightweight model GPT-5.4 mini, larger models perform more consistently. In contexts up to 50K, pass rates stay above 50% for all 3 models but quickly fall for GPT-5.4 mini as the context length approaches 100K, the current upper bound in our dataset. But the headline number is only half the story. Long context doesn’t just lower scores; it shifts what kind of mistakes models make. The data shows that Artifact Comprehension and Source Fidelity, or how well the model interprets and grounds a response in an artifact, almost always accounts for the largest proportion of failures, irrespective of length. However, the distribution of other failure modes shifts: Reasoning, Synthesis, and Problem Solving tends to account for a larger proportion as context length expands. Long context isn’t a harder version of short context; combined with multi-turn dialogue, it makes specific failure modes the dominant risk. Tab between the pass-rate curve and the failure-mode mix to see both.
Pass Rate by Context Length
Case-level pass rate, plotted against the artifact word-count floor for each band. Hover a point for sample size.
Among failed cases only , what share of the failed rubric components fall in each category — bucketed by context-length band. The mix shifts as the artifact grows: certain failure modes dominate at long context that barely register at short.
By Tokens
The most counterintuitive view. The intuition runs like this: thinking tokens are reasoning made visible, more of them means a model is working a problem harder — so hard cases should burn more thinking tokens, and that burn is the model earning its answer. The data tells a different story. Thinking-token spend is the strongest negative predictor of whether a case passes — across all three models, with the effect significant after controlling for prompt length, turn count, and context band. More reasoning effort doesn’t rescue a hard case; it tracks one. The model isn’t working harder and succeeding — it’s working harder and failing. Two cases with the same prompt length, the same turn count, and the same context band can produce wildly different thinking-token counts, and the longer trace is the one more likely to fail.
Prompt length tells a quieter version of the same story. The binned curves show degradation as prompts grow, but once you control for what kind of work a long prompt represents, raw prompt size is no longer a significant predictor on its own — its damage is already captured by the context band. Token spend, in either direction, is a signal of difficulty, not capability. A model whose thinking-token spend is climbing on a particular workload is flagging the data that needs a second look — whether or not its top-line leaderboard score has moved.
Pass Rate vs Thinking Tokens — Binned
Cases with thinking-tokens > 0 binned into deciles per model. Reveals diminishing returns — and where extra reasoning tokens stop buying additional pass rate.
All cases binned into deciles of input prompt tokens per model. Tests whether long-context cases degrade pass rate as the prompt grows.
Standardized coefficients with 95% confidence intervals. Each predictor is z-scored so coefficients are directly comparable. Positive = predictor pushes pass probability up; negative = down. Bars crossing the dashed zero line are not significant. * p<0.05 ** p<0.01 *** p<0.001 — significance after standardizing predictors.
Same model as the previous tab, extended with one-hot indicators for task type and input modality . Each task / modality coefficient is read relative to the most-common level for that dimension (shown in the predictor label). Coefficients of token and conversation predictors shift slightly because they now control for the added categorical structure. Sparse levels (fewer than 8 cases for a provider) are dropped to keep confidence intervals interpretable; levels with wide CIs that span zero are not distinguishable from the baseline at this sample size. * p<0.05 ** p<0.01 *** p<0.001
04 · Preliminary Leaderboard
Which model holds up best under sustained complexity?
s6-lede
Case Pass Rate by Model — Preliminary
A case passes when the panel of judges finds the response satisfies at least 90% of its rubric components. Component pass rate is the share of all individual rubric items the model satisfied across its evaluated cases. Each response is scored by a two-judge panel of frontier models, with a third model arbitrating any disagreements. Final rankings will shift as more cases are evaluated.
05 · Why This Matters For Your Models
Aggregate scores tell you whether a model is good. LCCI tells you where it breaks.
Aggregate scores can tell you whether a model is good. A model that scores well on standard benchmarks may still exhibit grounding drift on long documents, lose instruction constraints in conversations beyond three turns, or consume disproportionate compute on the cases that matter most for a given workload. Without a controlled framework that isolates these failure modes, those weaknesses are difficult to see until production exposes them.
LCCI gives you 15 cross-model comparison metrics and 25 per-model diagnostic metrics, sliced by failure mode, task type, context length, input modality, and conversation depth. That granularity supports deployment decisions a single composite score never could — whether a model’s weaknesses are concentrated in one specific failure mode, whether they appear only at certain context lengths, or whether a competitor is meaningfully stronger on the task types that matter for your use case.
Document-heavy workflows have different risk profiles than multi-turn agents or structured data extraction pipelines. LCCI’s metadata-tagged cases let you filter to the interaction types that match your environment, rather than evaluating against a generic distribution.
Want the full picture?
LCCI gives evaluation teams a controlled framework for measuring how models behave under long context, multi-turn interaction, and multi-modal inputs — not just whether they score well on average. Read the deep-dive in our blog post, or browse more of our work on LLM evaluations.