A structured dataset with prompts, artifacts, and case-specific scoring rubrics that measures how well LLMs hold up in realistic complex interactions. Results show that even leading frontier models fail on roughly 25% of items (Gemini 3.1 passes only 78.9%), with performance declining sharply as context grows: pass rates remain above 80% on short artifacts, fall into the low 70s at 2K–10K words, and drop to about 51% beyond 10K words.
Available Now · Sample Available
Long-Context Complex Instructions
Long-context, multi-turn evaluations with mixed media and evolving instructions.
Volume
0
Volume
Scalable Volume
Coverage
0.0
%
Gemini 3.1
Targets
LongBench v2, MultiChallenge, RULER
Modality & Delivery
TextDocumentImage→TextJSONMarkdown
Formats
JSONImages
What You Receive
Coverage
Long-context, multi-turn conversations involving documents, images, mixed media, evolving instructions, reference resolution, synthesis, and structured outputs such as JSON and markdown. Unlike single-turn benchmarks, the dataset is designed to expose failures that emerge over the course of an interaction, including lost-in-the-middle retrieval, grounding drift, instruction forgetting, modality neglect, and structural collapse.
Package Structure
Differentiators
- Multi-turn cases combine prompts, artifacts, mixed media, evolving instructions, and case-specific rubrics.
Record Schema
| Field | Description |
|---|---|
context_refs | Paths to the long-context source documents (text and images) the record draws on. |
distractor_notes | Notes on the planted distractors and contradictions the record tests for. |
instruction | The instruction that requires synthesizing across the full context. |
reference_response | The expert response, grounded in the referenced context. |
Results
| Model | Metric | Result |
|---|---|---|
| Gemini 3.1 | Pass@1 |
Methodology & Quality
Methodology
SME involvement
Domain-expert curated with up to L2 review
Stressors
Lost-in-the-middle retrieval, grounding drift, instruction forgetting, modality neglect, and structured-output collapse.
Design criteria
- Expose failures that emerge over an interaction rather than in one isolated turn.
Build assumptions
Quality approach
- Case-specific scoring rubrics accompany each interaction.
Known limitations
Customization & Scalability
Custom sets available. Additional volume in progress