Long-horizon software-engineering tasks requiring substantial new features to be implemented across multiple layers of real open-source repositories.
Available on Request · Sample Available
SWE-Bench — Long Horizon
Long-horizon engineering tasks: substantial new features built across multiple layers of real repositories.
Volume
0
Volume
Scalable Volume
Coverage
Rust, Python, C++, TypeScript
0.0
%
Opus 4.8
Targets
SWE-Evo
Modality & Delivery
TextCode→CodeText
Formats
Harbor-compatible task trees
What You Receive
Coverage
Package Structure
- dataset/
- dataset01.csv
- dataset01.json
- dockerfiles/
- base_dockerfile
- instance_dockerfile
- results/
- groundtruth.json
- attempt1.json
- attempt2.json
- attempt3.json
- scripts/
- script.py
Differentiators
- Long-horizon work spanning multiple layers of real repositories.
- Requires substantial multi-file implementations rather than localized patches.
Record Schema
Results
| Model | Metric | Result |
|---|---|---|
| Opus 4.8 | Pass@5 |
Methodology & Quality
Methodology
SME involvement
Domain-expert curated with up to L2 review
Stressors
Roughly 500–950 lines of implementation work plus extensive tests in the five-task sample.
Design criteria
- Each task includes a written specification, upstream base commit, container setup, graded tests, and reproducible results.
Build assumptions
Quality approach
- Containerized execution with graded tests and reproducible task results.
Known limitations
Customization & Scalability
Custom sets available. Additional volume in progress