A collection of 986 SWE-bench-style coding task records from 22 real open-source repositories. Agents diagnose regressions from short bug reports and produce fixes that pass hidden tests.
Available Now · Sample Available
SWE-bench
Leak-hardened SWE-bench-style software-engineering tasks from 22 real open-source repositories in Go, Java, and Python.
Volume
0
Volume
Scalable Volume
Coverage
Go, Java, Python, Coding
48.2%Claude Opus 4.7 · Pass rate
Targets
SWE-Bench Pro
Modality & Delivery
TextCode→Code
Formats
JSONL (.jsonl)
What You Receive
Coverage
Each record includes repository context, a base commit, a problem statement, reference patch and test patch fields, and fail_to_pass/pass_to_pass test lists for task validation.
Package Structure
Differentiators
- Leak-hardened tasks derived from real open-source repositories.
- Harbor-compatible task trees with hidden-test validation.
Record Schema
base_commitstringRepository commit used as the task starting point.
fail_to_passarrayTarget tests expected to fail before the fix and pass after the correct patch.
instance_idstringStable task identifier for one SWE-bench task record.
pass_to_passarrayRegression tests expected to pass before and after the fix.
patchstringReference code patch associated with the task.
problem_statementstringNatural-language issue or task description.
problem_statement_variantsarrayOptional standard/raw problem-statement variants when both source files exist.
repostringRepository associated with the task.
repo_languagestringProgramming language supplied by the source record when available.
test_patchstringTest patch associated with the task.
Results
| Model | Metric | Result |
|---|---|---|
| Claude Opus 4.7 | Pass rate |
Methodology & Quality
Methodology
SME involvement
Domain-expert curated with up to L2 review
Stressors
Repository navigation, defect localization, patch generation, and regression preservation.
Design criteria
- Require agents to diagnose a regression from a short bug report and produce a patch that passes hidden tests.
Build assumptions
Quality approach
- Hidden fail-to-pass and pass-to-pass tests support reproducible grading.
Known limitations
Customization & Scalability
Custom sets available. Additional volume in progress
Get Started
Request a Sample
Share your target domains and the OTS team will follow up with sample data and next steps.