A collection of 986 SWE-bench-style coding task records from 22 real open-source repositories. Agents diagnose regressions from short bug reports and produce fixes that pass hidden tests.
Available Now · Sample Available
SWE-bench
Leak-hardened SWE-bench-style software-engineering tasks from 22 real open-source repositories in Go, Java, and Python.
Volume
0
Volume
Scalable Volume
Coverage
Go, Java, Python, Coding
Model Benchmark
0.0
%
Claude Opus 4.7 · Pass rate
Targets
SWE-Bench Pro
Modality & Delivery
TextCode→Code
Formats
JSONL (.jsonl)
What You Receive
Coverage
Each record includes repository context, a base commit, a problem statement, reference patch and test patch fields, and fail_to_pass/pass_to_pass test lists for task validation.
Package Structure
Differentiators
- Leak-hardened tasks derived from real open-source repositories.
- Harbor-compatible task trees with hidden-test validation.
Record Schema
| Field | Description |
|---|---|
base_commit | Repository commit used as the task starting point. |
fail_to_pass | Target tests expected to fail before the fix and pass after the correct patch. |
instance_id | Stable task identifier for one SWE-bench task record. |
pass_to_pass | Regression tests expected to pass before and after the fix. |
patch | Reference code patch associated with the task. |
problem_statement | Natural-language issue or task description. |
problem_statement_variants | Optional standard/raw problem-statement variants when both source files exist. |
repo | Repository associated with the task. |
repo_language | Programming language supplied by the source record when available. |
test_patch | Test patch associated with the task. |
Results
| Model | Metric | Result |
|---|---|---|
| Claude Opus 4.7 | Pass rate |
Methodology & Quality
Methodology
SME involvement
Domain-expert curated with up to L2 review
Stressors
Repository navigation, defect localization, patch generation, and regression preservation.
Design criteria
- Require agents to diagnose a regression from a short bug report and produce a patch that passes hidden tests.
Build assumptions
Quality approach
- Hidden fail-to-pass and pass-to-pass tests support reproducible grading.
Known limitations
Customization & Scalability
Custom sets available. Additional volume in progress