V.03 · catalog/software-engineering
Software Engineering. Three engineering workflows, one simulated software organization.
Each environment runs on EngineeringWorld, a simulated software organization with repositories, specifications, CI, deployments and production services. Agents write and test code, run incidents and review pull requests, and are graded by independent oracles, never by the tests they wrote themselves.
The environments share repositories, specifications and defects, so work can cross them: a change written in SpecForge can be reviewed in ReviewGuard and fail in production in IncidentLab. Each episode still starts from its own isolated snapshot.
Why it matters. Where today's software agents fall short, from published research.
Published research points to three gaps: tests that can't be trusted, diagnosis that stops short of recovery, and reviews that miss bugs outside the diff. Each environment targets one of them.
SWE-bench Pro audit
~30%
of SWE-bench Pro tasks estimated broken: overly strict tests, low-coverage tests, and underspecified or misleading prompts.
Human annotation found 249 of 731 tasks broken. Scale's SWE-bench Pro V2 (September 2026) dropped 89 invalid tasks, so the figure applies to the original version.
OpenAI · July 2026 (opens in a new tab)TestSuiteBench
<60%
mutation scores for test suites that frontier coding agents wrote from a specification.
88% of the agents' tests passed on the reference implementation: the suites were mostly valid but missed many faults.
ETH Zürich SRI Lab · October 2026 (opens in a new tab)ITBench-AA SRE
47%
best score for identifying the root causes of Kubernetes incidents. Every frontier model tested scored below 50%.
Diagnosis from offline snapshots only. The benchmark doesn't measure restoring service.
IBM and Artificial Analysis · May 2026 (opens in a new tab)CRBugBench
71.9%
of human-verified bugs caught by the best review agent within its top five comments, on open-source repositories.
Performance dropped by up to 25.5% on proprietary industry repositories, where bugs more often need project context.
Microsoft · October 2026 (opens in a new tab)
Environments. Each runs on EngineeringWorld, and each has its own datasheet.
- ENV-SWE-101Custom build
01SpecForge
Coding & Test Engineering
A reproducible development environment where agents implement specifications, write tests, diagnose test failures and evolve code safely. A failing test doesn't always mean the code is wrong, and a passing test doesn't always mean it's right.
Read the datasheetCore competency
Tell a defect in the code from a defect in the test, and write tests that match the behavioural contract.
- ENV-SWE-102Custom build
02IncidentLab
Site Reliability Engineering
A reproducible production environment where agents diagnose incidents, isolate root causes, choose remediations and restore service safely, working from metrics, logs, traces, deployments and incident policy. Diagnosis is only half the job: the grade depends on whether service actually recovers.
Read the datasheetCore competency
Restore production safely and prove it stayed restored, not just name a plausible cause.
- ENV-SWE-103Custom build
03ReviewGuard
Code Review & Regression Prevention
A code-review environment with pull requests, repository context, tests, specifications and independently verified defects. Agents are graded on finding the bugs that matter, backing each finding with evidence, and staying silent when a change is correct.
Read the datasheetCore competency
Find the real defects in a change, prove them, and leave correct changes alone.
EngineeringWorld. One world spec, shared by every Software Engineering environment.
- type:
- Executable RL training and evaluation environment.
- world:
- EngineeringWorld: one simulated software organization with versioned repositories, pull requests, specifications, CI, deployments and production services, shared by every environment in this vertical.
- oracle:
- Independent reference implementations, verified faulty implementations and mutants, and hidden behavioural checks the agent can't see. Agent-written tests never grade the agent's own work.
- systems:
- Repository registry, development sandbox, specification registry, deployment and incident simulators, observability, review service and trace recorder.
- modes:
- RL training and evaluation, plus SFT trajectories and preference pairs on request.
- interaction:
- Stateful, multistep tool use with real code and test execution, over a simulated clock for production tasks.
- state:
- Resettable repository and system snapshots. The same seed and version reproduce the same starting state.
- tools:
- JSON-schema tools, with an optional MCP adapter and adapters for common RL training frameworks.
- isolation:
- Every agent workload runs in an isolated, resource-limited sandbox with no host credentials and no access to real infrastructure.
- verification:
- Program behaviour, test soundness, system recovery, review findings and policy constraints, checked independently. Multiple valid solutions are accepted.
- splits:
- Held-out repositories, scenario generators and defects, not just held-out examples.
- engagement:
- Custom environment build and accepted task instances, with optional ongoing expansion.
- licensing:
- Commercial use rights. Exclusivity, ownership and rights to any customer code are agreed for each engagement.
Datasets. Software Engineering data you can license without a full environment.
Each dataset is a separate deliverable. The matrix shows what each one contains, so you can tell a set of task descriptions from solved demonstrations, preference data, executable validators or complete environments. Every dataset is built to order.
| Dataset | Primary use | Task descriptions | Solved demonstrations | Preference annotations | Executable validators | Complete environments |
|---|---|---|---|---|---|---|
| DATA-SWE-001Contract-faithful coding tasksSpecifications, repository snapshots, reference implementations and validators.from ENV-SWE-101 | Coding RL, Evaluation | Included | Included | Not included | Included | Not included |
| DATA-SWE-002Test soundness and overconstraint datasetGenerated tests, conforming implementations, verified defects and mutants, and test repairs, with requirement-to-assertion mappings and overconstraint labels.from ENV-SWE-101 | Test-generation RL, TDD training | Included | Included | Not included | Included | Not included |
| DATA-SWE-003Red–green–refactor trajectoriesChronological expert coding and testing episodes.from ENV-SWE-101 | SFT, Imitation learning | Included | Included | Not included | Not included | Not included |
| DATA-SWE-004Incident investigation and recovery tracesAlerts, telemetry, root causes, remediations and recovery state.from ENV-SWE-102 | SRE RL, SFT, Evaluation | Included | Included | Not included | Not included | Not included |
| DATA-SWE-005Expert code review datasetPull requests, verified bugs, clean changes, findings and severities.from ENV-SWE-103 | Review post-training | Included | Included | Not included | Not included | Not included |
| DATA-SWE-006Reviewer false-alarm and correction datasetUnsupported review comments, reasons for rejection and corrected reviews.from ENV-SWE-103 | Preference optimization, Review precision | Included | Included | Included | Not included | Not included |
| DATA-SWE-007Agent failure and recovery trajectoriesFailed code changes, broken tests, incident misdiagnoses and corrected actions.from ENV-SWE-101…103 runs | Robustness training | Included | Included | Included | Not included | Not included |
- DATA-SWE-001Contract-faithful coding tasksSpecifications, repository snapshots, reference implementations and validators.from ENV-SWE-101
Primary use · Coding RL, Evaluation
- IncludedTask descriptions
- IncludedSolved demonstrations
- Not includedPreference annotations
- IncludedExecutable validators
- Not includedComplete environments
- DATA-SWE-002Test soundness and overconstraint datasetGenerated tests, conforming implementations, verified defects and mutants, and test repairs, with requirement-to-assertion mappings and overconstraint labels.from ENV-SWE-101
Primary use · Test-generation RL, TDD training
- IncludedTask descriptions
- IncludedSolved demonstrations
- Not includedPreference annotations
- IncludedExecutable validators
- Not includedComplete environments
- DATA-SWE-003Red–green–refactor trajectoriesChronological expert coding and testing episodes.from ENV-SWE-101
Primary use · SFT, Imitation learning
- IncludedTask descriptions
- IncludedSolved demonstrations
- Not includedPreference annotations
- Not includedExecutable validators
- Not includedComplete environments
- DATA-SWE-004Incident investigation and recovery tracesAlerts, telemetry, root causes, remediations and recovery state.from ENV-SWE-102
Primary use · SRE RL, SFT, Evaluation
- IncludedTask descriptions
- IncludedSolved demonstrations
- Not includedPreference annotations
- Not includedExecutable validators
- Not includedComplete environments
- DATA-SWE-005Expert code review datasetPull requests, verified bugs, clean changes, findings and severities.from ENV-SWE-103
Primary use · Review post-training
- IncludedTask descriptions
- IncludedSolved demonstrations
- Not includedPreference annotations
- Not includedExecutable validators
- Not includedComplete environments
- DATA-SWE-006Reviewer false-alarm and correction datasetUnsupported review comments, reasons for rejection and corrected reviews.from ENV-SWE-103
Primary use · Preference optimization, Review precision
- IncludedTask descriptions
- IncludedSolved demonstrations
- IncludedPreference annotations
- Not includedExecutable validators
- Not includedComplete environments
- DATA-SWE-007Agent failure and recovery trajectoriesFailed code changes, broken tests, incident misdiagnoses and corrected actions.from ENV-SWE-101…103 runs
Primary use · Robustness training
- IncludedTask descriptions
- IncludedSolved demonstrations
- IncludedPreference annotations
- Not includedExecutable validators
- Not includedComplete environments