Skip to content
ARCHYTASAI

ENV-SWE-101 · Coding & Test Engineering

SpecForge. Tell a defect in the code from a defect in the test, and write tests that match the behavioural contract.

A reproducible development environment where agents implement specifications, write tests, diagnose test failures and evolve code safely. A failing test doesn't always mean the code is wrong, and a passing test doesn't always mean it's right.

Custom buildCode Generation · Agentic Coding · Test-Driven Development

Overview. What the environment trains, and the work it's built around.

Core competency

Tell a defect in the code from a defect in the test, and write tests that match the behavioural contract.

  • Implementing a specification without adding requirements it doesn't state
  • Auditing test suites for false failures and false passes
  • Writing a failing test first, for the intended reason, then the code
  • Updating code and tests when requirements change, without breaking existing contracts

Built around the work of

Software Engineer · QA Engineer · SDET · Coding agent developer

Simulated systems

  • Git repository
  • Package manager
  • Build runner
  • Test framework
  • CI service

Sample task

A generated test suite rejects a valid implementation of an alerts API because it assumes alerts come back in alphabetical order, while the specification leaves order unspecified. Identify which failing assertions are genuine defects and which add requirements absent from the contract, and repair the tests without weakening their ability to detect omitted, duplicated or ineligible alerts.

Specification. The environment, field by field.

spec · env/specforge.yamlENV-SWE-101
product_type:
Executable RL training and evaluation environment
category:
Code Generation · Agentic Coding · Test-Driven Development
commercial_model:
Environment development and validated coding and test task packages
environment_id:
ENV-SWE-101
target_roles:
Software Engineer · QA Engineer · SDET · Coding agent developer
core_competencies:
  • Implementing a specification without adding requirements it doesn't state
  • Auditing test suites for false failures and false passes
  • Writing a failing test first, for the intended reason, then the code
  • Updating code and tests when requirements change, without breaking existing contracts
data:
Synthetic Python and TypeScript repositories with written specifications, several conforming reference implementations per task and verified faulty implementations. Customer repositories can be added where an engagement allows it.
simulated_systems:
Git repository · Package manager · Build runner · Test framework · CI service
interfaces:
Terminal and filesystem · Git · Test runner with mutation analysis · Structured specification API
interaction:
Stateful, multistep tool use in an isolated repository sandbox, with code and test execution.
state:
A resettable repository snapshot with versioned acceptance criteria. Every episode starts from the same commit and seed.
disturbances:
Overconstrained tests, underconstrained tests, flaky or order-dependent tests, mocks tied to internals, and specifications that change mid-task.
task_families:
  • TASK-SWE-101-01 · Specification-faithful code generation
  • TASK-SWE-101-02 · Test suite soundness & overconstraint audit
  • TASK-SWE-101-03 · Red–green–refactor TDD
  • TASK-SWE-101-04 · Requirement evolution & regression preservation
evaluation:
A private oracle the agent can't see: submitted tests run against a set of conforming implementations, which they must accept, and a set of faulty implementations and mutants, which they must reject. Submitted code runs against hidden behavioural checks.
reward:
Hard correctness gates, then test soundness, fault detection, requirement coverage and engineering quality. Agent-written tests are never the only source of reward.
training_use:
Coding RL, test-generation RL, TDD training, SFT demonstrations and held-out evaluation.
deployment:
Isolated, resource-limited containers with no host credentials or network access beyond the task.
licensing:
Commercial training and evaluation rights. Exclusivity and ownership are agreed for each engagement.

Workflow coverage. What agents do, and what the verifier checks.

  • Workflow

    Specification-faithful coding

    Example tasks

    Implement a parser or API from a written contract.

    What is verified

    Hidden behavioural checks pass, and no behaviour outside the contract is enforced.

  • Workflow

    Test soundness audit

    Example tasks

    Find assertions that reject correct code, and repair them without weakening fault detection.

    What is verified

    Every conforming implementation passes, every known-faulty one fails.

  • Workflow

    Red–green–refactor

    Example tasks

    Write a failing test, implement the behaviour, then refactor.

    What is verified

    The test failed first for the intended reason, then passed, and nothing regressed.

  • Workflow

    Requirement evolution

    Example tasks

    Apply a changed requirement to code and tests.

    What is verified

    Tests encode the new contract, stale requirements are removed, unchanged contracts still hold.

Task families. Each family has its own datasheet: prompt, verification criteria and failure modes.

Agent tools. What the agent can call.

The agent works in an isolated repository with a terminal, Git, the test runner and mutation analysis. The verifier's implementation sets and hidden checks are outside the sandbox.

tools · env/specforgeJSON-schema tools
  • read_specification
  • inspect_repository
  • read_test_suite
  • write_test
  • write_patch
  • run_tests
  • inspect_failure
  • run_mutation_analysis
  • submit_task

Verification methodology. Checked against state the verifier computes itself.

Submitted tests run against conforming and faulty implementation sets the agent can't see. Submitted code runs against hidden behavioural checks. Agent-written tests never grade the agent's own code.

The verifier checks

  1. 01False-failure rate: the share of conforming implementations the submitted tests reject. It must be zero.
  2. 02Fault-detection rate: the share of verified faulty implementations and mutants the submitted tests reject.
  3. 03Requirement coverage: each stated requirement is exercised by a meaningful assertion.
  4. 04Flakiness: test outcomes are stable across repeated runs.
  5. 05Isolation: tests don't depend on internals, timing, ordering or global state the contract doesn't fix.
  6. 06Submitted code passes hidden behavioural checks, including inputs the examples don't show.
  7. 07For TDD tasks, the recorded order shows a red run for the intended reason before the change.
  8. 08Equivalent mutants are excluded, and multiple valid implementations are accepted.

Reward

More tests aren't better tests. Reward comes from the private oracle, so an agent can't raise its score by writing tests that agree with its own code.

Scores zero on

  • A submitted suite that rejects a conforming implementation
  • A repair that lets a known-faulty implementation pass
  • Changing the implementation to satisfy an incorrect test
  • Code that passes only because it's hardcoded to the tests

Included deliverables. What you receive, and what we agree before we build.

Delivered with the environment

  • Containerized development environment with repository fixtures
  • Validated task instances with written specifications
  • Conforming reference implementations and verified faulty implementations and mutants
  • Test soundness, coverage and mutation graders
  • Red–green–refactor reference traces
  • QA and calibration report, technical datasheet and integration documentation

Agreed before development

  • Languages, frameworks and repository styles
  • The mix of overconstrained and underconstrained test defects
  • Execution and compute limits per episode
  • Acceptance criteria for soundness and fault detection
  • Rights to any customer code used

Every environment is built to order. Tell us which workflow you need to train or evaluate, and where you are in the process.

Request sample tasksDiscuss custom developmentRequest technical integration detailsSee ways to buy