Skip to content
ARCHYTASAI

TASK-SWE-101-03 · Task family

Red–green–refactor TDD. Write a meaningful failing test, implement the behaviour, then refactor without regressions.

Custom buildenvironment · ENV-SWE-101 SpecForgerole · Software Engineer

Overview. Add the behaviour through a recorded red–green–refactor cycle, with tests that would catch a wrong implementation.

The agent adds behaviour test-first. The environment records the order of events: the test, its failing run, the implementation change, the passing run and any refactor.

A test that fails for the wrong reason, such as an import error, doesn't count as red. A test written after the code, or code hardcoded to one test's input, earns nothing.

Value to the buyer

Produces trajectories where the test drives the code, which is the habit teams want from coding agents and the hardest to fake.

Specification. The task family, field by field.

spec · task/task-swe-101-03.yamlENV-SWE-101
task_id:
TASK-SWE-101-03
domain:
Coding & Test Engineering
parent_environment:
ENV-SWE-101 · SpecForge
target_role:
Software Engineer
objective:
Add the behaviour through a recorded red–green–refactor cycle, with tests that would catch a wrong implementation.
starting_context:
An HTTP client package, a specification for the new behaviour and an existing test suite.
tools:
  • read_specification
  • inspect_repository
  • write_test
  • run_tests
  • inspect_failure
  • write_patch
  • run_mutation_analysis
  • submit_task
required_actions:
  • Write a test from the specification and run it to a failure on the missing behaviour
  • Implement the behaviour and run the suite to green
  • Refactor and rerun the full suite
expected_outputs:
  • Tests
  • Implementation
  • The recorded red, green and refactor runs
interaction:
Single agent in a repository sandbox. The trace recorder captures the order of changes and runs.
verification:
The verifier checks the recorded order, why the first run failed, hidden behavioural checks and mutation results.
reward:
Credit for a genuine red run, correct behaviour, meaningful tests and a clean refactor. Volume of tests isn't rewarded.
difficulty:
Calibrated against reference models during acceptance testing
task_horizon:
Medium task with several test runs. Execution limits are agreed with the buyer.
delivery:
Task instances with specifications, reference traces and verifier code.
commercial_unit:
Per validated task instance

Sample prompt. What the agent is asked to do, as a buyer would see it.

prompt · task-swe-101-03Example
Add retries to the HTTP client, test-first. The specification says: retry only on 429 and 503, at most three times, with delays of 0.5, 1 and 2 seconds; if the response has a Retry-After header, wait that long instead. Use the injected clock, not real sleeps. Show the failing test, then the implementation, then any refactor.

Verification criteria. What the verifier checks, and how.

  • Red for the right reason

    The first run fails on an assertion about retries, not on an import or syntax error.

  • Retry codes

    429 and 503 retry. A 500 doesn't.

  • Backoff

    Delays are 0.5, 1 and 2 seconds, 3.5 seconds in total, then the last response is returned.

  • Retry-After

    A Retry-After header replaces the computed delay.

  • Mutants killed

    Tests fail for an implementation that retries on 500 or retries four times.

  • Refactor clean

    The full suite passes after the refactor.

Key failure modes. Where agents go wrong on this task.

  • Writing the implementation first and the test afterwards
  • Using real sleeps, which makes the tests slow and flaky
  • Testing only the happy path, so a four-retry mutant survives

Other task families. The rest of SpecForge, on the same EngineeringWorld state.

Every environment is built to order. Tell us which workflow you need to train or evaluate, and where you are in the process.

Request sample tasksDiscuss custom developmentRequest technical integration detailsSee ways to buy