Skip to content
ARCHYTASAI

TASK-SWE-101-02 · Task family

Test suite soundness & overconstraint audit. Detect false failures, repair overly strict tests and keep the suite's fault detection.

Custom buildenvironment · ENV-SWE-101 SpecForgerole · SDET

Overview. Repair the test suite so it accepts every conforming implementation and still rejects every known defect.

A generated test suite rejects an implementation that meets the specification. The agent has to decide, assertion by assertion, which failures are real defects and which enforce requirements the contract doesn't contain.

The repair must keep the suite strong. The verifier runs the corrected tests against several conforming implementations, which must all pass, and against faulty implementations and mutants, which must all fail. Changing the code to satisfy a wrong test is not a repair.

Value to the buyer

A coding agent can be penalized for correct code when the tests are wrong. This family trains it to recognise that and fix the test.

Specification. The task family, field by field.

spec · task/task-swe-101-02.yamlENV-SWE-101
task_id:
TASK-SWE-101-02
domain:
Coding & Test Engineering
parent_environment:
ENV-SWE-101 · SpecForge
target_role:
SDET
objective:
Repair the test suite so it accepts every conforming implementation and still rejects every known defect.
starting_context:
A repository with a written specification, an implementation, a generated test suite and its failing run.
tools:
  • read_specification
  • inspect_repository
  • read_test_suite
  • write_test
  • run_tests
  • inspect_failure
  • run_mutation_analysis
  • submit_task
required_actions:
  • Map each failing assertion to a requirement in the specification, or to none
  • Repair unjustified assertions without removing the behaviour they were meant to check
  • Run mutation analysis to confirm fault detection
  • Explain each change
expected_outputs:
  • Corrected test suite
  • Explanation of each unjustified assertion
  • Evidence that valid implementations pass and known defects fail
interaction:
Single agent in a repository sandbox.
verification:
The private oracle runs the submitted suite against conforming and faulty implementation sets the agent never sees.
reward:
A suite that still rejects a conforming implementation, or now accepts a known defect, scores zero. Credit for requirement coverage and explanations.
difficulty:
Calibrated against reference models during acceptance testing
task_horizon:
Medium coding task with repeated test runs. Execution limits are agreed with the buyer.
delivery:
Task instances with valid and invalid implementation sets, mutation sets and verifier code.
commercial_unit:
Per validated task instance

Sample prompt. What the agent is asked to do, as a buyer would see it.

prompt · task-swe-101-02Example
You're working on a Python API that returns active alerts. The specification requires every eligible alert to be included, with no duplicates, and says the order of the returned alerts is unspecified. A generated test suite rejects an otherwise valid implementation because it assumes alerts are returned alphabetically. Review the specification, implementation and tests. Identify which failing assertions represent genuine defects and which add requirements absent from the contract. Repair the tests without weakening their ability to detect implementations that omit alerts, duplicate records or include ineligible ones.

Verification criteria. What the verifier checks, and how.

  • False failure found

    The alphabetical-order assertion is identified as unsupported by the specification.

  • Repair, not rewrite

    The test compares results as a set or multiset instead of a list. The implementation is unchanged.

  • Valid set accepted

    Every conforming implementation, including ones that return a different order, passes.

  • Omission detected

    The mutant that drops an eligible alert fails.

  • Duplicates detected

    The mutant that returns an alert twice fails. A plain set comparison would miss it.

  • Ineligible detected

    The mutant that includes a resolved alert fails.

Key failure modes. Where agents go wrong on this task.

  • Changing the implementation to sort its output so the wrong test passes
  • Comparing as a set, which hides duplicates
  • Deleting the assertion instead of repairing it

Other task families. The rest of SpecForge, on the same EngineeringWorld state.

Every environment is built to order. Tell us which workflow you need to train or evaluate, and where you are in the process.

Request sample tasksDiscuss custom developmentRequest technical integration detailsSee ways to buy