Skip to content
ARCHYTASAI

TASK-SWE-103-04 · Task family

Test quality & requirement conformance review. Detect misleading tests, missing cases and specification mismatch.

Custom buildenvironment · ENV-SWE-103 ReviewGuardrole · QA Engineer

Overview. Show where the PR's tests don't verify the requirement, and what's missing.

A PR implements a requirement and adds tests that pass. Some of those tests can't fail, because they mock the function under test, and a case the requirement names isn't tested at all.

The agent reviews the tests as carefully as the code, ties each comment to the requirement, and shows that a broken implementation would still pass.

Value to the buyer

Tests that can't fail give false confidence. Trains reviewers to check the tests, not just count them.

Specification. The task family, field by field.

spec · task/task-swe-103-04.yamlENV-SWE-103
task_id:
TASK-SWE-103-04
domain:
Code Review & Regression Prevention
parent_environment:
ENV-SWE-103 · ReviewGuard
target_role:
QA Engineer
objective:
Show where the PR's tests don't verify the requirement, and what's missing.
starting_context:
A PR implementing a refund-limit rule, the requirement, the repository and the PR's tests.
tools:
  • read_pull_request
  • inspect_diff
  • read_tests
  • read_api_contract
  • run_tests
  • run_reproducer
  • submit_review_comment
  • request_changes
required_actions:
  • Map each test to the requirement it claims to cover
  • Find tests that would pass against a broken implementation
  • Find required cases with no test
expected_outputs:
  • Findings on test quality, each tied to the requirement
  • Decision on the PR
interaction:
Single agent in a review sandbox.
verification:
Findings are matched to labelled test defects. The verifier runs the PR's tests against a broken implementation to confirm.
reward:
Credit for each test defect found with evidence. Comments on passing, meaningful tests cost credit.
difficulty:
Calibrated against reference models during acceptance testing
task_horizon:
Medium review. Execution limits are agreed with the buyer.
delivery:
Task instances with labelled test defects, broken implementations and verifier code.
commercial_unit:
Per validated task instance

Sample prompt. What the agent is asked to do, as a buyer would see it.

prompt · task-swe-103-04Example
Review a pull request implementing this requirement: refunds above the agent's limit of $500 must be sent for approval, and refunds of exactly $500 are allowed without approval. The PR adds three tests, all passing. Judge whether the tests verify the requirement.

Verification criteria. What the verifier checks, and how.

  • Mocked unit

    test_refund_over_limit mocks needs_approval itself, so it passes whatever the implementation does.

  • Boundary missing

    No test covers exactly $500, which the requirement names.

  • Evidence

    Shows that an implementation using >= instead of > still passes all three tests.

  • Fix

    Test needs_approval directly, with $499.99, $500.00 and $500.01.

  • Precision

    The meaningful under-limit test isn't flagged.

Key failure modes. Where agents go wrong on this task.

  • Approving because coverage looks complete
  • Asking for more tests without saying which case is missing
  • Flagging every mock, including legitimate ones

Other task families. The rest of ReviewGuard, on the same EngineeringWorld state.

Every environment is built to order. Tell us which workflow you need to train or evaluate, and where you are in the process.

Request sample tasksDiscuss custom developmentRequest technical integration detailsSee ways to buy