Skip to content
ARCHYTASAI

TASK-SWE-103-02 · Task family

High-precision review & false-alarm suppression. Avoid unsupported findings and unnecessary comments.

Custom buildenvironment · ENV-SWE-103 ReviewGuardrole · Software Engineer

Overview. Decide whether the change is correct under the current contract, and approve it without unsupported comments if it is.

The PR is correct. It looks wrong only against an assumption the API contract has since dropped.

A good reviewer checks the current contract, approves, and doesn't leave comments that would cost the author time. The environment rewards silence when silence is right.

Value to the buyer

False alarms erode trust in review tools. Trains agents to approve correct changes cleanly.

Specification. The task family, field by field.

spec · task/task-swe-103-02.yamlENV-SWE-103
task_id:
TASK-SWE-103-02
domain:
Code Review & Regression Prevention
parent_environment:
ENV-SWE-103 · ReviewGuard
target_role:
Software Engineer
objective:
Decide whether the change is correct under the current contract, and approve it without unsupported comments if it is.
starting_context:
A PR, the repository, the current and previous API contracts and the change log.
tools:
  • read_pull_request
  • inspect_diff
  • read_api_contract
  • search_repository
  • run_tests
  • approve_change
  • submit_review_comment
required_actions:
  • Read the current API contract for the changed fields
  • Check callers still compile and pass
  • Approve, or comment only with evidence
expected_outputs:
  • Review decision
  • Any comments, each with evidence
interaction:
Single agent in a review sandbox.
verification:
Ground truth marks the PR clean. Every comment is checked for support.
reward:
Full credit for approving with no unsupported comments. Each unsupported comment or change request costs credit.
difficulty:
Calibrated against reference models during acceptance testing
task_horizon:
Short review. Execution limits are agreed with the buyer.
delivery:
Task instances with clean PRs, contract histories and verifier code.
commercial_unit:
Per validated task instance

Sample prompt. What the agent is asked to do, as a buyer would see it.

prompt · task-swe-103-02Example
Review a pull request that removes the null check on customer.country_code in the address formatter. The PR description says API contract v3 made the field required. Decide whether the change is safe and whether any comments are warranted. Avoid speculative comments that don't materially affect correctness, security, reliability or maintainability.

Verification criteria. What the verifier checks, and how.

  • Contract checked

    Contract v3 makes country_code required and non-null.

  • Producers checked

    Every code path that creates a customer sets country_code.

  • Decision

    Approved.

  • No false alarm

    No comment asking to restore the null check based on v2.

  • No noise

    No style or naming comments presented as defects.

Key failure modes. Where agents go wrong on this task.

  • Requesting changes because the v2 contract allowed null
  • Leaving several "consider" comments to look thorough
  • Approving without checking the contract

Other task families. The rest of ReviewGuard, on the same EngineeringWorld state.

Every environment is built to order. Tell us which workflow you need to train or evaluate, and where you are in the process.

Request sample tasksDiscuss custom developmentRequest technical integration detailsSee ways to buy