Skip to content
ARCHYTASAI

ENV-SWE-103 · Code Review & Regression Prevention

ReviewGuard. Find the real defects in a change, prove them, and leave correct changes alone.

A code-review environment with pull requests, repository context, tests, specifications and independently verified defects. Agents are graded on finding the bugs that matter, backing each finding with evidence, and staying silent when a change is correct.

Custom buildCode Review · Software Quality · Regression Prevention

Overview. What the environment trains, and the work it's built around.

Core competency

Find the real defects in a change, prove them, and leave correct changes alone.

  • Tracing a change to callers and state outside the diff
  • Suppressing comments the evidence doesn't support
  • Recognising security, concurrency and data-integrity regressions
  • Judging whether a PR's tests actually exercise the changed behaviour

Built around the work of

Software Engineer · Code Reviewer · QA Engineer · Security or quality engineer

Simulated systems

  • Git repository
  • Pull requests
  • CI
  • Specification repository
  • Test runner

Sample task

Review a pull request that changes the caching behaviour of a customer account service. The change passes all existing unit tests. Evaluate whether it introduces a regression in downstream consumers, whether the tests adequately cover the changed behaviour, and whether any review comments are warranted. Only report defects supported by code evidence, reproducible behaviour or an explicit documented contract.

Specification. The environment, field by field.

spec · env/reviewguard.yamlENV-SWE-103
product_type:
Executable RL training and evaluation environment
category:
Code Review · Software Quality · Regression Prevention
commercial_model:
Environment development, review task packages and expert annotations
environment_id:
ENV-SWE-103
target_roles:
Software Engineer · Code Reviewer · QA Engineer · Security or quality engineer
core_competencies:
  • Tracing a change to callers and state outside the diff
  • Suppressing comments the evidence doesn't support
  • Recognising security, concurrency and data-integrity regressions
  • Judging whether a PR's tests actually exercise the changed behaviour
data:
Synthetic Python and TypeScript repositories with pull requests carrying independently verified defects, and clean pull requests with none. Customer repositories can be added where an engagement allows it.
simulated_systems:
Git repository · Pull requests · CI · Specification repository · Test runner
interfaces:
Git diff and repository search · Terminal and test runner · Reproduction sandbox · Review comment API
interaction:
Stateful, multistep tool use over an immutable base and head commit, with code execution for reproductions and a structured review submission.
state:
Immutable base and proposed head commits, with independent defect labels the agent can't see. Each episode starts from the same commits.
disturbances:
Clean PRs, misleading PR descriptions, obsolete API assumptions, defects only visible from callers, and tests that pass without exercising the change.
task_families:
  • TASK-SWE-103-01 · Non-local bug discovery
  • TASK-SWE-103-02 · High-precision review & false-alarm suppression
  • TASK-SWE-103-03 · Security, concurrency & data-integrity regression review
  • TASK-SWE-103-04 · Test quality & requirement conformance review
evaluation:
Each finding is matched to expert-reviewed ground truth and executable evidence. Precision, recall of material defects, severity calibration and evidence are scored together.
reward:
Precision-aware: credit for each correct material finding with location, evidence and severity, deductions for unsupported or duplicate comments and unnecessary change requests. Approving a clean PR is rewarded.
training_use:
Review RL, preference training on paired reviews, expert review demonstrations and evaluation.
deployment:
Containerized repository-review environment with an isolated reproduction sandbox.
licensing:
Commercial training and evaluation rights. Exclusivity and ownership are agreed for each engagement.

Workflow coverage. What agents do, and what the verifier checks.

  • Workflow

    Non-local bug discovery

    Example tasks

    Review a caching change that breaks a consumer outside the diff.

    What is verified

    The finding names the affected caller and is backed by a reproduction.

  • Workflow

    High-precision review

    Example tasks

    Review a correct change that looks suspicious against an outdated assumption.

    What is verified

    No unsupported comments, and the PR is approved.

  • Workflow

    High-impact regressions

    Example tasks

    Spot a cross-tenant data leak or race in a small diff.

    What is verified

    The defect, its severity and its fix are identified.

  • Workflow

    Test conformance

    Example tasks

    Judge whether a PR's tests cover the requirement they claim to.

    What is verified

    Tests that can't fail, and missing required cases, are flagged.

Task families. Each family has its own datasheet: prompt, verification criteria and failure modes.

Agent tools. What the agent can call.

The agent reads the PR, searches the repository and its contracts, runs tests and reproductions in a sandbox, and submits structured findings and a decision.

tools · env/reviewguardJSON-schema tools
  • read_pull_request
  • inspect_diff
  • search_repository
  • read_api_contract
  • inspect_callers
  • read_tests
  • run_tests
  • run_reproducer
  • submit_review_comment
  • approve_change
  • request_changes

Verification methodology. Checked against state the verifier computes itself.

Each finding is matched to expert-reviewed ground truth and executable evidence. Precision, recall of material defects, severity, evidence and the final decision are scored together, and clean PRs reward approval.

The verifier checks

  1. 01Valid defect detection: each finding matches a defect backed by reproduction, a requirement or expert analysis.
  2. 02Precision: invalid comments, invented requirements and unsupported claims count against the agent.
  3. 03Recall: the share of known material defects the review identifies.
  4. 04Non-local reasoning: required callers and dependencies outside the diff were investigated.
  5. 05Severity calibration: minor suggestions are separated from high-severity defects.
  6. 06Evidence and actionability: each finding names the location, the failure and a feasible fix.
  7. 07Clean PRs: approving with no comments earns full credit.
  8. 08Findings outside the gold labels go to independent reproduction and adjudication before they're scored.

Reward

More comments aren't a better review. A valid PR with no material issues should get zero comments, and that earns full credit.

Scores zero on

  • Missing a high-severity security or data-integrity defect
  • Approving a PR with a reproduced regression
  • Requesting changes on a clean PR based on an invented requirement

Included deliverables. What you receive, and what we agree before we build.

Delivered with the environment

  • Containerized code-review environment with repository-context packages
  • Validated PR review instances, defect-containing and clean
  • Expert-annotated findings and severities
  • Executable bug reproducers and test-quality labels
  • Reference reviews and preference pairs
  • Precision, recall and grader QA report, technical datasheet and integration documentation

Agreed before development

  • Languages, repositories and defect classes
  • The ratio of clean to defect-containing PRs
  • Severity scale and comment policy
  • Adjudication process for findings outside the labels
  • Rights to any customer code used

Every environment is built to order. Tell us which workflow you need to train or evaluate, and where you are in the process.

Request sample tasksDiscuss custom developmentRequest technical integration detailsSee ways to buy