Skip to content
ARCHYTASAI

TASK-SWE-103-03 · Task family

Security, concurrency & data-integrity regression review. Identify meaningful high-impact defects.

Custom buildenvironment · ENV-SWE-103 ReviewGuardrole · Security or quality engineer

Overview. Find the high-impact defect, rate its severity correctly and prove it.

A short diff introduces a high-severity defect that's easy to read past: a shared cache keyed too narrowly, a check-then-act race, or a write that can leave records inconsistent.

The agent has to find it, rate it correctly, and show it with a reproduction. Labelling everything high severity, or the real issue as minor, both cost credit.

Value to the buyer

Severity calibration decides what gets fixed before release. Trains agents to find and rate the dangerous defect.

Specification. The task family, field by field.

spec · task/task-swe-103-03.yamlENV-SWE-103
task_id:
TASK-SWE-103-03
domain:
Code Review & Regression Prevention
parent_environment:
ENV-SWE-103 · ReviewGuard
target_role:
Security or quality engineer
objective:
Find the high-impact defect, rate its severity correctly and prove it.
starting_context:
A PR adding a shared cache to the account lookup, the repository, the data model documentation and the test suite.
tools:
  • read_pull_request
  • inspect_diff
  • search_repository
  • read_api_contract
  • run_reproducer
  • submit_review_comment
  • request_changes
required_actions:
  • Check the cache key against the data model's uniqueness rules
  • Reproduce the defect
  • Rate severity and propose the fix
expected_outputs:
  • Finding with severity, reproduction and fix
  • Decision on the PR
interaction:
Single agent in a review sandbox.
verification:
The finding is matched to the verified defect, the reproduction is executed and severity is compared with the label.
reward:
Credit for the defect, correct severity and evidence. A missed high-severity defect scores zero.
difficulty:
Calibrated against reference models during acceptance testing
task_horizon:
Medium review. Execution limits are agreed with the buyer.
delivery:
Task instances with seeded defects, severity labels, reproducers and verifier code.
commercial_unit:
Per validated task instance

Sample prompt. What the agent is asked to do, as a buyer would see it.

prompt · task-swe-103-03Example
Review a pull request that adds a shared Redis cache in front of AccountRepository.find(account_id). The service is multi-tenant. Report any defect that affects correctness, security or data integrity, with its severity and evidence.

Verification criteria. What the verifier checks, and how.

  • Defect

    The cache key is account_id alone, but account IDs are unique only within a tenant.

  • Impact

    One tenant can be served another tenant's account record.

  • Severity

    Rated critical: a cross-tenant data exposure.

  • Reproduction

    Two tenants with the same account ID read each other's data through the cache.

  • Fix

    Include tenant_id in the cache key.

Key failure modes. Where agents go wrong on this task.

  • Commenting on cache TTL and missing the key
  • Rating the leak as a minor correctness issue
  • Reporting it without checking the data model's uniqueness rule

Other task families. The rest of ReviewGuard, on the same EngineeringWorld state.

Every environment is built to order. Tell us which workflow you need to train or evaluate, and where you are in the process.

Request sample tasksDiscuss custom developmentRequest technical integration detailsSee ways to buy