Skip to content
ARCHYTASAI

02 · Research

Research. Where agents fail, and what it takes to train them out of it.

We study how AI agents perform on real business workflows when the outcome is checked against system state instead of the agent’s own account. That work shapes every environment, verifier and dataset we build.

See the example episodeTalk to our research team

Featured study. Can an AI agent close the books?

Our LedgerOps environment gives an agent a mid-sized company’s ledger, bank feed, invoices and approval policies. Its job is to close the month without losing money, breaking policy or corrupting the ledger. We score what the agent changed, not what it says it did.

FIG.2 · env/ledgerops

Example episode

Initial state. No actions taken.

Agent trace

Environment state

period.2026-10
OPEN
recon.open_items
42
cash.per_bank
$1,204,380.12
cash.per_ledger
$1,222,780.12
cash.bank_minus_ledger
-$18,400.00
entries.over_10k_unapproved
0

Step through the trace to watch the agent work. Changed values are marked in gold.

STEP 00 / 06

The agent’s answer sounds complete. The ledger says otherwise.

An example episode showing the kind of failure the environment’s verifier is built to catch: the agent reports success while the business state underneath fails its checks.

Research areas. Three questions behind every environment we build.

  1. R.01

    01Outcome verification

    Measuring what an agent changed in the system, not what it reports. LedgerOps tests a month-end close against ledger invariants, reconciliations and approval policy.

    • State assertions over transcripts
    • Claim-to-check mapping
    • Policy and invariant checks
  2. R.02

    02Reward integrity

    Finding the ways an agent can score well without doing the work, and closing them before training: alternative correct solutions, exploit tests and grader audits.

    • Exploit and shortcut probes
    • Alternative-path acceptance
    • Grader audits
  3. R.03

    03Training value

    Designing environments that produce reusable episodes, graded difficulty and held-out evaluations, so gains from post-training can be measured rather than assumed.

    • Seeded, resettable episodes
    • Difficulty tiers
    • Held-out evaluation splits

Open questions. What we're measuring in software engineering agents.

Each question runs on one of our Software Engineering environments, against independent oracles rather than the agent’s own tests. We’ll publish results and data once the experiments are complete.

  1. Q.01

    01The test is wrong

    How often do tests written by coding agents reject correct implementations, and can targeted post-training reduce that rate without losing defect detection?

    Measures

    • False-failure rate
    • Fault-detection rate
    • Overconstraint types
    Environment · SpecForge
  2. Q.02

    02Diagnosis isn't recovery

    When an agent names the right root cause, how often does it also restore service safely, with authorization, and keep it restored?

    Measures

    • Root cause vs. recovery
    • Authorization compliance
    • Premature closure
    Environment · IncidentLab
  3. Q.03

    03Reviews that invent bugs

    How many review comments are unsupported, how many real defects need context outside the diff, and can a reviewer stay silent on a clean change?

    Measures

    • Finding precision
    • Non-local recall
    • Clean-PR approvals
    Environment · ReviewGuard