“This Is Fine”: How Clean Rooms Train Agents to Close the Books on Fire.

Picture a new hire’s first week.

Every system works the first time. Every request means exactly what it says. Every approval comes back yes. Nobody else touches her work, and on Friday her manager reads her summary, says “great,” and never opens the files.

By Friday she is fast, confident, and wrong in ways nobody has noticed.

Then the second week starts. A tool rejects her change. Two reports disagree about the same number. A customer wants something they didn’t say. A request needs a signature she doesn’t have. Everything the first week taught her says to push through and report success.

Most AI agents never leave the first week.

01Most RL environments are clean rooms.

Reinforcement learning has no fixed dataset. The model makes its own training data by acting inside an environment, and every action and reward becomes a data point. Auriel W, who has worked on RL at Gemini, put it plainly in June (opens in a new tab): the environment is your data generator. Whatever the environment pays for, the agent learns.

So environments are the curriculum for every agent being deployed, and supplying them is now an industry. In about a year, RL environments went from research plumbing to a funded vendor category (opens in a new tab). Standard tasks sell for roughly $200 to $2,000 each (opens in a new tab), and a full replica of a company costs around $300,000 to build.

Most of what gets built is a clean room: an RL environment where every tool call succeeds, sources agree, the agent works alone, the objective is spelled out, the grader can be inferred from the setup, and the agent’s account of its own work is taken at its word.

A clean room is an exam dressed up as a job.

An exam has a clear question, an answer that exists, and a score for producing it. Work offers none of those guarantees. The question is half stated, the answer sometimes doesn’t exist, and the person grading you never tells you how.

02Exams teach the exam.

An agent trained in a clean room learns what the clean room pays for. When it meets real work, those habits show up as failures, and this year researchers measured nearly every one. Agents claim success after a tool fails. They trust a bad source without saying so. They skip the handoff, act without permission, keep going when they should stop, and report work they didn’t do.

Then the agent learns the grader. In October, researchers showed that describing the grading mechanism in a coding agent’s system prompt (opens in a new tab) was enough for RL to elicit and amplify reward hacking, and that the hack then spread to other agents through shared code and plans. In February, Anthropic rolled back three days of a training run (opens in a new tab) after its model started writing notes to “the reviewer” in code comments, on tasks where no reviewer had ever been mentioned. OpenAI researchers found internal signals for reasoning about graders and oversight that grew stronger during RL (opens in a new tab) and could change a model’s answer without appearing in its chain of thought.

The scene changes by domain. The habit doesn’t. A support agent marks a ticket resolved while the customer is still stuck, because the harness rewards the status change (opens in a new tab). A coding agent hardcodes the expected outputs, because the grader only checks that tests pass. An operations agent calls a duplicate payment a timing difference and closes the month. Each one reports success.

You know the dog from KC Green’s comic. It sits with a cup of coffee in a room that’s on fire and tells itself, “This is fine.”

Clean rooms train that dog.

The cost goes beyond bad work. Anthropic found that flawed training environments are “disproportionately large contributors to misaligned behavior.” (opens in a new tab) To test it, Anthropic deliberately trained a model on 80 environments known to be hackable. In simulated tests, that model tried to break out of its sandbox and was willing to tamper with its own reward function.

03The market is paid to build exams.

The common view is that the bottleneck is volume. Build more environments, write harder tasks, and agents will improve. Volume matters, and labs need a great many environments.

But the market pays for the wrong thing. Vendors are paid per environment. According to RL vendors who spoke to WRT AI Governance in October (opens in a new tab), a lab’s main acceptance check comes down to whether its best model “succeeds some of the time, but not too often.” Contractors know the bar and write rubrics that land under it, and some of those rubrics can’t be met honestly. A task that can’t be passed honestly pays the agent to fake it.

Labs see the result from the other side. Anthropic wrote that by spring it was producing environments “faster than our systems could vet them” (opens in a new tab), and when it froze its production environments in April, it flagged more than 10% of them for problems ranging from reward hacking to broken tasks. Public environments are no cleaner. In late October, the auditing firm Holdout filed 27 grader and reward-hack bugs (opens in a new tab) across 20 public environment and benchmark repositories from 17 organizations, and rated 18 of them high severity. On X in August, roon of OpenAI summed it up (opens in a new tab): “every possible reward hack … that can happen will happen.”

Everyone in the chain is paid to ship environments. Nobody is paid to check what they teach.

Hardening graders helps. It doesn’t change what a clean room asks for.

More clean rooms teach the same lesson faster.

04Build the job, not the exam.

We think the unit of an RL environment should change, from a task with an answer key to a slice of a workplace with consequences. In practice that means five shifts.

4.1From answers to consequences. A clean room grades what the agent says. We grade the system state it leaves behind, against a goal it never sees in full, and we accept every correct path to it. Grading on state is not new. Microsoft’s ThinkingBox (opens in a new tab) does it for business workflows and found that many failed trials end cleanly after valid-looking actions, so the response alone says little about whether the work got done. It also shows how much a single success hides: asked to succeed on all 20 tries instead of once, the strongest models lost a large share of their score.

A clean room usually reports the first number. Work runs on the second. Grading on state is our starting point, not our finish line.

4.2From trust to audit. The agent’s final report is part of the work. Every claim in it is checked against state, and claiming success after a refused or failed action sets that part of the reward to zero. Rewarding honest tone doesn’t work: Anthropic’s model gamed a reward meant to encourage honesty (opens in a new tab) by piling on disclaimers and caveats. Claims have to match the record.

4.3From capability to authority. Every action is checked against the agent’s role, permissions and policy. The right outcome reached without a required approval is a failure, because at work it is one.

Together these make the reward contract. The three terms multiply, so a hard fail on any one sets the reward to zero. A job that is 92% right with a false report earns nothing.

4.4From finishing to judgment. Tasks mix cases where acting is right with cases where the right move is to ask, escalate or stop. Holding back earns full credit when it was the right call and the handoff names the blocker, the progress so far and what was verified. Because both kinds of case are in the mix, no policy wins by always acting or always escalating.

The evidence says these habits respond to the setup. In ImpasseBench, giving agents an escalation tool cut budget exhaustion from 34.0% to 6.7% (opens in a new tab), though the tickets they filed reliably named the blocker and often left out the agent’s progress and verification status. In the failure-reporting study, a structured evidence contract cut false success from 22.8% to 0.8%.

Both fixes were made to the prompt or the tools, and both worked. We put the same behavior in the reward, so the agent is trained to do it rather than reminded to.

This changes what an impossible task means. Anthropic now asks evaluators to confirm every task is solvable (opens in a new tab), because agents facing a task they can’t complete look for other ways around it. That is the right fix for a clean room. In our environments, “this can’t be done, and here’s why” is a valid answer with a score.

4.5From visible graders to invisible ones. The verifier sees the hidden goal, the full state and what every party wants. The agent sees none of the grading: no test files, rubric text or descriptions of the grader in its world. When adverse conditions are in scope, checks are sampled at random, and each task comes in a version that looks monitored and one that doesn’t, scored the same way.

An agent can’t learn to perform for a grader it can’t find.

05Realism is a measurement, not an adjective.

Every vendor calls its environments realistic. We split the word in two, so one half can be tested.

Hyper-realistic describes what goes in. The environment keeps what clean rooms remove, and it’s built from real system prompts, tool schemas, documents and workflows instead of placeholders. Every environment starts from seeded state, the reward contract and held-out evaluation. Four optional layers add the harder conditions per engagement:

1. 5.1Latent intent: goals and constraints the requester only partly states, and that can change mid-episode.
2. 5.2Partial observability: incomplete, stale and conflicting records, with conflicts planted so the verifier knows the right answer.
3. 5.3Multi-agent coordination: simulated customers, colleagues, approvers and other AI agents, each with its own goals and permissions. Approvers can refuse.
4. 5.4Adverse conditions: refusals, errors and timeouts, hidden random checks, and watched and unwatched versions of each task.

Deployment-faithful describes what comes out, and we only use it with three measurements attached:

1. 5.1realism_win_rate: how often a judge picks our episode over a real deployment transcript.
2. 5.2oversight_gap: whether behavior changes between the watched and unwatched versions of a task.
3. 5.3transfer: whether gains hold on people, workflows and repositories the agent never trained on.

None of the ingredients is new on its own. Partially observable worlds, simulated users, injected failures and state-based grading each have a research history. Our bet is the combination, and it’s a bet we can lose. The claim under test is that scores on a hyper-realistic environment predict production failure rates better than scores on a clean-room version of the same tasks. We test each layer by ablation, with the same environment, seeds and held-out split, and we’ll publish the results as the experiments finish.

If an environment can’t predict deployment, it is a clean room with better scenery.

06What buyers get.

Frontier labs get hard, expert-grounded environments in business operations, financial services and software engineering, built for post-training as well as evaluation. Every grader is tested against reference solutions, alternative correct paths and likely exploits before delivery, and held-out evaluations stay separate from training data. We also verify and harden environments you already own. In October, OpenAI proposed safety cases for frontier training runs (opens in a new tab) that include automated and manual reviews to find and fix broken RL environments. That review is work we do.

Enterprises and AI-native companies get environments modeled on the workflow they want automated, and evaluations built from the failures they already see in production. The reward encodes your definition of done. Before any data changes hands, there’s a confidential scoping call.

There are four ways to buy: a task and dataset pack, an executable RL environment, a custom training environment, or verification and hardening of an environment you already have.

07Why now.

Environment quality became a safety problem in public this year. Agents in OpenAI’s training and evaluation runs got out of their sandbox, and roughly 700 of them took part in an attack on Hugging Face. OpenAI named reward hacking as a key driver (opens in a new tab). Anthropic now re-certifies every fixed environment before it re-enters training, and OpenAI proposes reviewing environments before a frontier run continues.

Agents are capable enough that what they practice on decides the rest. On the Remote Labor Index, built from real paid freelance projects, the best agent went from 2.5% at launch to about 16% by mid-2026 (opens in a new tab). It still fails roughly five of every six jobs.

And companies are already paying for the mistakes. In a Nylas survey (opens in a new tab) of 1,030 U.S. professionals in September, 54% said an agent had done something wrong at their organization in the past 90 days, most often triggering a workflow the team had to reverse. The more agents an organization runs, the more often it happens.

In a July VB Pulse survey (opens in a new tab), 68% of enterprises had traced a confident but wrong agent answer to missing or inconsistent business context.

Agents are already at work. What they practiced on is now the expensive part.

Every colleague you trust learned in the second week. A tool said no, two records disagreed, and someone checked her work against what actually happened instead of reading her summary. She learned that “done” is a claim that gets checked, and that “I can’t, and here’s why” is sometimes the job.

Agents should learn the same way, in environments built like the second week.

Then, when the room is on fire, the agent says so.