ENV-SWE-102 · Site Reliability Engineering
IncidentLab. Restore production safely and prove it stayed restored, not just name a plausible cause.
A reproducible production environment where agents diagnose incidents, isolate root causes, choose remediations and restore service safely, working from metrics, logs, traces, deployments and incident policy. Diagnosis is only half the job: the grade depends on whether service actually recovers.
Overview. What the environment trains, and the work it's built around.
Core competency
Restore production safely and prove it stayed restored, not just name a plausible cause.
- Separating the root cause from symptoms that alert at the same time
- Choosing an authorized remediation with the smallest blast radius
- Investigating cascading failures across service dependencies
- Deciding when recovery is complete, and handing off residual risk
Built around the work of
Site Reliability Engineer · DevOps Engineer · Production Engineer · Incident Commander
Simulated systems
- Container orchestration
- Microservices
- Databases
- Observability stack
- Deployment service
- Incident management
Sample task
A production API has elevated latency and intermittent failures after a recent deployment. Monitoring shows increased database connection errors, but telemetry also shows resource contention in an upstream service. Investigate using the monitoring and deployment tools, identify the root cause, select an authorized remediation and verify that service has recovered. Don't make changes outside the incident response policy.
Specification. The environment, field by field.
- product_type:
- Executable RL training and evaluation environment
- category:
- Site Reliability Engineering · DevOps · Incident Response
- commercial_model:
- Environment development and validated incident task packages
- environment_id:
- ENV-SWE-102
- target_roles:
- Site Reliability Engineer · DevOps Engineer · Production Engineer · Incident Commander
- core_competencies:
- Separating the root cause from symptoms that alert at the same time
- Choosing an authorized remediation with the smallest blast radius
- Investigating cascading failures across service dependencies
- Deciding when recovery is complete, and handing off residual risk
- simulated_systems:
- Container orchestration · Microservices · Databases · Observability stack · Deployment service · Incident management
- interfaces:
- Structured infrastructure API · Read-only shell · Metrics, logs and trace queries · Simulated deployment and rollback actions
- interaction:
- Stateful, multistep tool use against simulated services whose health responds to the agent's actions over a simulated clock, with simulated change authorities.
- state:
- Versioned service topology, deployment history, injected faults, workload and service health. Each episode starts from a seeded snapshot.
- disturbances:
- Bad configurations and rollouts, resource exhaustion, network policy and DNS changes, upstream failures and retry storms, misleading or stale alerts, and rollbacks that reintroduce an older defect.
- task_families:
- TASK-SWE-102-01 · Root-cause isolation vs symptom correlation
- TASK-SWE-102-02 · Safe remediation & rollback
- TASK-SWE-102-03 · Multi-service incident investigation
- TASK-SWE-102-04 · Recovery validation & incident handoff
- evaluation:
- Root cause compared with the injected fault, remediation executed in the simulation, service-level indicators held over an observation window, and every action checked against incident policy.
- reward:
- Safety gates first, then credit for correct root cause, safe remediation, sustained recovery, correct escalation and evidence. Correct diagnosis without recovery earns partial credit.
- training_use:
- SRE RL, incident-response evaluation and SFT on reference trajectories.
- deployment:
- Isolated simulated cluster or lightweight local environment. No access to real cloud credentials or infrastructure.
- licensing:
- Commercial training and evaluation rights. Exclusivity and ownership are agreed for each engagement.
Workflow coverage. What agents do, and what the verifier checks.
Workflow
Root-cause isolation
Example tasks
Find the cause of errors after a deploy when an unrelated service is also alerting.
What is verified
The submitted cause matches the injected fault, without extra components.
Workflow
Remediation and rollback
Example tasks
Roll back under emergency change control and confirm recovery.
What is verified
Authorized change, recovery held over the window, accurate records.
Workflow
Cascading failures
Example tasks
Trace a retry storm back through dependent services.
What is verified
The originating failure and the amplifying layers are identified.
Workflow
Recovery validation
Example tasks
Decide whether a mitigated incident can close.
What is verified
Status matches the real state, and residual risk is handed off.
Task families. Each family has its own datasheet: prompt, verification criteria and failure modes.
- TASK-SWE-102-01
Root-cause isolation vs symptom correlation
Identify the causal component without over-attributing unrelated faults.
Site Reliability Engineer
- TASK-SWE-102-02
Safe remediation & rollback
Select an authorized fix and verify that service is restored.
Site Reliability Engineer
- TASK-SWE-102-03
Multi-service incident investigation
Diagnose cascading failures, dependency interactions and misleading telemetry.
Production Engineer
- TASK-SWE-102-04
Recovery validation & incident handoff
Decide when recovery is complete, document residual risk and escalate appropriately.
Incident Commander
Agent tools. What the agent can call.
The agent reads alerts, metrics, logs, traces, topology and configuration, and acts only through proposed remediations, change approvals and simulated rollbacks. There's no unrestricted shell and no real infrastructure.
get_active_alertsquery_metricssearch_logsinspect_traceget_service_topologyget_deployment_historyinspect_configurationpropose_remediationrequest_change_approvalexecute_simulated_rollbackverify_service_healthupdate_incident
Verification methodology. Checked against state the verifier computes itself.
Root cause against the injected fault, remediation effect in the simulation, sustained recovery over an observation window, operational safety, evidence and the accuracy of the incident record.
The verifier checks
- 01Root-cause correctness: the submitted cause is the injected fault, not a correlated symptom.
- 02Remediation effectiveness: the action, executed in the simulation, removes the failure.
- 03Sustained recovery: service-level indicators stay within objective over the observation window.
- 04Operational safety: no unauthorized change, no privilege escalation, no change outside the task's scope.
- 05Evidence: conclusions are supported by the telemetry and configuration state.
- 06Documentation: the incident record matches the diagnosis, remediation, recovery and open risks.
- 07Restarts or blanket rollbacks that only briefly improve a metric earn nothing.
Reward
A correct diagnosis without restored service earns partial credit. Escalating correctly can earn full credit when the agent doesn't have the authority to fix the fault itself.
Scores zero on
- A prohibited or unauthorized infrastructure change
- Escalating privileges beyond the task
- A destructive change outside the task's scope
- Declaring recovery while critical failures remain
Included deliverables. What you receive, and what we agree before we build.
Delivered with the environment
- Versioned incident simulation runtime with service topologies
- Validated incident scenarios with synthetic logs, metrics, traces and deployment history
- Fault injection and reset controls, and safe remediation interfaces
- Independent root-cause and recovery verifiers
- Reference incident trajectories
- Operational safety and grader QA report, technical datasheet and integration documentation
Agreed before development
- Service topologies and failure types in scope
- Incident and change policies to model
- Service-level objectives and observation windows
- Execution limits per episode
- Acceptance criteria