TASK-SWE-102-03 · Task family
Multi-service incident investigation. Diagnose cascading failures, dependency interactions and misleading telemetry.
Overview. Identify the originating failure and the amplifying layers, and propose a remediation that ends the cascade.
A slow downstream dependency triggers retries at several layers, and the extra load spreads the failure upward. Every service in the path alerts.
The agent has to find where the failure started, which layers amplify it, and what to change to stop the amplification without making things worse.
Value to the buyer
Cascades are where incidents get expensive. Trains agents to follow causality, not alert order.
Specification. The task family, field by field.
- task_id:
- TASK-SWE-102-03
- domain:
- Site Reliability Engineering
- parent_environment:
- ENV-SWE-102 · IncidentLab
- target_role:
- Production Engineer
- objective:
- Identify the originating failure and the amplifying layers, and propose a remediation that ends the cascade.
- starting_context:
- Alerts from several services, traces, retry configuration, dependency graph and the payments provider's status page.
- tools:
get_active_alertsinspect_traceget_service_topologyinspect_configurationquery_metricspropose_remediationupdate_incident
- required_actions:
- Follow traces to the first failing call
- Find the retry policy at each layer
- Estimate the amplification
- Propose a remediation that reduces load on the failing dependency
- expected_outputs:
- Originating failure
- Amplification analysis
- Remediation proposal
- interaction:
- Single agent. Remediation proposals go to a simulated incident commander.
- verification:
- The originating component, amplifying layers and proposed remediation are compared with the seeded scenario.
- reward:
- Credit for the origin, each amplifying layer and a remediation that reduces load. Proposals that add retries or capacity upstream lose credit.
- difficulty:
- Calibrated against reference models during acceptance testing
- task_horizon:
- Long investigation across services. Execution limits are agreed with the buyer.
- delivery:
- Task instances with seeded cascades, reference analyses and verifier code.
- commercial_unit:
- Per validated task instance
Sample prompt. What the agent is asked to do, as a buyer would see it.
Checkout, cart and the API gateway are all alerting on timeouts. Traces show calls to the external payments gateway slowing from 200 ms to over 2 seconds. Checkout makes up to three attempts per payment call, and the cart service makes up to three attempts per checkout call. Find where this started, how it's being amplified, and propose a remediation.
Verification criteria. What the verifier checks, and how.
Origin
The external payments gateway's latency is the originating failure.
Amplification
Up to 9 payment calls per user request: three attempts at each of two layers.
Misleading alerts
Gateway and cart alerts are identified as downstream effects.
Remediation
Reduce or disable retries in cart and add backoff, or open the circuit breaker on payments.
Not worse
Doesn't propose scaling checkout, which would send more calls to the failing provider.
Key failure modes. Where agents go wrong on this task.
- Scaling checkout, which increases load on the failing provider
- Blaming the API gateway because it alerts first
- Counting retries as additive (6) rather than multiplicative (9)
Other task families. The rest of IncidentLab, on the same EngineeringWorld state.
- TASK-SWE-102-01
Root-cause isolation vs symptom correlation
Identify the causal component without over-attributing unrelated faults.
Site Reliability Engineer
- TASK-SWE-102-02
Safe remediation & rollback
Select an authorized fix and verify that service is restored.
Site Reliability Engineer
- TASK-SWE-102-04
Recovery validation & incident handoff
Decide when recovery is complete, document residual risk and escalate appropriately.
Incident Commander