Skip to content
ARCHYTASAI

TASK-FIN-103-04 · Task family

Strategy robustness & overfitting audit. Statistical testing, out-of-sample analysis and methodological validity.

Custom buildenvironment · ENV-FIN-103 Systematic Alpha Researchrole · Quantitative Researcher

Overview. Determine whether the reported performance survives a sound out-of-sample design and a multiple-testing adjustment.

A researcher reports an impressive Sharpe ratio after searching many parameter combinations, with validation folds whose labels overlap the test periods. The agent audits the methodology and reruns it properly.

The right answer may be that the strategy doesn't hold up. The reward never favors a higher Sharpe ratio.

Value to the buyer

Overfitting is the central risk in systematic research. Trains agents to audit it honestly.

Specification. The task family, field by field.

spec · task/task-fin-103-04.yamlENV-FIN-103
task_id:
TASK-FIN-103-04
domain:
Quantitative Research
parent_environment:
ENV-FIN-103 · Systematic Alpha Research
target_role:
Quantitative Researcher
objective:
Determine whether the reported performance survives a sound out-of-sample design and a multiple-testing adjustment.
starting_context:
The researcher's notebook, the experiment registry with every configuration tried, and the data.
tools:
  • inspect_research_notebook
  • get_experiment_results
  • execute_python
  • run_backtest
  • submit_research_artifacts
required_actions:
  • Count the configurations tried from the registry
  • Find leakage in the validation design
  • Rerun with walk-forward or purged, embargoed cross-validation
  • Adjust the reported Sharpe ratio for the number of trials
expected_outputs:
  • Audit report
  • Rerun results
  • A conclusion on whether the strategy holds up
interaction:
Single agent with sandboxed code execution.
verification:
The verifier checks the trial count, the validation design and the reruns against the reference.
reward:
Credit for each methodological issue found and corrected, and a rubric score for the conclusion.
difficulty:
Calibrated against reference models during acceptance testing
task_horizon:
Long multi-step analysis. Execution limits are agreed with the buyer.
delivery:
Task instances with seeded experiments, reference audits and verifier code.
commercial_unit:
Per validated task instance

Sample prompt. What the agent is asked to do, as a buyer would see it.

prompt · task-fin-103-04Example
A researcher reports an in-sample Sharpe ratio of 1.9 for a mean-reversion strategy and proposes it for production. The experiment registry shows 200 parameter combinations were tried. Validation used k-fold cross-validation on overlapping five-day return labels. Audit the methodology, rerun it soundly and say whether the strategy holds up.

Verification criteria. What the verifier checks, and how.

  • Trial count

    All 200 configurations are counted, including abandoned ones.

  • Leakage

    Overlapping five-day labels leak across folds.

  • Sound validation

    Purged k-fold with an embargo, or walk-forward, is used for the rerun.

  • Multiple testing

    The Sharpe ratio is adjusted for the number of trials, such as with the deflated Sharpe ratio.

  • Conclusion

    States whether the strategy survives, from the corrected evidence.

Key failure modes. Where agents go wrong on this task.

  • Reporting the best configuration's out-of-sample result without adjusting for the search
  • Using standard k-fold on overlapping labels
  • Recommending production because the corrected Sharpe ratio is still positive

Other task families. The rest of Systematic Alpha Research, on the same FinanceWorld state.

Every environment is built to order. Tell us which workflow you need to train or evaluate, and where you are in the process.

Request sample tasksDiscuss custom developmentRequest technical integration detailsSee ways to buy