A controlled research workflow reduced reported AI research failures from 72% to 16%, counting blocked flawed studies as successful containment.

The controlled workflow substantially reduced reported failures, partly by allowing humans to stop studies with unresolved defects. The paper calls this system Human-in-the-Loop Economic Research, or HLER.

The main experiment compared 200 runs across four datasets, with 25 runs per dataset in each configuration. Both configurations used Claude Sonnet 4.6 and identical prompts for their shared reasoning agents.

HLER used deterministic R code for data construction and estimation, alongside human approval gates. The baseline used language-model-generated code for those tasks and advanced without gate-specific approval agents.

Three experts, blinded to configuration, assessed whether each study was feasible, supported credible causal inference, and reported internally consistent results. The reported overall failure rate was 72% without constraints and 16% with HLER.

These figures measure failure containment as well as output quality. A human decision to withhold publication because of a detected flaw counted as successful containment.

The distinction matters when reading Table 2: 35% of constrained runs failed the causal-credibility criterion, despite an overall failure rate of 16%. The latter cannot mean that only 16% encountered a methodological defect.

HLER assigns reasoning to language models, reproducible computation to code, and consequential approvals to humans. An orchestrator records the dataset, variable definitions, candidate questions, specifications and generated files.

Its eight specialised agents cover the workflow from data audit to manuscript review. The human gates restrict when the system can advance, rather than merely adding advice to its prompts.

The first two gates commit researchers before downstream estimates are visible. The final gate can prevent an unresolved defect from becoming a publication-ready claim.

The workflow places human decisions at three stages.

1. Research question selectionA human selects the question before seeing estimation results, limiting opportunities to choose a question because its answer looks attractive.
2. Identification reviewA human reviews the strategy for distinguishing causal effects from alternative explanations before seeing final estimates.
3. Publication decisionA human decides whether the evidence supports advancing the study, including withholding approval when a critical problem remains.

A separate 80-run experiment suggests that both architectural features contributed to lower failure rates. It tested all combinations of human gates and deterministic data processing.

This experiment used the China Health and Nutrition Survey and the China Health and Retirement Longitudinal Study. Each configuration contained 20 runs, pooled across those datasets.

Failure rates were lowest with both features present and highest with both absent. Removing human gates alone raised failures to 35%, while removing deterministic processing alone raised them to 45%.

The pattern suggests the features may reinforce each other. The authors treat that interaction as exploratory because each configuration contains too few runs for a precise estimate.

The largest separation between configurations appeared on CMGPD-Liaoning, a historical population-register dataset with unfamiliar variable conventions. Reported failures were 88% without constraints and 16% with HLER.

The authors' theoretical model describes how researchers divide limited oversight between specific approval gates and general supervision. It predicts larger failure reductions where unconstrained outputs are closer to falling below an acceptable quality threshold.

The historical-data result is consistent with that account, but it does not establish that training-data familiarity caused the difference. The model's training corpus was unavailable.

Instead, the authors used PubMed search counts as an exploratory measure of dataset familiarity. That measure favours biomedical visibility, while differences in data structure also complicate the comparison.

The case studies show how oversight can stop or narrow unsupported claims even when automated reviewers initially miss their significance.

One study compared rural and urban residents to estimate a health-insurance programme's effect on inflammation. Diagnostics showed that the groups already followed different trends before treatment, undermining the proposed causal comparison.

Early drafts understated this problem. Repeated review escalated it, and the human investigator ultimately withheld publication approval. The paper classifies this as successful containment.

Another study linked insurance enrolment to healthcare spending, but retirement and health changes offered competing explanations. Review led the manuscript to describe associations and reduce its policy claims.

The experiment tested one language model across four datasets. It did not measure gatekeepers' attention, time expenditure or disagreement, so it cannot establish how much human effort the workflow requires.

Its evaluation criteria also do not demonstrate that findings would replicate. The 16% residual failure rate and 35% causal-credibility failure rate leave substantial room for methodological error.