AI researchers call for scrutiny of models that help build their successors

Watch what happens inside the lab

A working paper released on September 28 asks governments to pay closer attention to AI systems used in the development of their successors. The Foundation for American Innovation published the report alongside the Cambridge Programme on AI Science and Policy. It is a policy proposal, not a new law.

The author list includes GovAI's Alan Chan, Cambridge's Christoph Winter, Geoffrey Hinton and researchers affiliated with major AI companies. The paper explicitly says their views do not necessarily represent those organisations. A co-author's affiliation is not a company endorsement.

The feedback loop they are worried about

The report considers an intelligence explosion: a sharp acceleration in AI development, potentially compressing years of progress into much less time. The proposed mechanism is that AI helps produce better systems, which then do more of the research needed to produce the next generation.

The authors argue that governments should obtain better information about this work and prepare ways to steer or constrain it. Their executive summary points to reporting research-automation indicators, embedded auditors and oversight of high-stakes research decisions. These are recommendations to consider, not measures shown to be in force.

There is an important gap between AI helping researchers today and a self-sustaining acceleration. Faster completion of one activity does not tell us how fast a whole research programme can move.

One concrete measure still includes people

Anthropic's own measurement proposal illustrates both the change and the limits of the evidence. For August 2026, it reports Claude leading 26% of measured AI research and development work. In its scale, leading means completing most of a task from a high-level prompt while a human supervises. It says no measured subset is fully autonomous.

This is not an independently audited percentage of all research across the industry. Anthropic uses its models to classify internal work and acknowledges difficulties with model-based judging and cross-company comparisons. It also fixes a basket of tasks for comparison, so a rising score does not by itself capture new kinds of work people may take on.

That makes it a signal to investigate, not proof that researchers have been replaced or that AI can reliably build its successor without them.

A warning with unresolved assumptions

The paper acknowledges constraints: computing capacity, data, tasks that remain hard to automate and time-consuming experiments can all slow the proposed feedback loop. It describes evidence for overcoming these obstacles as preliminary and sometimes mixed.

Our reading is that the strongest immediate question is a measurement question. What work did the AI actually complete? Who checked it? Did that make the full development cycle faster, or simply shift work to review and repair? Comparable answers would be more useful than treating a forecast as a fact.

The confirmed development is the publication of a proposal for greater scrutiny. Whether the acceleration it describes occurs, on what timescale and with which consequences remains unsettled.