Casting the Runes: Can Superforecasters Predict AI's Pace?

Science & technology | Casting the runes: Can we predict how quickly AI will progress?

Big promises keep pouring in. One leader says next year's systems will produce "novel insights." Another suggests artificial general intelligence could land on a similar timeline. A third sees medicine transformed within a decade, even hinting at cures for all diseases.

Ambition is healthy. But planning teams, researchers, and funders need something sturdier than optimism. That's where a new push comes in: use disciplined forecasting to estimate how fast AI will move, and update those estimates as evidence arrives.

Why this matters now

Budgets, hiring, safety protocols, and research bets all hinge on timelines. If core capabilities jump next year, you reallocate. If progress stalls, you double down on foundational work. Most organizations still guess.

Forecasting turns guesses into numbers with accountability. It forces clear definitions, specific dates, and public scoring. You get probabilities, not narratives.

So what exactly is being forecast?

The most useful questions are concrete and observable. Expect the project to track milestones like these:

• Demonstrations of "novel insights": peer-reviewed discoveries where an AI system materially advances a result beyond its training data.
• General reasoning competence: performance on standardized benchmarks that reflect real problem-solving, not just memorization.
• Autonomy: multi-step task completion in open-ended environments with minimal human scaffolding.
• Science and health impact: AI systems assisting in wet-lab design, drug discovery, or target identification at a pace competitive with top research teams.
• Economic reach: measurable displacement or amplification of knowledge-work tasks in production settings, not just demos.

These map to statements you've heard. Next-year breakthroughs. AGI "within the same timeframe." Medical breakthroughs within a decade. Forecasts will translate talk into time-bounded, testable claims.

Who are "superforecasters," and why involve them?

Superforecasters are individuals who consistently produce accurate, well-calibrated probability estimates across diverse topics. They break problems down, use base rates, update often, and avoid wishful thinking. They're scored and held to outcomes.

Bringing them into AI timelines adds rigor and a track record of accountability many hype cycles lack. If you want an outside view to counter lab marketing and Twitter heat, this is it.

For context on the method and history, see resources from Good Judgment. Community platforms such as Metaculus also contribute useful signals through aggregated predictions.

What makes AI forecasting hard

Three moving targets complicate everything: compute, algorithms, and data. A surprise in any one can bend the curve. A surprise in all three can break it.

There's also the benchmark trap. Scores jump, then saturate, then the benchmarks change. Real capability is uneven: language, tools, planning, and safety don't advance in lockstep.

Finally, deployment creates new feedback loops. Tools that write code accelerate AI research itself, compressing timelines in ways past tech waves did not.

Signals worth watching

• Training compute growth: actual FLOPs used in state-of-the-art models, not announced budgets. Independent trackers like Epoch AI are useful here.
• Algorithmic efficiency: how much performance improves at fixed compute. Gains here often sneak up and compound.
• Tool use and autonomy: success rates on multi-step tasks without heavy human scaffolding.
• Scientific contribution: peer-reviewed results with clear AI provenance, especially in biology and materials.
• Safety and reliability: progress on controllability, interpretability, and evaluation of long-horizon risks.
• Policy constraints: export controls, licensing, and liability shifts that can slow or redirect research.

How a serious forecasting effort should run

• Operationalize each milestone: define exact criteria for "AI produced a novel insight" or "automated X% of task Y."
• Time-box questions: six, twelve, and thirty-six month horizons promote frequent updating.
• Score publicly: Brier or log scores keep everyone honest.
• Blend models: outside-view base rates, inside-view technical analysis, and market signals.
• Update on evidence, not vibes: new papers, evals, and deployments move probabilities up or down.

Where current claims fit

"Novel insights next year" translates to: what evidence would confirm it? A paper credited to an AI system as a co-discoverer? A new theorem proof? A verified biological target that reaches preclinical success?

"AGI soon" needs a definition. Is it broad task coverage at human level across relevant domains under cost constraints? Forecasting forces the definition, which is the real value.

"Cure all diseases" is inspirational; forecasting reframes it as a sequence: target identification, candidate generation, trial acceleration, and post-market surveillance assisted by AI. Each step can be forecast and scored.

Practical takeaways for researchers and leaders

• Plan by probabilities, not point estimates. Treat 20%, 50%, and 80% scenarios differently in budgets and hiring.
• Create trigger points. If capability X is achieved by date Y, unlock budget Z or shift roadmap A → B.
• Diversify bets. Balance core research with tooling that compounds regardless of timelines: data pipelines, evals, and safety practices.
• Institutionalize updates. Monthly forecast reviews beat annual strategy documents that age in place.
• Measure impact in production. Demos inform curiosity; deployments inform cash flow and risk.

What superforecasters will likely do differently

• Start with base rates: historic compute growth, algorithmic efficiency trends, and prior tech diffusion curves.
• Decompose grand claims into smaller, testable questions with shorter clocks.
• Use conditional forecasts: "If training compute doubles by Q4, probability of milestone M by year-end rises from 25% to 45%."
• Stay calibrated: resist overconfidence, even when narratives are loud.

How you can use forecasts today

• Tie forecasts to decisions. A 60% chance of capable code-generation at lower cost next year? Delay certain tool purchases; accelerate internal evals.
• Budget for uncertainty. Split investments between capability bets and safety/evaluation infrastructure that pays off under most futures.
• Skill up your team. Forecasts inform what to learn and when. If reasoning and autonomy climb, prioritize prompt engineering, evaluation design, and workflow integration.

If you're aligning training to job roles, this catalog is a quick start: AI courses by job.

The bottom line

Bold claims get attention. Calibrated probabilities drive better choices.

Forecasting won't settle the AGI question overnight, but it will give you a living map of where we are, what could happen next, and what to do if it does. That's enough to make smarter bets-this quarter, and the next.