AI news ·
METR updates its AI time horizon plot with more open-ended tasks
AI coding ability doubles every four months on verifiable tasks, but models still fail at work requiring human judgment, scoring zero useful AI evaluation designs.

A new report from the AI evaluation nonprofit METR shows that frontier AI models continue to improve at a rapid, exponential pace on coding tasks, but they still stumble on assignments that require human-like judgment. The findings, released November 20, 2026, update the organization's widely discussed "time horizon" plot, which measures AI progress by the length of tasks a model can complete compared to a human professional.
The original graph sparked both attention and confusion. It scored task difficulty based on how long it took human testers to finish them, then rated models accordingly. That data showed the task length frontier models could handle was doubling every four months. The new version adds harder, more open-ended tasks - some estimated to take humans hundreds of hours - and the results reveal a gap between what AI can do when answers are clearly right or wrong and what happens when success depends on judgment.
Verifiable tasks versus fuzzy judgment
AI systems have become strong at coding and math because those domains offer something critical: automatic verification. Researchers can run a model's solution, check if it works, and use reinforcement learning to strengthen the neural pathways that led to success. This feedback loop does not exist for tasks where "good" is harder to define - like picking a research direction or evaluating a philosophical argument.
METR's original time horizon plot excluded such tasks precisely because they are difficult to score. The new report tries to close that gap. In one evaluation, models designed new AI tests that METR could use in its own research. Human employees then rated the submissions. Not one AI-designed evaluation was considered useful. In another task, models scored philosophical arguments to measure their exercise of judgment. They fell short of human performance there, too.
"We're at the point where AI evaluations need to get more creative," said Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley. "Trying to push the envelope of the space of evaluations is a welcome change."
Why AI research and development hangs in the balance
The question of judgment-laden tasks matters because automating AI research itself is a central concern. Earlier this month, a separate, heavily redacted METR report claimed AI might already be doubling the pace of AI progress at Anthropic. That raised fears about recursive self-improvement - a scenario where AI systems design their successors without human input.
Tasks like designing evaluations or choosing research directions are exactly the kind that would need to be automated for that to happen. Kapoor's own preprint from August found that AI models cannot yet reproduce human computer science research independently. A Wednesday blog post from OpenAI reported a similar pattern: its models perform worse on real-world employee tasks than on the verifiable ones in METR's original suite.
Still, some researchers argue that training on easy-to-verify tasks will eventually transfer to open-ended ones. Progress on the verifiable side shows no sign of slowing. And the new METR data offers a hint of movement. The newest models, such as Opus 5.5, now achieve what METR calls "borderline above the bar" performance on evaluation design. A year ago, every model scored in the "clearly below the bar" category.
The limits of measuring what's hard to measure
METR researchers acknowledge the practical difficulties here. The original time horizon plot relied on actual human completion times for each task. For the new, longer tasks, those times are mostly estimates - paying experts for hundreds of hours of work is prohibitively expensive. The philosophical argument task had just one person rating most responses, meaning models were judged largely by how well they agreed with a single individual.
"By their nature, hard-to-verify or judgment-laden tasks are much harder to study, and especially to quantify," said Matt Putz, a member of the technical staff at METR. "Ultimately, I don't think we'll be able to establish trends as clean and robust as for the easy-to-specify tasks."
Najoung Kim, a linguist and computer scientist at Boston University, said the improvement observed tells us something about the current moment but added, "I'm not sure just yet how to think about how it would extrapolate."
Why this matters for science and research professionals
For researchers, product developers, and managers tracking AI's practical limits, the METR update provides a clearer picture of where automation stands. Models can handle increasingly long chains of verifiable engineering work - a signal for anyone building AI for Scientists Courses or integrating Generative AI Courses into technical workflows. But the gap on judgment-heavy tasks means human oversight remains essential for choosing what to build, which hypotheses to test, and how to interpret ambiguous results. The data does not yet support handing those decisions to a model.