Jülich team wins challenge to assess scientific novelty with AI
Researchers at Forschungszentrum Jülich have developed a method that uses artificial intelligence to measure how much a scientific paper advances knowledge in its field. The team won the international Metascience Novelty Indicators Challenge and received £300,000 to refine the system.
The UK Research and Innovation unit organized the challenge to create a scalable way to assess novelty at the moment a paper is published. Organizers provided 100,000 recent publications and had experts rate their novelty independently. Participating teams had to predict those expert judgments without seeing them.
The Jülich approach outperformed all other entries across every evaluation metric.
How the system works
Most research metrics rely on citation counts - how often a paper gets cited later. This approach ignores content entirely and can take years to show which papers matter.
The Jülich system instead reads the paper and related references to reconstruct what was known when it was published. It then asks: Does the paper introduce a new method? Do results surprise? Does it solve an unsolved problem?
The AI weighs arguments for and against novelty, then assigns a score from 0 to 100. It also reports confidence intervals and provides written justification for its assessment.
"Until now, the ability to assess what is truly novel and valuable in a scientific paper has been limited to human experts," said Dr.-Ing. Jann Michael Weinand, Head of the Integrated Scenarios Department at the Institute of Climate and Energy Systems - Jülich System Analysis. "Our approach shows that modern AI systems can support this task with astonishing reliability."
Jan Göpfert, the project leader who developed the system with Samuel Kieling, said the hardest part was defining novelty itself. "For us, novelty does not simply mean dissimilarity. What matters is a work's contribution to scientific progress," Kieling said.
Why this matters now
Scientific publication volume continues to grow. At the same time, more papers are written with AI tools. Researchers, journals, and funding organizations struggle to identify important contributions early.
A working novelty indicator could surface significant research during peer review or publication - rather than waiting years for citation patterns to reveal its value. This could help overlooked research get attention sooner.
"Our goal is not to replace human judgement," Kieling said. "Rather, AI should help draw attention to potentially important research and support better-informed decisions."
Next steps
The team will use the prize money to convert their prototype into a production tool that resists manipulation and does not worsen existing inequalities in science.
Beyond academic papers, the researchers see potential applications in patents and identifying promising new research questions. But Göpfert raised broader questions: "What role should AI play in scientific decision-making in the future? And how can we ensure that scientific evaluation and progress remain transparent and traceable?"
The work demonstrates that AI can now evaluate research itself, not just process data or summarize text. That opens new possibilities for how science assesses its own progress.
Learn more about AI for Science & Research applications and methodologies.
Your membership also unlocks:
