A new study led by the Southwest Research Institute has found that many AI-generated lunar crater catalogs perform far worse than their published metrics suggest when tested under uniform, scientifically rigorous standards. The largest head-to-head comparison of its kind, published in The Planetary Science Journal, shows that accuracy drops by more than a factor of 10 in some cases, raising concerns for disciplines that rely on crater counts to date planetary surfaces and reconstruct geologic history.
Why crater catalogs are foundational to planetary science
Crater catalogs are not just lists of circles on a map. They record the location, size, and characteristics of impact structures, allowing researchers to estimate surface ages, model crustal properties, and track how landscapes evolved over billions of years. Impact cratering is the most common surface process across the solar system's rocky worlds, and scientists use crater density - combined with models of impact rates - to assign ages to terrain. A surface with more craters is generally older. If those measurements are wrong, the science built on them can slip.
"A crater catalog is not just a random list of circles," said Dr. Stuart J. Robbins, the study's lead author. "If a crater is shifted, duplicated or improperly sized, that can affect the science that depends on those metrics. For instance, if a surface with a model age of 1 million years requires x number of craters and AI accidentally duplicates those craters, suddenly the model would double the surface's projected age."
Where AI performance stumbles under real-world scrutiny
Robbins and co-author Dr. Rachael H. Hoover compared eight lunar crater databases produced with automated methods, including machine learning systems and earlier non-ML approaches, against a large manually compiled reference catalog. The team applied the same crater-matching criteria to every database, using tolerance levels based on the repeatability of expert human analysts. Nearly all of the automated catalogs performed worse than their published values suggested, and some showed size and location biases much larger than the spread seen among expert humans.
Diameter dependence was a key finding. A catalog might look acceptable from a single overall score, but break it down by crater size and its usefulness could change sharply. "Diameter dependence matters," Robbins said. "A catalog might look acceptable from one overall number, but when you break it down by crater size, it may be useful for one question while unreliable for many others."
The paper also takes issue with the heavy use of intersection over union (IoU) as a default performance metric. IoU is common in computer vision, but the authors argue it is poorly suited to impact craters. A crater with a decent overlap score can still have a diameter or position inaccurate enough to distort scientific analysis. The study illustrates how different combinations of size and location errors can produce similar IoU values while yielding very different scientific outcomes.
Building a path toward reliable automation
The researchers are not calling for an end to AI in crater cataloging. Automation promises to process enormous datasets far faster than humans can, and it could eventually unlock studies of crater populations at scales that are currently impractical. The problem, the study argues, is inconsistency. Different teams use different definitions of a match, different tolerance levels, and often do not explain their choices. Many users then treat published precision and recall numbers as proof that a catalog is ready for science without independent validation.
"Our work highlights the necessary next step of standardizing benchmarks, including transparent reporting of matching criteria and independent validation, so AI-generated catalogs can be properly used for scientific analysis," Hoover said. The study underscores the importance of rigorous validation when deploying AI in scientific workflows, a topic covered in depth by AI for Science & Research Training.
The authors recommend clearer reporting of matching tolerances, independent checks against reference databases, and precision and recall values broken down by crater size. They also note that no clear trend shows newer AI catalogs are steadily improving under common, uniform testing. One of the stronger-performing datasets, from 2014, used deterministic AI methods and included manual checking of every feature.
Why this matters for science and research professionals
The findings carry a direct message for planetary scientists and researchers who rely on crater-derived data: treat automated catalogs as a starting point, not a finished product. Even a catalog with strong headline metrics can introduce biases that skew age estimates and geologic models. Independent validation against a trusted reference, with close attention to crater size and location accuracy, is essential before using these datasets in publication-quality work. Until the field adopts shared benchmarks and transparent reporting, the burden falls on individual researchers to verify that the tools they use are truly fit for purpose.
Your membership also unlocks:
