A new Southwest Research Institute-led study has found that AI-generated lunar crater catalogs perform far worse than their published metrics suggest when evaluated against the same scientific standards applied to human catalogers. The results, published July 6, 2026, show that some performance measures dropped by more than a factor of 10, raising concerns about using automated databases to date planetary surfaces and reconstruct solar system history.
The study, led by Dr. Stuart J. Robbins and Dr. Rachael H. Hoover of SwRI, compared eight global or large-coverage lunar crater catalogs generated by automated methods. Each was evaluated against a large, manually compiled catalog that Robbins spent years constructing, applying uniform matching criteria.
The role of crater catalogs in planetary science
Impact craters are the dominant geologic feature on the Moon and many other solid worlds. Scientists estimate surface ages by counting craters, because small asteroid strikes occur at a roughly steady rate. Surfaces with more craters are older. Cataloging accurate crater locations, sizes and physical characteristics is essential for these age models, which researchers use to reconstruct geologic histories and study how planetary surfaces evolve.
Where AI metrics mislead
The team discovered that common computer-vision metrics can make automated detection look acceptable even when a crater's location or diameter is scientifically inaccurate. "A crater catalog is not just a random list of circles," Robbins said. "If a crater is shifted, duplicated or improperly sized, that can affect the science that depends on those metrics. For instance, if a surface with a model age of 1 million years requires x number of craters and AI accidentally duplicates those craters, suddenly the model would double the surface's projected age."
When the study applied stricter criteria based on the repeatability of human analysts, nearly every database's performance fell below its published numbers. The findings highlight that AI for Science & Research demands verification against human benchmarks before it can be trusted for quantitative analysis.
Single summary metrics also hid critical weaknesses. "Diameter dependence matters," Robbins said. "A catalog might look acceptable from one overall number, but when you break it down by crater size, it may be useful for one question while unreliable for many others."
The path toward science-ready AI catalogs
The researchers emphasized that the study is not an argument against AI. "Our work highlights the necessary next step of standardizing benchmarks, including transparent reporting of matching criteria and independent validation, so AI-generated catalogs can be properly used for scientific analysis," Hoover said.
Robbins added that AI may eventually transform crater cataloging and save years of manual work. "For now, researchers need to not chase it as the solution to everything," he said. "We need to understand how these tools work, where they fall short and whether their performance is good enough to support the science being done."
Why this matters for science and research
For researchers who depend on large-scale datasets to model surface ages or reconstruct planetary histories, this study offers a concrete warning. AI-generated databases can silently introduce errors that numerical metrics mask. Independent validation using the same repeatable criteria applied to human analysts is not optional - it is the only way to ensure automated tools produce science-ready data. Without it, the risk of doubling a surface's projected age from duplicated craters becomes a real scenario, not a hypothetical one.
Your membership also unlocks:
