Researchers at the Department of Energy's SLAC National Accelerator Laboratory have developed an AI-based method for compressing the massive datasets produced by next-generation science experiments without losing the fine details that drive discovery. The work, published in Nature Machine Intelligence, addresses a looming bottleneck: instruments like SLAC's Linac Coherent Light Source (LCLS) will soon generate terabytes of data per second, outstripping current storage and analysis capacity.
"There is going to be such a flood of data that there's really no way to handle it in the way we've done before," said Joshua Turner, lead scientist at SLAC and the Stanford Institute for Materials and Energy Sciences, and principal investigator of the work. "There are many applications in science now where data storage and analysis speed are really important problems, and I think this method is a clever way to solve them."
Conventional compression algorithms erase some of the subtle features in measurements that carry scientific value. In X-ray imaging of materials, for instance, tiny speckles in the data can reveal how a material is structured and how that structure changes over time. Losing those speckles means losing insight. The new method, developed by SLAC researcher Yuan Ni and colleagues, uses neural networks to compress data while letting users control which details are preserved.
"Depending on the underlying data and the desired quality/fidelity, we can typically achieve 10- to 100-fold reductions in file size," Ni said.
How the compression works
Rather than compressing an entire dataset uniformly, the method first applies wavelet analysis, a mathematical technique that separates features of the data by scale. The neural network then compresses each scale separately, ensuring fine-grained details survive the process. The network learns a compact representation of those features, storing them in a form that can be selectively retrieved later.
That selective retrieval is a key advantage. Researchers often need to revisit only a small portion of a compressed dataset. With conventional compression, that means decompressing the whole file, a process that can take hours. The new approach lets users decode only the region of interest, at the resolution they need.
"If you are using a conventional compressor, you would need to decompress the entire file, which could take you minutes, hours or days," said Zhantao Chen, assistant professor at the University of Texas at Austin who worked on the method as a SLAC research associate. "This method can decompress only the region of interest rather than the entire dataset, so it's much more efficient."
Tested across data types
The team tested the method on data from several experimental techniques, including molecular and material measurements, solar magnetic field observations, and photographs. The neural network adapted to each type, learning which features mattered for different measurements. The method is designed to complement, not replace, existing data-reduction strategies.
"Rather than replacing existing compression methods," Ni said, "our work provides an additional AI-based approach."
The researchers trained the neural networks using Perlmutter, a supercomputer at the National Energy Research Scientific Computing Center (NERSC) at Lawrence Berkeley National Laboratory. The work was supported by the DOE Office of Science and SLAC's Laboratory Directed Research and Development program.
For professionals working in IT and development, the practical takeaway is that AI-driven compression is moving beyond media files into scientific and industrial datasets where fidelity matters. The method's ability to compress data 10- to 100-fold while preserving fine details, and to decompress only what's needed, points to a broader trend: AI for IT & Development is increasingly about managing data pipelines intelligently, not just processing data faster. Teams building storage and analysis systems for large-scale experiments or high-volume sensor data should watch this approach - it suggests that future infrastructure may need to treat compression as a selective, model-driven layer rather than a one-size-fits-all utility.
The technique also fits into wider AI Data Analysis workflows, where the bottleneck is often moving and storing data rather than analyzing it. For developers, the practical question is whether their data pipelines can benefit from compression that preserves more than the standard formats allow.
Why this matters for IT and development professionals
For IT teams supporting scientific computing or data-heavy industries, this method offers a concrete path to cut storage costs and speed up data retrieval without sacrificing data quality. The selective decompression feature alone could change how data archives are designed - instead of decompressing entire files to access one slice, systems can fetch only the relevant portion, reducing I/O bottlenecks and compute time. As more experiments and industrial sensors produce data at rates that strain existing infrastructure, AI-based compression tools like this one may become a standard part of the data stack.
Your membership also unlocks:
