Columbia Project Investigates How AI Models Encode Artistic Style
Researchers at Columbia University are mapping how deep neural networks recognize and reproduce artistic style, a collaboration between art historians and computer scientists funded by the Data Science Institute.
The project, led by Associate Professor of Art History Noam M. Elcott and Henry and Gertrude Rothschild Professor of Computer Science Kathleen McKeown, focuses on detecting which artist created a given image and explaining the aesthetic features behind the prediction. The team is examining latent spaces-the abstract, high-dimensional areas within neural networks where patterns are encoded but not easily interpretable by humans.
Elcott and McKeown discovered they were researching the same question from different angles: how style is encoded in the "black box" of neural networks. McKeown had completed work on latent space in literary style and wanted to investigate visual arts. Elcott was theorizing latent space's role in art history. The collaboration expanded to include Kate Crawford, a leading AI scholar at the University of Southern California, plus legal scholars and science-and-technology-studies researchers.
How the Model Works
McKeown's team developed an interpretable model that identifies which concepts a vision-language model uses when predicting artistic style. The model explains its reasoning by showing image regions and generating short descriptions of each area.
The researchers began with the WikiArt dataset, covering architectural styles including Art Nouveau, Baroque, and Gothic, plus painting styles such as Cubism, Minimalism, and Pop Art. They constructed a separate benchmark of architecture that hasn't been used to train existing vision-language models, ensuring models haven't simply memorized the data.
Cultural Bias in AI Systems
McKeown's prior research revealed that off-the-shelf vision-language models exhibit consistent Western bias. In experiments using artwork labeled with emotions from China and America, nearly all models performed better on examples from English-speaking countries and Western Europe than on examples from China and neighboring regions.
The finding reflects how language affects visual understanding in AI systems. Since vision-language models train on both images and text, the language used in training data shapes how models interpret visual content.
Copyright and Legal Questions
The project raises practical legal questions. Styles traditionally cannot be copyrighted, but AI models can restyle images in the manner of specific creators-including famous studios like Studio Ghibli. The researchers want to understand how encoding style in latent space affects copyright law and intellectual property.
One possibility: could a trained model serve as an expert witness in copyright cases? Elcott said the team plans to pose these questions to legal scholars, computer scientists, and art historians.
Next Steps
The team will complete its first computational study by May 2026, when they're hosting a workshop with leading art historians, computer scientists, and legal scholars. The workshop is intentionally open-ended-researchers want to see what emerges from their findings rather than predetermine outcomes.
After analyzing how models predict style in paintings from WikiArt, the researchers plan to study three-dimensional architecture from images outside current vision-language models' training data. This approach will help separate the impact of text about style in training data from the visual features models actually detect.
McKeown said the project could continue for years, with multiple research directions still available.
Learn more: Explore topics in Generative AI and LLM and Research to deepen your understanding of how these systems work.
Your membership also unlocks:
