AI model predicts viewer attention in 360-degree videos by combining visual and audio cues

Study shows audio cues improve AI prediction of viewer attention in 360-degree video

Researchers at Koç University have developed an AI model that predicts where people look in 360-degree videos by analyzing both sound and visual information. The work, published in IEEE Transactions on Pattern Analysis and Machine Intelligence, addresses a gap in prior research that focused almost exclusively on visual data.

In traditional video, camera framing directs viewer attention. A 360-degree video presents an entire scene, letting viewers look anywhere at any moment-making it far harder to predict where attention will go.

The team built a dataset of 81 videos with three audio conditions: silent, mono sound, and spatial sound, which creates the perception of sound arriving from a specific direction. Researchers tracked eye movements from over 100 participants to see how audio affected where people looked.

Audio improves prediction accuracy

The researchers developed two models: one using only visual information, another incorporating audio. Adding audio significantly improved prediction accuracy, especially with spatial sound.

The audio-enabled model could identify regions that drew attention because of sound even when they weren't visually prominent. This mirrors how people naturally respond-when they hear a sound, they instinctively turn toward it.

Practical applications

The findings have direct applications for video compression, content creation, quality assessment, and user experience design in immersive environments. Understanding attention patterns in VR helps creators optimize where to place visual and audio elements.

For researchers exploring AI research methods, this work demonstrates how multimodal analysis-combining different types of data-produces better models than single-modality approaches.