Sign language AI to train on real Deaf conversations, not interpreter videos
A new £3.5 million UK-Japan research project will reshape how sign language AI is built. Over five years (2026-2031), the team will train models on natural Deaf-to-Deaf conversations and develop human-centred AI and augmented reality systems for real-time translation across British Sign Language (BSL), Japanese Sign Language (JSL), English and Japanese.
The collaboration is led by Professor Richard Bowden at the University of Surrey, with partners across the UK and Japan. The goal is simple: model how Deaf people actually communicate in conversation-turn-taking, backchannels, repair, and shared visual attention-rather than relying on interpreter-to-camera footage.
Professor Richard Bowden said: "Most AI research on sign language has used video of interpreters signing to cameras. We know that's not how Deaf people naturally communicate. What excites me about this project is that we're working with authentic conversations between Deaf signers. That will give us much richer insight into how people really interact - and help us build AI systems that reflect that complexity."
Why this matters for researchers
Interpreter data is cleaner and more staged, which can bias models away from the messiness of real dialogue. Training on authentic conversations introduces overlap, occlusion, varied signing styles and viewpoint changes-precisely the conditions systems need to handle in the field.
This shift supports better segmentation, alignment and disambiguation in spontaneous settings. It also enables evaluation beyond isolated sign recognition to conversational flow and mutual attention.
What UMCS will deliver
- New datasets of Deaf-to-Deaf BSL and JSL conversations, linked to English/Japanese where appropriate.
- Annotations for turn-taking, backchannels, repair strategies, gaze and visual attention.
- Human-centred AI and AR prototypes for real-time translation and interaction.
- Community co-design to address ethics, consent, privacy and usability from the start.
Professor Mayumi Bono said: "Sign language corpora have been built to capture natural Deaf-to-Deaf interaction, yet they remain largely unused in AI research because today's AI systems demand large-scale, text-linked data. As the field moves from "corpus to dataset," researchers are calling for an inclusive science that bridges linguistics and AI while centring on the lived realities and linguistic intuitions of Deaf signers."
Core research challenges
- Multimodal fusion of manual signs, facial expressions, mouthings and gaze without studio constraints.
- Turn-taking and repair modelling that respects conversational timing and simultaneity.
- Evaluation that reflects dialogue use-cases, not just lab benchmarks.
Team
- Professor Richard Bowden (University of Surrey, UK) - Principal Investigator; AI and computer vision
- Professor Annelies Kusters (Heriot-Watt University) - Sociolinguistics and Deaf communication
- Dr Robert Adam (Heriot-Watt University) - Comparative sign linguistics and interpreter studies
- Professor Mathini Sellathurai (Heriot-Watt University) - Augmented and virtual reality systems
- Professor Kearsy Cormier (University College London, DCAL) - Sign linguistics and corpus research
- Professor Mayumi Bono (National Institute of Informatics, Japan) - Japanese Sign Language linguistics
- Professor Yutaka Osugi (Tsukuba University of Technology) - Deaf Studies and sign language education
- Professor Hideki Nakayama (University of Tokyo) - Generative AI and multimodal language modelling
- Professor Koji Inoue (Kyoto University) - Human-robot and conversational interaction
Industry partners
Signapse Ltd (UK) and NHK Enterprises (Japan) will contribute expertise in translation technologies and sign avatar systems, helping move research prototypes into real-world use.
Funding and timeline
UMCS is jointly funded by UK Research and Innovation through the Engineering and Physical Sciences Research Council (EPSRC) and the Japan Science and Technology Agency (JST). The total combined investment is approximately £3.5 million (¥700 million), supporting exchanges, data collection, model development and community co-design from 2026 to 2031.
For practitioners: what to watch next
- Open datasets that prioritise conversation-level phenomena over isolated signs.
- Metrics that track interaction quality, latency and user experience in AR settings.
- Deployment pilots with Deaf communities via the project's industry partners.
Learn more
The project sits within the Japan-UK Joint Call for Collaborations in Advancing Human-Centred AI. Media interviews can be arranged via the University of Surrey's media relations team.
Your membership also unlocks:
