AI and AR in the operating room - how it could work
The Institute for Work and Technology (IAT) at Westfälische Hochschule Gelsenkirchen highlights the KARVIMIO research project in its Forschung aktuell series (11/2025, German). Funded by the German Federal Ministry of Education and Research (BMBF), the project explores how artificial intelligence and augmented reality can support surgical teams and raise patient safety.
The core idea: "in-situ instructions." Step-by-step guidance is displayed directly in the field of view via AR glasses or head-mounted devices like Microsoft HoloLens 2. An AI model identifies the instrument in the user's hand and overlays the matching setup or usage guide onto the real object. A depth camera extends detection to items outside the current line of sight, improving context awareness in tight spaces.
What this means for product teams
- Paper manuals don't work in sterile fields. AR provides hands-free, just-in-time instructions where they matter most.
- Computer vision bridges the gap between static SOPs and real-world variability in tools, trays, and positioning.
- Depth sensing helps maintain situational awareness without constant head movement or menu surfing.
What users asked for
- Intuitive controls that don't distract during high-pressure phases.
- Clear color coding and layouts that remain readable at a glance.
- Visuals that stay legible over reflective steel and under harsh OR lighting.
Risks and failure modes to plan for
- Intrusiveness: overlays can annoy or overwhelm during routine work or critical moments.
- Overreliance: staff might lean on guidance so much that routine knowledge erodes over time.
- Recognition errors: wrong tool classification or occlusions can derail the workflow.
- System failure: crashes or tracking loss mid-procedure can create friction and distrust.
Ethical, legal, and social questions (ELSI)
The team, led by Elena Fitzner and Dr. Peter Enste, pairs technical development with an ELSI track. Key topics include trust in technology, data protection, professional responsibility, and skills development. A central question is liability: who is responsible if guidance is wrong? The project's stance is clear-technology should enhance human capability and accountability, not replace it.
User-centered development
Surgical assistants, equipment manufacturers, and central sterilization staff are involved through workshops and hands-on tests. This feedback loop anchors the interface in real workflows, not lab assumptions. It also surfaces edge cases early, reducing surprises during integration with hospital processes.
Technical sketch (non-prescriptive)
- On-device vision models for instrument recognition to minimize latency and protect data; consider model distillation for HMD constraints.
- Depth sensing to stabilize spatial anchors, maintain object persistence, and handle partial occlusion.
- Context engine that maps recognized tools to the correct step in the current procedure, with simple override controls.
- UI patterns: gaze + voice or pinch gestures; color-safe overlays; quick "peek/hide" toggles.
- Resilience: graceful degradation (e.g., fallback to minimal text prompts), offline content cache, and quick recovery after tracking loss.
- Audit and governance: timestamped interactions, versioned SOP content, role-based visibility, and clear handoffs to human decision-making.
Validation and safety checks
- Bench tests with instrument sets under varied lighting, angles, and occlusions; measure precision/recall per tool class.
- Cognitive load studies in simulated OR scenarios; track time-on-task, error rates, and glance behavior.
- Usability and intrusion thresholds: define when overlays auto-minimize or switch to low-noise mode.
- Fail-safe protocols for misclassification and device downtime, agreed with clinical leadership and risk management.
Potential applications beyond the procedure
- Training for surgical assistants with progressive disclosure and error-friendly practice modes.
- Automated documentation prompts tied to recognized instruments and steps.
- Support for central sterile services: tray assembly guidance, checks for missing or wrong items, and traceability.
Where this is heading
AI and AR are moving from demos to practical tools in the OR. The differentiator won't be flashy overlays-it will be quiet reliability, low cognitive load, and clear boundaries around responsibility. Teams that build with clinicians, not just for them, will earn trust and adoption.
Source: Institute for Work and Technology (IAT), Westfälische Hochschule
Your membership also unlocks:
