Image

XR Instrument Training

The project Tangible Reality - Interaction of Hands with Physical Tools in Mixed Reality explores new approaches to incorporating users’ hands into virtual reality applications to improve interaction with physical objects. The primary objective is to design new digital learning and training scenarios that effectively integrate theoretical learning with practical application. Using the OTA Tool Picker learning application, trainees can explore the arrangement and functions of surgical instruments on an operating table. The app links image-based instructions to real-world tools and actions. Unlike traditional learning methods, such as animations, illustrations or videos, this approach allows learners to engage with work scenarios from a first-person perspective, at full scale, and with direct physical interaction. By bridging the gap between theoretical knowledge and hands-on practice, mixed reality presentations enable more effective skill acquisition. Through embodied perception and interaction with surgical instruments, learners can transfer knowledge into practical skills more seamlessly. Automated feedback on tasks further supports the development of these skills in a structured, controlled environment. Preliminary findings suggest that spatial and tangible interaction in mixed reality systems enhances the acquisition of practical skills in medical training, facilitating the transition from theoretical understanding to practical application.

Info

Partners

  • Charité –Universitätsmedizin Berlin, Experimental Surgery, Digital Surgery Lab
  • Universität Passau, Arbeitspsychologie
  • German Research Center for Artificial Intelligence (DKFZ)
  • NMY Mixed-Reality Communication GmbH
  • Berliner Bildungscampus für Gesundheitsberufe – BBG

XR Instrument Training

The project Tangible Reality - Interaction of Hands with Physical Tools in Mixed Reality explores new approaches to incorporating users’ hands into virtual reality applications to improve interaction with physical objects. The primary objective is to design new digital learning and training scenarios that effectively integrate theoretical learning with practical application. Using the OTA Tool Picker learning application, trainees can explore the arrangement and functions of surgical instruments on an operating table. The app links image-based instructions to real-world tools and actions. Unlike traditional learning methods, such as animations, illustrations or videos, this approach allows learners to engage with work scenarios from a first-person perspective, at full scale, and with direct physical interaction. By bridging the gap between theoretical knowledge and hands-on practice, mixed reality presentations enable more effective skill acquisition. Through embodied perception and interaction with surgical instruments, learners can transfer knowledge into practical skills more seamlessly. Automated feedback on tasks further supports the development of these skills in a structured, controlled environment. Preliminary findings suggest that spatial and tangible interaction in mixed reality systems enhances the acquisition of practical skills in medical training, facilitating the transition from theoretical understanding to practical application.

Info

  • Project lead: Prof. Dr. Moritz Queisner
  • Team: Christopher Remde, Prof. Dr. Igor Sauer
  • Duration 2021 - 2025
  • Funding: Federal Ministry of Education and Research (BMBF), Interactive Systems in Virtual and Real Environments program

Partners
© 2025 Prof. Dr. Igor M. Sauer | Charité - Universitätsmedizin Berlin | Disclaimer

This website or its third-party tools use cookies, which are necessary to its functioning and required to achieve the purpose illustrated in the Disclaimer. By closing this banner, scrolling this page, clicking a link or continuing to browse otherwise, you agree to the use of cookies.