Current Projects

FutureLab | Head Mounted Displays

The FutureLab focusses on new technologies in surgery. One major aspect is the use of Virtual Reality and Mixed Reality for preoperative planning and during surgery.

Stacks Image 628
Stacks Image 630
Stacks Image 632

The term, Virtual Reality (VR) is used for immersive experiences, which could be based on real-world content, purely synthetic content or a hybrid of both. Tricking the eye via stereoscopic pictures, videos and software generated three-dimensional scenarios is the most common way. However immersion in VR is limited because it isolates you from the real world around you. For preoperative planning Virtual Reality appears to be useful.

Mixed Reality (MR), i.e. the real and digital worlds integrated and enhanced with holograms, might be mandatory when interacting with the real world, e.g. during surgery. In 1994 Paul Milgram and Fumio Kishino defined a mixed reality as "...anywhere between the extrema of the virtuality continuum.“, i.e. the spectrum between the real environment, Augmented Reality (AR), Augmented Virtuality and the Virtual Reality. Ideally, Mixed Reality integrates your real and digital worlds seamlessly, enables intuitive and natural interactions and provides digital content is wherever needed.

Head Mounted Displays in Surgery

Stacks Image 610
So far, Head Mounted Displays (HMD) were bulky, heavy devices and their displays had an inadequate resolution. Furthermore, hiding the real world Virtual Reality HMD are not allowing the surgeon to directly see the surgical site and(or the co-workers in the operation room: This limited their use significantly.
The recent availability of high-resolution displays, tremendously faster processors, and – most importantly – the development of Mixed Reality technology will dramatically change the way surgeons might use this technology in the future.

Together with the Interdisciplinary Laboratory Image Knowledge Gestaltung at the Humboldt-Universität zu Berlin, the Charité’s Department of Radiology and private companies we are currently working on

  • Preoperative surgical planning in visceral surgery (VR, MR),
Stacks Image 640
  • use of Mixed Reality systems in the operating room (MR assisted surgery, tele-consulting & mentoring) and
  • Training of surgical technicians via 360° videos and immersive 3D spaces.

Head Mounted Displays – current hardware under evaluation

Microsoft HoloLens
Microsoft HoloLens is a pair of Mixed Reality head-mounted smart-glasses made up of certain components that together enable holographic computing. The optical system works in lockstep with advanced sensors. The HPU that makes light work of processing a large amount of data per second. All those components and more enable you to move freely and interact with holograms.
Sensors and related hardware, including the cameras and processors are located in the front of the device. Main component is a pair of transparent lenses, in which the projected images – the holograms – are displayed. The device features an inertial measurement unit (IMU) with gyroscope, accelerometer and a magnetometer), four „environment understanding cameras“, a depth camera, a 2.4-megapixel photographic video camera, a four-microphone array, and an ambient light sensor.
Interaction is provided via spatial sound, gaze tracking, gesture input and voice support.

Meta 2
The Meta 2 development kit enables holographic mixed reality experiences, too. The headset displays holograms and has a larger field of view (90°), when compared to the HoloLens. The technology behind is simpler: the device uses a 2550x1440 resolution display mounted above the plastic shield which serves as the semi-translucent medium to reflect light from the display into your eyes. 6-axis IMU and an array for hand interactions and positional tracking, a 720p front-facing camera, and speakers are integrated into the HDM.
In order to operate the device an external PC with powerful CPU and graphics card is needed. In comparison to the HoloLens it appears to be less autonomous and versatile but offers a larger field of view.

HTC Vive
HTC Vive is a virtual reality headset, based two OLED screens with a resolution of 2160x1200 (1080x1200 per eye). Orientation in the room is based on sensors within the HMD including a MEMS gyroscope, accelerometer and laser position sensors. A single front-facing 720p camera can be used for superimposing a video-stream of the real world in front of the HDM into virtual reality. The HDM is operated in a 4.6 meters by 4.6 meters tracking space defined with two "Lighthouse" base stations. Emitting pulsed IR lasers these bases stations are used to track the movement of the headset and corresponding input-devices. Similar to the Meta 2 the device is connected to a powerful PC via HDMI and USB-cables.
Stacks Image 620