A Look into the Body – Augmented Reality in Computer Aided Surgery

Computer scientists from the TU München and surgeons from LMU develop a computer aided visualisation and navigation system. This system allows for more precise keyhole surgery. The system is to be tested clinically in the coming months.

Keyhole Surgery - Minimally Invasive Surgery

The surgeon enters the operating theatre and puts on a head mounted display. On the screen and in the head mounted display she sees the patient exactly as he lies on the operating table. But at the push of a button she will recognise much more: She can look into the body of the patient, layer by layer, looking through skin and muscles, down to the bones. Ater gaining an overview, the surgeon starts the operation. She makes a small opening in the body of the patient where she introduces the tools - endoscope, catheter or drill. Where she has to make the cut, is marked in the head mounted display or the screen. In the head mounted display she can also see where the instruments are located in the body of the patient, right between organs, bones and blood vessels, as if she had a X-ray vision.

This scenario is still a dream of the future. Today the surgeon in keyhole-surgery must rely on the information (X-rays, endoscopic camera) displayed beside the operation table. Thus the surgeon has to turn his head constantly - away from the patient and back. A reserach team of computer scientists and surgeons from the TU München has set the goal to facilitate the work in minimally invasive operations. The team works on a system for visualisation and surgical navigation, using augmented reality. "Augmented" since the real picture is augmented with additional information from the computer.

A direct look into the patient - a merger of reality and X-ray

At the chair of computer applications in medicine and augmented reality of the TU München, the team around Professor Nassir Navab engages in a number of projects. The goal is a system that shows the surgeon a 3D picture of a patient's inner body along with the instruments needed during the operation. This image is not visible on a separate display but rather can be seen directly while looking at the patient. To ensure the use of this technology the system should be user friendly.

Computer scientists get output images from the computer tomography (CT) for the virtually produced image of the inner body. A modern spiral-CT makes several X-Ray photographs with diverse perspectives and then reconstructs their 3-dimensional perspective. A computer-aided tomogram is clearer than a normal X-ray photograph because it enables differentiation of the body's various types of tissue. The computer scientist then superimposes the saved CT scans with a real photo of the patient on the operating table. For surgeons the impression produced is that of lookinv through the skin and throughout the various layers of the body in 3-dimensions and in color.

Head Mounted Display and Tracking System - Complicated Technology for the Operation Theatre

The physician sees this half real, half virtual image in the display of his head mounted display (HMD). In the HMD two colour cameras are integrated, which record the live scene. They are mounted slightly apart, like two eyes. Thus the compound picture of both cameras becomes three-dimensional. This live scene and the stored CT-pictures can be superimposed. To superimpose the two images correctly, the computer program must know at any time, where the physician and the patient are, where the physician looks and how he/she moves. This task is accomplished with a tracking system that follows points of reference in infrared, attached to the HMD and the patient. The tracking system works precise to fractions of millimeters. If the surgeon turns his/her head, the tracking system makes sure that the computer picture of the inside of the body corresponds accurately to the surgeon's point of view. In addition it tracks the position of the instruments in the surgeon's hand and computes its exact position in the body of the patient. For the tracking system at least two additional infrared cameras are needed. They are mounted on a scaffold surrounding the operating table.

This structure represents still another difficulty of the system, explains Christian Bichlmeier, scientist at the chair: „The Trackingsystem is very expensive and complex to adjust.“ How to integrate it in the operating room is another unresolved problem. The computer programme itself however is already relatively far. The researchers first worked with dolls made out of plastic. These dummies with a anatomically correct structure replaced the human body. Further attempts accomplished with dead animals. First live tests with humans were conducted: Two patients, made their CT-pictures available. The scientists could test whether the programme projects the pictures correctly onto the body.

Continuous Feedback – Cooperation between Informatics and Medicine

Whether the project is helpful and useful for the surgeons is continuously monitored by computer scientists. Development and tests occur in close co-operation with surgeons from the surgical clinic at LMU. Thus the researchers get constant feedback to adjust the system to the needs of the surgeons. Team work is easy, since the researchers' laboratory is located directly beside the operation theatre. Christoph Bichlmeier stresses the importance of the close settings: "As computer scientists we can watch the operations and thus understand the processes in surgery." Surgeons on the other hand, stop by to test the application. So a mutal understanding of the two disciplines develops. Professor Nassir Navab, full professor, explains the close cooperation: "As soon as a version is finished, surgeons will try it and will suggest changes. This immediate feedback is crucial for our work."

Precision Via Broadened Perspectives - The Virtual Mirror

In another chair project another application was developed: The virtual mirror. It can be used in addition to surgical instruments. It shows an additional perspective of the virtual body, namely its back. This has its advantages in operations where the highest precision is crucial, e.g. when screws are set into vertebrae. By use of the virtual mirror the surgeon sees the back of the vertebra and recognizes where the drill will exit, depending on the angle used. The optimal drilling angle is determined before the operation and is projected into the HMD. With a small twist of the instrument the surgeon can rotate the virtual mirror to control the position of the drill.

Orthopaedic operations are only one field of application for augmented reality. Other chair projects deal with the removal of tumors, visualisation of blood vessels during laparoscopic operations. Here there exists a co-operation with surgeons of the TU München. The HMD shall be used for the training of surgeons and students. HMD could also play a role in the planning of operations. The first clinical studies are planned in about two years. By the time an HMD can be used as a medical product, about eight years will pass says Professor Navab.

Professor Ekkehard Euler is head of trauma surgery at LMU's clinic. He tried the system. He can well imagine using an HMD during an operation one day: "Of course one must train to use the system. But this is also the case for any new prodedure. I think, the system is especially useful for training."

The C-Arm with Cameras - Enhancement of Existing Technology

Another development from the chair of Augmented Reality is one big step ahead: The researches have enhanced an imaging method, the so-called C-arm, to be used in augmented reality. The technique will be tested in the operation theatre in the months coming.

The existing C-Arm is a movable bow with an X-ray camera. It can circle around a patient, providing images from all angles. The pictures are then shown on a screen beside the operating table. The C-arm is used for the planning of operations as well as for control during operations. For example, whan a surgeon is fixing a complicated fracture, the C-arm is continously making new images from different perspectives. This helps the surgeon to orient himself and to control the progress of the operation. Alas the method has one big disadvantage: during one single operation up to twelve X-ray images may be needed. This means high radiation exposure for the patient and especially for the medical staff that is working with the C-arm on a daily basis.

The idea of the researchers was to now enhance the C-Arm with a regular camera which takes the same photograph as the X-ray apparatus. The doctor marks the spot for a minimally invasive surgery on the X-ray photo. The marking is saved and transmitted to the live camera image. When the surgeon uses his tool he can follow it on the screen to see where it is in relation to the marking and to the X-ray photo. A second camera is in the X-ray camera's right angle and is affixed to the C-Arm and to the first camera. It delivers important additional information, for instance about the gradient and penetration depth of the instrument. The advantage of both additional cameras attached to the C-Arm: The doctor can follow the position of the intrument on the screen and connect anytime to the X-ray image. In the ideal case the doctor can conduct the operation with a single X-ray photograph. The amount of radiation exposure is drastically reduced.

Reduction in Radiation Exposure– Clinical use is imminent

Computer scientists refer to the new technology as CAM-C: Camera Augmented Mobile C-Arm. In contrast to the expensive and complex tracking systems for data helms this upgrade must simply be converted: The existing technology would only be enhanced by two visual cameras. It signifies no big change for the doctors and does not necessitate any training. "Every surgeon can utilise the equipment immediately", says Professor Euler. The cameras could simply be connected as an additional navigation aid to the existing C-Arm. If the surgeon would rather waive this option he/she can simply disconnect it.

It is important that the X-ray and camera photographs coincide exactly so that they both show exactly the same section. Computer scientists can achieve this by superimposing the camera's visual focus and the X-ray source. The different optical distortions will be evened out with software. "The system makes precision work possible, for example, for the drilling of holes in the spinal column, when an injury of the nerves and arteries are a threat", explains Professor Euler. In trauma surgery this would include fractures of the vertebral body or of the hip joint. Professor Euler summarizes the advantages of the C-Arm with a fixed camera: "It is safer, faster and means less radiation exposure for patients as well as for the OR staff."

The system has already tested on synthetic models and on animal cadavres and found to be safe. Clinical studies on humans should already be conducted in a few months in order to make future routine use of the enhanced C-Arm in the operating room possible.


Bichlmeier, Christoph et al.: Virtually Extended Surgical Drilling Device: Virtual Mirror for Navigated Spine Surgery.
Bichlmeier, Christoph et al.: Contextual Anatomic Mimesis. Hybrid In-Situ Visualization Method for Improving Multi-Sensory Depth Perception in Medical Augmented Reality. 
Computer Aided Medical Procedures & Augmented Reality (CAMP), Technische Universität München
Merten, Martina: Verschmelzung zweier Welten. In: Deutsches Ärzteblatt, Jg. 104, 2007, Heft 13.
Navab, Nassir et al.: Action- and Workflow-Driven Augmented Reality for Computer-Aided Medical Procedures. In: IEE Computer Graphics and Applications, Jg. 2007, Heft September/Oktober.
Navab, N., Bani-Hashemi, A., Mitschke, M.: Merging visible and invisible: two camera-augmented mobile C-arm (CAMC) applications. Proceedings, 2nd IEEE and ACM International Workshop on Augmented Reality (IWAR'99); Okt. 1999; S. 134-141.
Schröder, Tim: Das Navi für den Körper. In: Faszination Forschung, Jg. 2008, Heft 2.
Traub, Jörg et al.: A Multi-View Opto-Xray Imaging System. Development and First Application in Trauma Surgery.

Lehrstuhl für Informatikanwendungen in der Medizin & Augmented Reality, Computer Aided Medical Procedures (CAMP), Technische Universität München (http://campar.in.tum.de/Chair/ResearchGroupCamp)

NARVIS Projekt (http://www.narvis.org)

Photos: Lehrstuhl für Informatikanwendungen in der Medizin & Augmented Reality, Technische Universität München (11), DAK/Scholz (1), Shutterstock (1)Filme: Lehrstuhl für Informatikanwendungen in der Medizin & Augmented Reality, Computer Aided Medical Procedures (CAMP), Technische Universität München (http://campar.in.tum.de/Chair/ResearchGroupCamp), Prof. Dr. E. Euler, Chirurgische Klinik Innenstadt, Ludwig-Maximilians-Universität München