GE Mixes 3-D and Live Surgery Video

Researchers at General Electric and Brigham and Women's Hospitalin Boston have developed a way to superimpose a live video imageof the brain during surgery on a three-dimensional model of apatient's brain reconstructed from MR data. This is not virtualreality, the engineers and physicians emphasize. It is enhancedreality--a subtle but important difference.

The combined video and MR image provide an other-worldly senseof looking in real time at structures that lie beneath the surfaceof the brain.

The technology was put to the test recently as an aid in theremoval of a large tumor from a patient's brain. A portion ofthe skull was removed to give surgeons access to the tumor.

Using the combination of live video and computer-generated3-D image, they "looked" through the skull and sheathsof neural tissue to the tumor they could not otherwise have seen.The live video showed not only the patient and 3-D model but thesurgeons' hands and surgical instruments.

"The mixed image adds a degree of reality by allowingsurgeons to examine the underlying anatomy represented by the3-D image in the context of the actual anatomy," said WilliamLorensen, a graphics engineer at the GE research and developmentcenter in Schenectady, NY.

A similar technique was used prior to surgery, when the 3-Dmodel of the brain was superimposed onto a live video of the patient'shead. With the combined TV image as a guide, the surgeons tracedon the patient's head the location of the tumor and critical structuresto be avoided in surgery.

The most crucial of these was the motor strip of nerve tractsthat controls the body's motor activities. The surgeons watchedon the TV monitor as their hands sketched in the outlines of thestructures.

This technique is somewhat similar to virtual reality in thatit comprises a series of technologies designed to immerse onein a computer-generated world. But it is different from virtualreality because the computer images provide references to thereal world rather than completely supplanting it.

Another distinction between enhanced and virtual reality isthe way the technology is used. In virtual reality, technologyplays a high-profile role.

"In our context, the main issue is not the user interface,"said Dr. Ron Kikinis, director of the Brigham and Women's surgicalplanning group and an assistant professor of radiology at HarvardMedical School. "On the contrary, we want (the technology)to be as unobtrusive as possible."

The researchers use mostly off-the-shelf products--a SUN Microsystemsworkstation and standard video equipment--integrated with a special-purposegraphics accelerator board built by GE. Underlying the hardwareare two imaging algorithms, patented by Lorensen and Harvey Cline,a GE physicist.

The algorithms permit 3-D models to be rapidly constructedfrom conventional CT and MR slice data. Additionally, the GE/Brighamand Women's team has developed a "segmentation algorithm"that allows tumors to be distinguished from other types of softtissue.

In the recent surgery, the tumor to be removed was coloredgreen while the surrounding tissue was gray. Its volume was alsocalculated, allowing surgeons to compare the volume of the tumorremoved during surgery to the volume calculated from the 3-D model,as a check to assure that as much of the tumor had been removedas possible.