Brain surgery training from an avatar

Benjamin Warf, a renowned neurosurgeon at Boston Children’s Hospital, stands within the MIT.nano Immersion Lab. Greater than 3,000 miles away, his virtual avatar stands next to Matheus Vasconcelos in Brazil because the resident practices delicate surgery on a doll-like model of a baby’s brain.

With a pair of virtual-reality goggles, Vasconcelos is capable of watch Warf’s avatar reveal a brain surgery procedure before replicating the technique himself and while asking questions of Warf’s digital twin.

“It’s an almost out-of-body experience,” Warf says of watching his avatar interact with the residents. “Perhaps it’s the way it feels to have an analogous twin?”

And that’s the goal: Warf’s digital twin bridged the space, allowing him to be functionally in two places directly. “It was my first training using this model, and it had excellent performance,” says Vasconcelos, a neurosurgery resident at Santa Casa de São Paulo School of Medical Sciences in São Paulo, Brazil. “As a resident, I now feel more confident and cozy applying the technique in an actual patient under the guidance of a professor.”

Warf’s avatar arrived via a brand new project launched by medical simulator and augmented reality (AR) company EDUCSIM. The corporate is a component of the 2023 cohort of START.nano, MIT.nano’s deep-tech accelerator that gives early-stage startups discounted access to MIT.nano’s laboratories.

In March 2023, Giselle Coelho, EDUCSIM’s scientific director and a pediatric neurosurgeon at Santa Casa de São Paulo and Sabará Children’s Hospital, began working with technical staff within the MIT.nano Immersion Lab to create Warf’s avatar. By November, the avatar was training future surgeons like Vasconcelos.

“I had this concept to create the avatar of Dr. Warf as a proof of concept, and asked, ‘What could be the place on this planet where they’re working on technologies like that?’” Coelho says. “Then I discovered MIT.nano.”

Capturing a Surgeon

As a neurosurgery resident, Coelho was so frustrated by the shortage of practical training options for complex surgeries that she built her own model of a baby brain. The physical model accommodates all of the structures of the brain and might even bleed, “simulating all of the steps of a surgery, from incision to skin closure,” she says.

She soon found that simulators and virtual reality (VR) demonstrations reduced the training curve for her own residents. Coelho launched EDUCSIM in 2017 to expand the variability and reach of the training for residents and experts trying to learn recent techniques.

Those techniques include a procedure to treat infant hydrocephalus that was pioneered by Warf, the director of neonatal and congenital neurosurgery at Boston Children’s Hospital. Coelho had learned the technique directly from Warf and thought his avatar could be the best way for surgeons who couldn’t travel to Boston to learn from his expertise.

To create the avatar, Coelho worked with Talis Reks, the AR/VR/gaming/big data IT technologist within the Immersion Lab.

“Lots of technology and hardware might be very expensive for startups to access as they begin their company journey,” Reks explains. “START.nano is a method of enabling them to utilize and afford the tools and technologies we’ve at MIT.nano’s Immersion Lab.”

Coelho and her colleagues needed high-fidelity and high-resolution motion-capture technology, volumetric video capture, and a variety of other VR/AR technologies to capture Warf’s dexterous finger motions and facial expressions. Warf visited MIT.nano on several occasions to be digitally “captured,” including performing an operation on the physical baby model while wearing special gloves and clothing embedded with sensors.

“These technologies have mostly been used for entertainment or VFX [visual effects] or CGI [computer-generated imagery],” says Reks, “But it is a unique project, because we’re applying it now for real medical practice and real learning.”

One in every of the largest challenges, Reks says, was helping to develop what Coelho calls “holoportation”— transmitting the 3D, volumetric video capture of Warf in real-time over the web in order that his avatar can appear in transcontinental medical training.

The Warf avatar has synchronous and asynchronous modes. The training that Vasconcelos received was within the asynchronous mode, where residents can observe the avatar’s demonstrations and ask it questions. The answers, delivered in a wide range of languages, come from AI algorithms that draw from previous research and an intensive bank of questions and answers provided by Warf.

Within the synchronous mode, Warf operates his avatar from a distance in real time, Coelho says. “He could walk across the room, he could seek advice from me, he could orient me. It’s amazing.”

Coelho, Warf, Reks, and other team members demonstrated a mix of the modes in a second session in late December. This demo consisted of volumetric live video capture between the Immersion Lab and Brazil, spatialized and visual in real-time through AR headsets. It significantly expanded upon the previous demo, which had only streamed volumetric data in a single direction through a two-dimensional display.

Powerful impacts

Warf has a protracted history of coaching desperately needed pediatric neurosurgeons all over the world, most recently through his nonprofit Neurokids. Distant and simulated training has been an increasingly large part of coaching because the pandemic, he says, although he doesn’t feel it should ever completely replace personal hands-on instruction and collaboration.

“But when in actual fact sooner or later we could have avatars, like this one from Giselle, in foreign places showing people the best way to do things and answering questions for them, without the fee of travel, without the time cost and so forth, I believe it could possibly be really powerful,” Warf says.

The avatar project is very necessary for surgeons serving distant and underserved areas just like the Amazon region of Brazil, Coelho says. “It is a technique to give them the identical level of education that they’d get elsewhere, and the identical opportunity to be in contact with Dr. Warf.”

One baby treated for hydrocephalus at a recent Amazon clinic had traveled by boat 30 hours for the surgery, based on Coelho.

Training surgeons with the avatar, she says, “can change reality for this baby and might change the longer term.”