Thanks to the rise of mixed reality and 3D holography technology, holograms have made the leap from the realms of science fiction into reality.

In fact, since 2021, holograms have already made their way into the consultation rooms and operating theatres of Singapore’s National University Hospital (NUH).

Using Microsoft’s mixed reality (MR) headset HoloLens 2, NUH clinicians can view highly precise 3D hologram visualisations of virtual objects (such as MRI scans or anatomy), as well as interact with these holograms using hand gestures.

Dr Gao Yujia

Holograms have proven useful for surgical planning, as well as during surgeries as an additional guide or supplement for surgeons, said Dr Gao Yujia, a Consultant in the Division of Hepatobiliary & Pancreatic Surgery, under the Department of Surgery at the NUH. He is also the Assistant Group Chief Technology Officer of the National University Health System (NUHS), overseeing the development of the academic health system’s Holomedicine programme. An example of how holograms are being used is during brain surgery, where surgeons donning MR devices can superimpose the brain scan directly over the patient’s head to help better locate a brain tumour, as opposed to the current 2D projection on a computer screen.

 

The use of holograms in medicine or clinical care is referred to as “Holomedicine”. It falls under the growing Extended Reality (XR) field, which encompasses all technologies that alter reality with digital elements.

So far, this technology has been used in over 100 surgical cases in NUH and Ng Teng Fong General Hospital under the NUHS, across nine specialisations such as neurosurgery, cardiac, liver transplant, endocrine and facial reconstruction surgeries.

Besides surgical use cases, holograms have also proved useful in patient counselling, with visualisations helping patients better understand their condition and surgical procedures and relieving their anxiety.

 

Pioneering holomedicine in Asia

“NUHS is probably the only healthcare system, not just in Singapore but across the region, that is using holomedicine technology at a productionised scale. I think in terms of the programme size, deployment, and maturity, we are probably one of the leading hospitals in the world,” Dr Gao noted.

NUHS’ exploration of holomedicine began back in 2019, when the team was invited to a workshop by Microsoft to experience the new HoloLens 2 MR device.

“What we brought home from that session was that there’s real potential for this technology (in healthcare). There was also, of course, the initial wow factor when we put on the device and are able to see the holograms and control them with our hands – those are things we see in movies which have now come to life.”

With their curiosity and interest piqued, the team started a six-month proof of concept trial of the MR devices around August 2020. The device was brought to doctors, nurses and other healthcare professionals across different specialities for testing and feedback.

At the end of the trial, feedback collected was unanimously positive. Clinicians understood that while the platform is not the most mature, it held huge potential in improving patient care, noted Dr Gao.

With these results in hand, the NUHS Holomedicine Programme was formally approved by the senior management of the academic health system in 2021, paving the way for the launch of a dedicated Holomedicine unit staffed by clinicians, engineers, developers, data scientists and network architects, as well as kick-off discussions with various specialties on suitable use cases.

 

Interoperability as a key challenge

With little to no precedent cases to refer to, the implementation of this novel technology was not without its challenges.

“Most of the medical use cases for Holomedicine are not out of the box solutions, and that’s why we needed our own development team to create the applications and interfaces that are relevant for the medical field and our hospitals. One of the biggest challenges was to adapt the MR tools to what we needed to do,” said Dr Gao.

Another challenge lies in interoperability. Essentially, the MR devices need to be able to connect to the necessary data sources, which in the case of healthcare could include ultrasound probes and other imaging equipment.

“Companies or research teams working on mixed reality devices in healthcare are trying to create applications that can be used across different use cases, modalities, or devices, so the number of people who can use it is not limited. This is still work in progress,” Dr Gao explained.

“It’s also important to ensure the device is easy to adopt and implement. We should not have to create new processes or workflows just so that we can use this technology; it should be the other way around. The technology should be able to easily fit into existing workflows, so there is minimal disruption to what we are already doing on a day-to-day basis.”

 

Managing potential risks to safeguard safety

As with any technology tools, there are potential risks of usage. To manage these risks and safeguard safety and quality of care, NUHS has taken several steps, shared Dr Gao.

Firstly, onboarding and introductory sessions are held regularly for new users who want to start using the HoloLens. They are also encouraged to bring one home to further familiarise themselves with it.

Subsequently, for their first two to three attempts of use, a member of the NUHS Holomedicine team will be on-site to provide support if needed. Remote support will be provided once the user becomes familiar with the device and no longer requires on-site support.

The team is identifying specialty champions, or ‘super users’ familiar with the technology within each department, who can help train and support the rest of their colleagues moving forward.

Crucially, it is emphasised to surgeons that they should not solely rely on the MR device for surgical navigation.

“The main risk of the MR devices is the accuracy of the images that you see. For surgeons, the device should serve just as a guide to plan their surgery. They still have to make a decision on whether the holograms match what they see in front of them. If they feel there are any errors or deviation, there is always the fall-back option of going back to what was routine without the HoloLens,” Dr Gao pointed out.

 

What holomedicine could mean for healthcare

Looking ahead, the team is looking at expanding Holomedicine to Alexandra Hospital (another hospital in the NUHS academic health system), other healthcare clusters in Singapore, and healthcare institutes worldwide. They are also exploring other use cases, such as “live-streaming” of ultrasound images from imaging equipment to the HoloLens – so doctors can view the 3D images right as the scans are being conducted.

Another example is real-time 3D environment mapping. With cameras streaming 3D feeds across the room, healthcare professionals located remotely will be able to see a 3D holographic projection of the entire operating theatre, including the surgeons and the patient, as if they are in the room themselves. This would help enhance their field of sight and situational awareness when providing remote assistance, as compared to video calls.

But a priority would be in validating the existing use cases – working together with other hospitals across the world to pull together data from their trials of holomedicine technology to prove its effectiveness, noted Dr Gao.

“For this to be accepted as mainstream or standard of care in medicine, we need to have evidence to support its benefits. We have to prove its safety and efficacy, and subsequently its superiority, or how it is better than what we are doing now without it.”

Eventually, Dr Gao envisions mixed reality devices to be part of an integrated ecosystem of numerous devices and solutions, interconnected for seamless, efficient care delivery.

“At the end of day, MR devices cannot exist as silos within the healthcare sector,” said Dr Gao.

“We are seeing mixed reality moving away from simply a holographic display device, towards an integrated platform for data from multiple sources. The front-end hardware and software is definitely important, but unless we also have a robust and capable back-end infrastructure – such as cloud computing, edge computing, machine learning, and strong wireless connectivity – you’ll never have true integration into the healthcare ecosystem.”

“The next few years will definitely be very exciting with a host of new XR devices coming out and developments in 5G, AI and machine learning. Many technologies at the proof of concept stage are reaching the implementation or early adoption stages, so we can look forward to seeing a lot of these becoming the standard of care and transforming healthcare delivery.”