Holoportation: At The Van(guard) Of Healthcare

A patient had waited more than two decades to see a medical specialist. When the day finally came, he was not in a hospital but in the back of a van parked at a clinic in Koforidua, a regional hub in Ghana’s Eastern Region, and being examined by experienced reconstructive surgeons in Scotland, Rwanda and Brazil at the same time. What makes this appointment quite remarkable is that this isn’t futuristic, cutting-edge technology. There is no fiberoptic cable feeding the consultation, no operating theater or six-figure imaging suite, only a cluster of off-the-shelf cameras, two standard computers and a mobile phone signal.  

This is how Dr Kwame Darko, a consultant plastic surgeon at Korle Bu Teaching Hospital in Accra, is assessing patients who rarely get to a specialist. By his own count, Ghana only has about 30 plastic surgeons for a population of 35 million, most of them clustered in a handful of teaching hospitals. (Korle Bu’s plastic surgery unit has had a partnership with Glasgow Royal Infirmary since the 1990s, with visiting surgeons flying back and forth to Accra to help on complex cases.) 

The road ahead 

To see a plastic surgeon in Ghana, a patient would typically leave their village for a bigger town, then wait for a bus that might run only once or twice a day. “So, if I get a vehicle from my village and get to the town, and I miss the bus, that means I need to wait till the next day before I can continue,” says Darko, who adds that from the far north, the trip to Accra can take up to three days. And this is only for the first visit, which in regular circumstances is the consultation that happens before a surgery is booked. “The first day I see you, I would do an assessment,” Darko says. “Once I’ve seen that, I would order investigations, CT scans, MRIs, X-rays, blood tests, then you’d come back and see me.” It is only after this initial consultation that the anaesthetist can sign them off as fit, and only then is a surgery date set.  

According to Darko, clinics usually run once a week, so each review means waiting until the next one. With round trips of up to 14 hours each way, and most patients traveling with a caregiver (often the family breadwinner), costs quickly escalate. With a 3D consultation done closer to home, Darko says some patients can avoid several of these long, expensive trips and make the journey to Accra only when surgery is actually imminent. Of course, this scarcity of specialists is not unique to Ghana. According to the World Health Organization’s Regional Office for Africa, the continent currently has only about one-third of the medical doctors needed to meet its health needs.  

“Even in some parts of the U.S. and Canada, you still have people who need to make exceedingly long journeys to access higher-level care,” adds Darko. 

The third dimension 

Telemedicine is not new. Two-dimensional video calls between a doctor and a patient have been possible for years. The problem, in a speciality like plastic surgery, is that flat footage isn’t enough. “It’s not just about looking at you and then turn left, turn right,” says Darko. “It’s about looking at you, bend your knee, turn it this way, flex your wrist, extend your wrist, open up your fingers, close your fingers. I need to be able to see all of that.” Before removing a sarcoma from a thigh, for example, he needs to know which muscle groups it involves and afterwards, which ones are still working. That is a spatial assessment, something a webcam alone cannot do.  

Dr Spencer Fowers, a principal researcher on Microsoft Research’s Special Projects team, saw the gap first-hand when he shadowed Glasgow surgeon, Dr Steven Lo, whose long-standing partnership with Korle Bu set the whole project in motion. (Lo had already secured a small grant from the University of Strathclyde to build an early prototype in Glasgow and replicate it in Ghana; after he published that work, he says someone at Microsoft spotted the idea and offered to help scale it.) 

The obvious fix would be to hand a nurse a 2D-camera and have them walk around a patient, but that turned out to be easier said than done. “Trying to explain to a patient with limited mobility how to move their body, while explaining to a nurse how to move a camera, it’s almost impossible to do that,” says Fowers. “Now imagine adding to that the fact that you’re in Glasgow and they’re in Accra.” This is the problem Holoportation was built to solve. Developed by Microsoft, the system uses six to 10 cameras to capture the patient from every angle and a single computer to stitch the feeds into a live 3D model. The surgeon, not the patient, does the moving, rotating and zooming through the image on screen to see exactly what they need. Fowers recalls a patient during the pandemic who had had surgery on her back and shoulder. She said that this was the first time she was able to see what had been done to her, because she had never been able to see her own back.  

Shrunk and shipped 

The technology underneath is Holoportation, a live 3D-capture system. In its original form, it was a research showpiece that relied on expensive research-grade cameras, banks of graphics processors and a fiberoptic connection to move the data. It was impressive, but almost impossible to deploy anywhere that actually needed it. “We took what was basically a research prototype… and condensed it down,” says Fowers. The hardware now contains off-the-shelf parts and a single graphics card that does the processing on site. This is key because the heavy work happens on the device itself rather than in the cloud, so only a light video stream needs to be sent over the internet. When still images and recordings are stored on local drives, less bandwidth is needed and patient data stays under the control of the hospitals actually using it. “As long as the hospital or the clinic has the infrastructure to support just a basic 2D-video call, they can get the benefits of a 3D-consultation,” Fowers explains, adding that in Accra, it runs over 4G. “You can package this all up in the back of a van that can drive out into the middle of sub-Saharan Africa, and as long as you have a cell phone connection, you can perform this consultation,” says Fower.  

The mobile rig that reached Koforidua, a district with no plastic surgeon of its own, was built for well under $100,000. Four or five patients sat in it that day; two, a young person whose burned arm had healed with the elbow locked shut, and an elderly man with a sarcoma, went on to surgery they might otherwise never have reached. “We could have solicited for lots of money and made it extremely expensive, high-tech cameras and extra gadgets,” says Darko. “But we wanted to see how far we can go on a significantly low budget.”  

Open to all 

Ease of use was a big part of the design brief–if you can run a Teams meeting and share a screen, you can run a full consultation. A surgeon can draw directly onto a 3D image of a patient’s body to show them exactly where a skin flap will come from and where it will go. Darko says that this has helped to relieve anxiety in patients and as an added bonus, those who were so desperate for specialist medical care aren’t afraid of the technology. “They’re really embracing ,” he says. “To them, it’s the very next step to getting their problem solved.” Something else that surprised Darko is how far the word has spread. At a breast reconstruction workshop in Ghana, a visiting surgeon from Canada recognized his name immediately and asked him about the project. “I didn’t realize so many people had actually read about it,” he says, “even if it wasn’t directly in their field.” 

At the end of last year, the team chose to release the entire code base under an open-source licence, off-the-shelf hardware list and all, so anyone can build their own telemedicine van or adapt the tool to a speciality no one has thought of yet. Fowers says that both remote physiotherapy and orthopaedics are possibilities. Being open-source and still in the research phase, Darko’s hope is that this this tool will extend far beyond Ghana’s borders. He is also watching AI, which he says could one day let a patient see not just where the surgeon will cut but what they will look like afterwards. “We’re very far away from the world ever having enough doctors in every speciality,” he says. “This is one way to fill that gap. You don’t physically need the specialist in that locality. They can log on.”