6 Sept 2023 • Through the use of a brain implant and digital avatar, researchers from the University of California, San Francisco have created a novel method for persons with locked-in syndrome to communicate and display facial expressions.

People with locked-in syndrome usually communicate by blinking and moving their eyes. These days, they can communicate with people using computer programs and other assistive technology such as infrared eye movement sensors, head-mouse devices, and brain-computer interfaces. Additionally, new communication alternatives for people with locked-in syndrome have been made available owing to the advancements in computer engineering and cutting-edge technologies like artificial intelligence (AI).

Speech neuroprostheses have the potential to help people with paralysis regain their ability to communicate, but naturalistic speed and expressivity are difficult to achieve. Researchers from UCSF were able to accomplish high-performance real-time decoding across three complimentary speech-related output modalities: text, speech audio, and facial-avatar animation, using high-density surface recordings of the speech cortex in a clinical trial subject with severe limb and vocal paralysis.

The new technology was tested on a 47-year-old woman named Ann, who was diagnosed with locked-in syndrome following a brain stem stroke.

• Artificial intelligence algorithms were designed to recognize and respond to her unique brain signals for speech for weeks.
• A digital avatar of Ann was also created through software that simulates and animates facial muscle movements.
• The researchers were also able to recreate Ann's actual voice using audio from a pre-injury film so that when she communicates via the digital avatar, it is her voice rather than a standard computer voice that is audible.

These findings established a multimodal speech-neuroprosthetic approach that has shown significant potential to help people living with severe paralysis, regain full, embodied communication.

Source: Nature | Read full story