Pager Plays MindPong | Blog

(Fig 01)

Pager playing MindPong

Neuralink is developing a fully-implanted, wireless, high-channel count brain-computer interface (BCI) with the goal of enabling people with paralysis to directly use their neural activity to operate computers and mobile devices with speed and ease.

If You Are Just Joining Us...

In a 2019 white paper, we outlined the design of our novel electrodes and our unique surgical approach, along with preliminary electrophysiology obtained in a rodent model. That generation of the Link had wired leads and a connector that protruded through the skin, and was an important platform for developing and validating our robotic surgical approach and our ultra low-power custom application-specific integrated circuits (ASICs) for amplifying and processing neural signals. In 2020, we publicly shared a wireless version of the Link that was able to stream 1,024 channels of action potentials (also called “spikes”) wirelessly and in real time (Fig. 02). We demonstrated its functionality by recording somatosensory (touch) signals in pigs exploring their environment. The electrodes were placed in a part of the brain involved in processing signals from the pig’s exquisitely sensitive snout. As it snuffled about, the responses of the neurons to sensory cues could be readily observed.

Implant Exploded Diagram — (Fig 02)
An exploded view of our wireless Link implant

(Fig 03)

Snout Boops

Neural data is wirelessly transmitted from a Link that was placed in the somatosensory cortex of a pig (named Gertrude). The modulation of the neural signals is visually and audibly apparent.

This demonstration was a small but important step towards our vision of providing direct neural control of a computer cursor to people with paralysis. Swine will continue to be an important animal to validate the safety of the Link. However, to develop and advance the functionality of the Link, it is necessary for us to employ an animal model whose brain similarity (homology) and behavioral abilities enable the development of a hand and arm-based motor cortical BCI. The rhesus macaque model allows us to design, validate, and advance the performance and robustness of a complete “closed-loop” motor BCI system intended to improve the quality of life of people with neurological disorders.

Wireless, Fully Implanted BCI

Today we are pleased to reveal the Link’s capability to enable a macaque monkey, named Pager, to move a cursor on a computer screen with neural activity using a 1,024 electrode fully-implanted neural recording and data transmission device, termed the N1 Link. We have implanted the Link in the hand and arm areas of the motor cortex, a part of the brain that is involved in planning and executing movements. We placed Links bilaterally: one in the left motor cortex (which controls movements of the right side of the body) and another in the right motor cortex (which controls the left side of the body).

Neurons in somatosensory cortex respond to touch, and neurons in the visual cortex respond to visual cues. Analogously, neurons in motor cortex modulate their activity prior to and during movement, and are thought to be involved in planning, initiating and controlling voluntary movements. Many neurons in motor cortex are directionally tuned, that is, more active for particular movement directions than others. Different neurons are tuned to different movement directions. An example of this directional modulation can be seen in the raster plot figure (Fig. 04).

A view of a spike raster — (Fig 04)
Modulation of neural activity with (intended) movement. Each row in the top panel represents the neural activity recorded from one electrode. The top 100 electrodes with the strongest upward preferred directions are shown in blue, and the 100 electrodes with the strongest downward preferred directions are shown in red. Brighter colors indicate higher firing rates. In the bottom trace, the yellow line indicates the vertical velocity of the MindPong paddle that results from the decoded neural activity in the top panel. (This plot is also shown as a real-time animation in the movie below.)

By modeling the relationship between different patterns of neural activity and intended movement directions, we can build a model (i.e., “calibrate a decoder”) that can predict the direction and speed of an upcoming or intended movement. We can go further than simply predicting the most likely intended movement given the current pattern of brain activity: we can use these predictions to control, in real time, the movements of a computer cursor, or in the video below, a MindPong paddle. The neurons with upward preferred directions clearly increase their firing rates as the monkey moves his MindPong paddle upward, and the ones with downward preferred directions increase their firing rates as Pager moves his paddle downward.

(Fig 05)

Extended MindPong with neural signals overlay

Inset shows his concomitant Link-recorded neural activity across 200 channels that are well-modulated to vertical movements (top panel) and the Y-velocity decoded in real time from that neural activity (bottom trace), which is used to control the movements of Pager’s MindPong paddle in real time.

Decoding Neural Signals

The Link amplifies and digitizes the voltage recorded from each of its 1024 electrodes. These tiny voltage traces contain signatures of the activity of nearby neurons (called action potentials or “spikes”). Custom algorithms running aboard the Link automatically detect spikes on each electrode, which are then aggregated into vectors of spike counts [1 count every 25 ms x 1024 channels]. Every 25 milliseconds, the Link transmits these spike counts over bluetooth to a computer running custom decoding software. First, this software re-aggregates the spike counts at several timescales, from the most recent 25 ms to the past 250 ms, to account for differing temporal properties in the activity of the motor neurons. Next, the weighted sum of these current and recent spike counts are computed for each dimension of control by passing their firing rates through a decoding model. The output of the decoder is a set of velocity signals for each 25 ms bin, which are integrated over time to direct the movement of a cursor (or MindPong paddle) on a computer screen.

(Fig 06)
Graphical representation of a BCI decoding pipeline.

The video below shows the spatial pattern of directional tuning on each of the electrodes in Pager’s implant while playing a 2D target acquisition game.

(Fig 07)

Tuning on Neuralink electrodes (watch until the end)

Each circle represents one electrode. Its size represents the amount of consistent modulation with movement that is observed in the neural activity recorded by that electrode, and its color represents the “preferred direction” of movement around a circle (0 deg = to the right) that maximally activates the neurons recorded there. Image taken during surgery shows the locations where the threads penetrate the cortical surface.

On The Horizon

With monkeys, we calibrate the decoder by mapping neural activity patterns to actual (joystick) movements. However, we won't be able to use such a strategy for people with paralysis. Prior research by the BrainGate consortium have shown that neurons in the motor cortex remain directionally tuned to movement intention even in people with paralysis, and that it is possible to calibrate a decoder as the person simply imagines moving a mouse on a mousepad or a finger on a trackpad to guide a cursor that automatically moves to presented targets. After the decoder is calibrated, the person is able to type emails and text messages, browse the web, or anything else that can be done with a computer, just by thinking about how they want the cursor to move.

Neuralink's technology builds on decades of research. The BCI systems used in previous studies have no more than a few hundred electrodes, with connectors that pass through the skin, requiring a technician or caregiver to "connect" the BCI. Our mission is to build a safe and effective clinical BCI system that is wireless and fully implantable that users can operate by themselves and take anywhere they go; to scale up the number of electrodes for better robustness and higher information throughput; and to automate the implant surgery to make it as rapid and safe as possible. Recent engineering advances in the field and new technologies developed at Neuralink are paving the way for progress on each of these key technical hurdles.

Our first goal is to give people with paralysis their digital freedom back: to communicate more easily via text, to follow their curiosity on the web, to express their creativity through photography and art, and, yes, to play video games. After that, we intend to use the Link to help improve the lives of those with neurological disorders and disabilities in other ways. For example, for people with paralysis the Link could also potentially be used to restore physical mobility. To achieve this, we'd use the Link to read signals in the brain and use them to stimulate nerves and muscles in the body, thereby allowing the person to once again control their own limbs.

As you can see, MindPong is an initial demonstration of the potential capabilities of the N1 Link. However, it's important to remember that it is a small slice of what our device is intended to achieve. If you'd like to be a part of making a brighter future through neural devices, we'd love to hear from you.

Robotics

To make Neuralink implants a reality, the robot is absolutely necessary on two fronts: first, the threads are smaller than a human hair and cannot be grasped, manipulated, and accurately inserted by a human surgeon. Doing this requires micron precision for the grasp, tens of microns of precision for localizing and tracking the moving brain during insertion, and high speed in order to get hundreds of threads in quickly. Second, scaling to hundreds of thousands then millions of patients means that not only must the thread insertion be automated but also the planning and execution of the rest of the surgery.

(Fig 08)

Working on the Neuralink Robot

Animal Care

Like all animals, pigs and non-human primates deserve our utmost respect and appreciation. Every aspect of their care is thoroughly evaluated by a team of veterinarians and behaviorists, ensuring access to high quality nutrition, socialization, and enriched spaces to perform species-specific behaviors. For example, Pager lives with his best mate, Code. They enjoy swinging from their treehouse and napping in their hammocks after an engaging gaming session. All of the behavioral tasks that Pager and his friends participate in are voluntary, and trained using positive reinforcement. Pager’s first choice in a preference test is often banana smoothie, and some days it is strawberries!

(Fig 09)

Animal Care at Neuralink