A bioelectric neural interface towards intuitive prosthetic control for amputees

An overview of the experimental setup is presented in figure 2((A),(B)). A transradial amputee subject is implanted with four microelectrode arrays targeted to four different fascicles within the median and ulnar nerves. The mirrored bilateral paradigm is utilized where ENG signals are recorded from the injured hand, while the intended movements are captured by the data glove from the able hand (ground truth). The acquired ENG dataset with ground-truth labels is used to train AI models that can report the subjects' motor intention.

Zoom In Zoom Out Reset image size

The human experiment protocols are reviewed and approved by the Institutional Review Board (IRB) at the University of Minnesota (UMN) and the University of Texas Southwestern Medical Center (UTSW). The patient voluntarily participates in our study and is informed of the methods, aims, benefits and potential risks of the experiments prior to signing the Informed Consent. Patient safety and data privacy are overseen by the Data and Safety Monitoring Committee (DSMC) at UTSW. The implantation, initial testing, and post-operative care are performed at UTSW by Dr Cheng and Dr Keefer while motor decoding experiments are performed at UMN by Dr Yang's lab. The clinical team travels with the patient in each experiment session.

Scorpius: figure 2(C) shows the overview of the Scorpius system—a miniaturized front-end recorder equipped with the fully-integrated Neuronix neural recording chip. It consists of two sub-units: the head-piece and the auxiliary-piece, connected by a flexible section. The head-piece contains the Neuronix chip—the recorder's key functional component, along with an electrode connector, and other passive components. The auxiliary-piece consists of a field-programmable gate array (FPGA) (AGLN250, Microsemi, CA), a high-speed universal serial bus (USB) encoder (FT601Q, FTDI, UK), and power management circuitry with various voltage regulators (ADR440, ADP222, ADA4898-2, Analog Devices, MA). The auxiliary-piece sole function is to pass-through the digitized neural data while powering the Neuronix chip through a single micro-USB connector. Recorded data are retrieved and processed offline on the back-end on an external computer.

Neuronix: members of the Neuronix chip family are a fully-integrated neural recorder application-specific integrated circuits (ASIC) based on the frequency-shaping (FS) architecture [13, 35, 42–47] and high-resolution analog-to-digital converters (ADC) [48–51] that we have pioneered. At the sampling rate of 40 kHz, the recorder is optimized to acquire neural data with a maximum gain of 60 dB in the bandwidth 300–3000 Hz and approximately 40 dB in the bandwidth 10–300 Hz while suppressing motion and stimulation artifacts. This frequency-dependent transfer function is achieved with the switched-capacitor circuit implementation, which offers a minimal noise floor while simultaneously preventing input circuit saturation due to large-amplitude artifacts. The FS architecture's unique characteristics allow the Neuronix chip to capture high-quality neural signals by 'compressing' the effective dynamic range (DR) of the recorded data, giving an additional 4.5-bit resolution across the input full-range as shown in Xu et al[ 44]. This is coupled with a 12-bit successive approximation register (SAR) ADC with automatic calibrations including comparator's random offset cancellation and capacitor mismatch error suppression. The resulted system has a '11+4.5' bits effective DR, where 11-bit is the effective number of bits (ENOB) of the ADC and 4.5-bit is the DR compression ratio. These capabilities have been extensively validated and demonstrated in various Neuronix chip generations, in both in-vitro and in-vivo experiments [35, 42, 43]. Here we solely focus on the system integration where Neuronix is deployed for human clinical applications. The latest iteration of the Neuronix chip shown in figure 2(D) has ten recording channels with a further optimized layout that occupies a 3 mm × 3 mm silicon die area when fabricated in the GlobalFoundries 0.13µm standard CMOS process. Eight channels are used for neural data acquisition and two channels are reserved for testing purposes.

Table 1 summarizes the specifications of the Scorpius system in comparison to commercial counterparts [52–54]. Here we only include systems that are being used in human clinical trials for neuroprosthesis studies involving peripheral nerves or cortical recordings. The Scorpius has a substantially smaller footprint (size and weight) compared to others because it is the only miniaturized system specifically designed towards implantable and wearable applications while showing improved sensitivity and specificity in sensing nerve signals. Furthermore, the FS architecture allows the proposed system to achieve an effective system resolution of '11+4.5' bits which is significantly higher than the commercial counterparts. In practice, the overall system resolution is the effective DR which is determined by the input range, noise floor, and ADC resolution. A combination of a higher system resolution as well as system miniaturization and low power implementation allows Scorpius to isolate weak neural signals from large amplitude artifacts and interferences, which in turn enable the high DOF motor decoding through deep neural networks.

Table 1. Specifications of Scorpius in comparison to commercial systems being used in human clinical applications.

	Cerebus/NeuroPort (Blackrock Microsys.)	Grapevine (Ripple Neuro)	RHD2000 (Intan Tech.)	Scorpius (This work)
Die size (mm)	–	–	4.1 $\times$ 4.8	$3 \times 3$
System size a ( mm)	$88 \times 325 \times 425$	$43 \times 110 \times 190$	$20 \times 99 \times 155$	$4 \times 12 \times 68$
System weight (g)	6800	907	130	3
No. of channels b	8–512	32–512	16–256	10
Sampling rate (kHz)	30	30	30	10–80
Bandwidth (Hz)	0.3–7 500	0.1–7 500	0.1–10 000	10–5 000
Noise floor ($\mu\textrm{V}_{\textrm{rms}}$)	3.0	2.1	2.4	2.5
ADC resolution (bits)	16	16	16	12
System resolution c ( bits)	12.4	12.5	12.0	11+4.5
Neuroprosthesis human clinical trial	Davis et al( 2016) Irwin et al( 2017) Vu et al( 2018)	Wendelken et al( 2017) George et al( 2018)	Bullard et al( 2019) Kannape et al( 2016)	This work

^a System size & weight includes the headstages and power/data interface board. ^b No. of channels depend on the number of headstages connected (1–8). ^c System resolution is the effective dynamic range (DR) that is determined by the input range, noise floor, and ADC resolution.

Figure 3 shows the construction of the microelectrode array that is an extended version of the design reported in [33, 34]. Each array consists of four cuff contacts and an intrafascicular shank with ten longitudinal intrafascicular electrodes (LIFE). The intrafascicular contacts are 50 µm in diameter and spaced 0.5 mm apart. The design aims at recording neural data from both single-axon and population of neurons as well as providing sensory feedback through electrical stimulation.

Zoom In Zoom Out Reset image size

The implantation surgery is performed by Dr Cheng at the Clements University Hospital in UTSW, Dallas. Microsurgical fascicular targeting (FAST) technique is used to guide the microelectrode array into the nerve fascicle as shown in figures 3(B) and (C). The patient is implanted with four FAST-LIFE microelectrode arrays, two in the median nerve, and the other two in the ulnar nerve. Within one nerve, the arrays are placed into two discrete fascicle bundles. Their functional specificities are examined during the surgery such that one is more likely to carry efferent motor control signals while the other carries the afferent sensory/proprioceptive signals. The 'motor' fascicles are identified by using a handheld nerve stimulator (Vari-Stim III, Medtronic, MN) to induce contraction of intrinsic muscles still present in the residual limb. The 'sensory' fascicles are found by placing needle electrodes in the scalp over the primary sensory cortex and observing somatosensory evoked potentials (SSEP) elicited by the nerve stimulation. Nevertheless, it is worth noting that there is no clear separation between motor and sensory functions as shown in [34]. An electrode with strong motor control signals can frequently be found on the 'sensory' fascicles and vice versa.

Figure 3(D) shows a photo of the implantation site during the operation. Figure 3(E) shows an x-ray image of the patient arm after the operation with the implanted microelectrode arrays, percutaneous wires, and the external connector. In total, 56 wires are brought out through 14 percutaneous holes where each hole supports a 4-wire bundle. One of the cuffs is left out and replaced with a platinum wire that acts as the common ground and return-electrode for electrical stimulation. The external wires are attached to two connector blocks which can be connected to external recorders via two standard 40-pin Omnetics nano connectors.

Motor decoding data are obtained via the mirrored bilateral training with a data glove, which has been successfully implemented in similar studies [17, 55]. The patient is asked to perform various gestures with the able hand while imagining doing the same movement simultaneously with the injured (phantom) hand. Nerve data from the injured hand are acquired from implanted microelectrodes while the intended motions from the able hand are concurrently captured by a data glove (VMG 30, Virtual Motion Labs, TX). The glove can acquire 15 DOF corresponding to the movement of key joints as shown in figure 4(A). They include the flexion/extension of all five fingers (D1–D10), abduction/adduction (D11-D14), and thumb-palm crossing (D15). The glove's data are used as the ground-truth label for both training and benchmarking the deep learning model.

Zoom In Zoom Out Reset image size

While it may be appealing to be able to demonstrate independent and exclusive control of 15 individual DOF, there are several practical challenges to this approach.

• (i)  
  It technically requires training and testing all 2¹⁵ combinations of all the DOF, which is not possible.
• (ii)  
  In daily living activities, most people only use a small fraction of these possible gestures which almost always involve the movement two or more DOF.
• (iii)  
  It may not even be possible to control 15 DOF independently. For example, many people struggle to flex the little finger without also bending the ring finger.

In fact, the coherent motion of multiple DOF contributes to the intuitiveness and dexterity of the human hand that cannot be easily replicated by a mechanical prosthesis. Here we must accept the trade-off between practicality and generativity by starting with the most common hand gestures such as flexing fingers, grasping, pinching, etc In future studies, additional gestures could be added like thumb up, pointing, etc. Thanks to the robustness of deep learning, this can easily be done by adding more data into the existing dataset and fine-tuning the models without any other modifications to their architecture.

Figure 4(B) illustrates different hand gestures utilized in this experiment which are selected to elicit at least two or more DOF at once to various degrees. They comprise of individual finger flexion (M1–M5), two or three fingers pinching (M6–M8), and fist/gripping (M9). Resting (M0) is used as the reference position. In each experiment session, the patient repeatedly alters between resting (M0) and one of the other hand gestures (M1–M9) at least 100 times where neural data and intended motions are continuously recorded. Incorrect trials are manually discarded upon visual inspection. They mostly consist of trials where the burst of neural activities do not synchronize with the movement ground-truth due to the inconsistent timing of the data glove. The final dataset which is used to train the AI models has a total of 760 trials. Figure 4(C) shows typical examples of the raw data acquired by the data glove during the experiment. In certain gestures such as index flexion (M2), only 4–5 DOFs out of 15 DOF are strongly excited, while in others such as fist/grip (M9), almost all DOF are elicited to some degree.

Figure 5(A) presents an overview of the back-end data analysis paradigm including pre-processing procedures and motor decoding using deep learning. In this work, the data are processed offline with the primary goals of (i) demonstrating ENG recordings obtained from the proposed Scorpius system contain sufficient information for multi-DOF motor decoding; and (ii) determining the best machine learning approach to extract and interpret this information.

Zoom In Zoom Out Reset image size

Pre-processing: The recorded data including neural signals from the Scorpius system and the labeled ground-truth movements from the data glove are segmented into individual, non-overlapping trials of approximately four seconds. The raw neural data are first pre-processed by applying anti-aliasing filters and downsampling four times. Next, neural features are extracted using the short-term Fourier transform (spectrogram) with 20 ms steps. For each channel, we compute the average power spectral density of 32 frequency bins spaced on a log scale from 150 to 3000 Hz. We choose a conservative cut-off frequency of 150 Hz to capture most of the ENG signals' power while minimizing the EMG signals and motion artifacts that may occur during the experiment and tend to locate in the lower frequency band. This results in an input of dimension [16 channels] × [32 bins] × [200 time-steps], which is reshaped into an [512 × 200] 'image' for the deep learning model. Here spectrogram serves as feature extraction and dimensionality reduction which significantly lower the complexity of the AI decoder. While it is possible to design deep neural networks that directly use raw neural data, the number of parameters/layers and computational resources needed would be considerably higher making it further difficult to translate into real-world applications.

Deep learning model: figure 5(B) presents the design of the core AI decoder which is based on the RNN architecture with the use of long-short term memory (LSTM) cells. The RNN architecture is among the most advantageous approaches for handling sequential modeling problems [56–59]. Here we developed a self-attention mechanism with the RNN to encode the recorded high-dimensional signals into a 64-dimensional neural code that can be used to generate the motion trajectories. The network architecture comprises of 21 convolutional layers, two recurrent layers, two attention layers, and an output layer. Following common practices in deep learning, we optimize these numbers by gradually adding network layers. We track the increase of the decoder performance using 5-fold cross-validation. The performance converges as we stack more residual blocks, LSTM layers, and attention blocks. Additional layers beyond this point tend to cause over-fitting.

We train 15 AI models separately where each model decodes the trajectory of an individual DOF. All 15 models have the same architecture (i.e. number of layers, parameters, etc.) and take the input from all 16 recording channels. There are two technical reasons for this. First, the training process (i.e. learning rate, decay rate, number of epochs, etc.) can be optimized for individual DOF to achieve the best prediction accuracy while preventing over-fitting. For example, here the D5 model (thumb) tends to converge in fewer epochs than the D1 model (little finger) because the signals are more distinguishable on the median nerve. Second, the system would be more robust because if one model has poor accuracy or fails during normal operation, it would not affect the performance of others.

Convolutional layers: Convolutional layers are used to encode local patterns of the input. All convolutional layers have 32 filters with a kernel width of three. After each convolutional layer, we apply batch normalization and a rectified linear unit (ReLU) activation. The batch normalization is a technique that normalizes the inputs of a layer to result in a mean output activation of zero and standard deviation of one [60]. It can improve the performance and stability of neural networks. The ReLU activation function which is defined as f(x) = max(0, x) is the most popular activation function for deep neural networks [61]. It overcomes the vanishing gradient problem, allowing networks to learn faster and perform better. Inspired by the design of residual networks [62], we stack 10 residual blocks with two convolutional layers each, following a convolutional layer that reduces the input dimensionality. Every other residual block subsamples its inputs by a factor of two. We also apply dropout [63] between residual blocks with a probability of 0.1.

Recurrent layers with attention: To accumulate local information across time, we feed the convolutional layer outputs into a two-layer recurrent network, one temporal step at a time. LSTM cells are used as the basic RNN unit. Each LSTM cell has 64 hidden units. After each recurrent layer, we apply a dropout with a probability of 0.2. The recurrent layer outputs are linearly combined across time, with dynamic combination weights determined with a self-attention mechanism. The self-attention block is composed of two dense layers with a ReLU in between, which computes one attention weight for each temporal step. The softmax-normalized weights are multiplied with the hidden states of the LSTM, generating a 64-dimensional code that represents the entire input signal.

Output layer: Since our experiments only consist of simple finger motion, a linear projection output is sufficient to decode the hidden code into an output motion trajectory. For more complex motion patterns, this linear decoder can be replaced with a deeper network structure.

Training process: The dataset is randomly split across all the trials with 50% of the data used for training and 50% used for validation. The validation data are not seen by the model during the training process and do not overlap with the training data. We train the model using a loss function that combines both mean squared error (MSE) and variance accounted for (VAF), which is denoted as:

$$\begin{equation} \textrm{Loss}(y, \hat{y}) = \textrm{MSE}(y, \hat{y}) + \alpha \textrm{log}(2 - \textrm{VAF}(y, \hat{y})) \end{equation} \tag{ 1 }$$

where α = 0.05, y is the ground-truth trajectory, $\hat{y}$ is the estimated trajectory. The values of y and $\hat{y}$ are normalized in a [0, 1] range, where 0 represents the resting position and 1 represents the fully flexing position.

The model parameters are randomly initialized as described by [64]. We use Adam optimizer [65] with the default parameters $\beta_1 = 0.99, \beta_2 = 0.999$, and a weight decay regularization $L_2 = 10^{-5}$. The minibatch size is set to 38 with each training epoch consists of 10 mini-batches. The learning rate is initialized to 0.005 and reduced by a factor of 10 when the training loss stopped improving for two consecutive epochs. These hyperparameters are chosen via a combination of grid search and manual tuning.

Baseline models: The proposed deep learning-based AI model is compared with support vector machine (SVM), random forest (RF), multilayer perceptron (MLP), and convolutional neural network (CNN). The SVM parameter is set to C = 1. The RF model has 40 trees with a maximum depth of three. The MLP has three hidden layers of 128 units each. The CNN model is created by replacing the recurrent and attention layers of the proposed RNN model with two dense layers of 128 units.

Hardware specifications: Training the AI models is a computationally intensive task, thus is done on a desktop PC (Intel Core i7-8086K CPU, NVIDIA Titan XP GPU). On the other hand, data acquisition and motor decoding (i.e. DL inference) require much lower computational resources. We collect training data and perform decoding using both the desktop PC and a laptop (Intel Core i7-6700HQ CPU, NVIDIA GTX 970M GPU) with no significant differences in performance.

Performance metrics: We measure the quantitative regression prediction of the decoding algorithm using two metrics MSE and VAF that have the following formulae:

$$\begin{equation} \textrm{MSE}(y, \hat{y}) = \frac{1}{N} \Sigma _{i = 1}^{N} (\hat{y}_i-y_i)^2 \end{equation} \tag{ 2 }$$

$$\begin{equation} \textrm{VAF}(y, \hat{y}) = 1 - \frac{\Sigma _{i = 1}^{N} (\hat{y}_i-y_i)^2 }{\Sigma _{i = 1}^{N} (y_i - \bar{y}_i)^2 } \end{equation} \tag{ 3 }$$

where N is the number of samples, where y is the ground-truth trajectory, $\bar{y}$ is the average of y, and $\hat{y}$ is the estimated trajectory.

We use two Scorpius devices to acquire ENG signals from 16 differential electrodes across four implanted arrays. The top four electrodes with the best signal-to-noise ratio (SNR) are selected from each array. We want to include signals from a wide selection of nerve fascicles even though the 'sensory' channels may have fewer activities than 'motor' channels. A customized adapter is used to manually route recording channels to desirable electrodes. Figure 6 presents a typical data sequence recorded during an experiment session. Here the patient is asked to perform a mirrored bilateral task of opening and closing all five fingers (M9: fist/grip) on both able and injured (phantom) hands. ENG signals are acquired by the Scorpius system from the injured hand while the intended movements are simultaneously captured by the data glove from the able hand. Figures 6(A) and (B) show the filtered data in the low (30–600 Hz) and high (300–3000 Hz) frequency band corresponding to the trajectory of a finger obtained by the data glove. 8-order Butterworth filters are used with the forward-backward zero-phase filtering technique to avoid any distortion. Figure 6(D) and (E) show the zoom-in of the data window marked in 6(A) and (B).

Zoom In Zoom Out Reset image size

The acquired signals show rich dynamics with characteristics that suggest the presence of voluntary compound action potentials (vCAP) ⁷ and single-axon potentials (spikes). vCAP are likely associated with the low-frequency band (figures 6(A) and (D)) while spikes tend to be more prominent in the high-frequency band (figures 6(B) and (E)). The proportion of each type of ENG signals on a particular electrode depends on its placement during the implantation surgery and cannot be easily controlled. To maximize the chance of extracting information encoding motor intention, we specifically choose a mixture of electrodes that express both types of data.

Figure 6(C) presents the spectrogram of two channels (channel 2 and 3) that have particularly large low-frequency activities. The signal's dynamics show a clear correlation with the intended movements and distinguishable from the noise floor. Next, we divide the data into segments of resting (M0), movement (M9); and perform the power spectral density estimation using Welch's method. As shown in figure 6(F), there are vibrant differences in ENG dynamics between two gestures M0 and M9.

To evaluate the characteristics of the high-frequency band data, we perform spike sorting analysis using the amplitude-based detection and principal component analysis (PCA)-based sorting techniques as in [35]. All intrafascicular channels display clear neural spikes activities. Figure 6(G) presents examples of the spike cluster isolated from one recording channel over the entire experiment session (60 min). Figure 6(H) shows the distribution of inter-spike intervals (ISI). Figure 6(I) shows the firing rate of two spike clusters modulated by the finger movement. One cluster is in phase and the other is out of phase with a joint movement. The results show standard waveform and statistics that are consistent with single-axon nerve activities.

Additional spike clusters isolated from every channel in a single experiment session are provided in the supplementary material (https://stacks.iop.org/JNE/17/066001/mmedia). Each cluster has at least 50 spikes with SNR ranging from 4.7 to 11.9 so that they can be adequately separated from the noise floor using spike sorting techniques. The spike clusters have the peak-to-peak amplitude substantially smaller than vCAP signals yet can still be distinctly separated from the noise floor. The noise floor is calculated by taking the rms value of data sequences during resting where no neural activities are present. This results in a 5-6 µVrms noise floor in 300–3000 Hz which is typical for the electrical noise of the FS amplifier plus the electrode interface.

Figure 7 compares the spectrogram of four channels during three different training sessions including thumb flexion (M1), ring flexion (M4), and fist/grip (M9). Two channels (Ch-2, Ch-3) are from the median nerve while the others (Ch-14, Ch-15) are from the ulnar nerve. The results indicate that each hand gesture has a distinct spectro-temporal signature that is highly correlated with the finger movements. In this study, these signatures are used as the basis to decode the patient's motor intention. Here we specifically select the channels and gestures with visually observable signatures. In other training sessions and channels, the discrepancy is dimensionally complex and not always obvious to the naked eyes. Therefore, machine learning is the most advantageous approach to implement the motor decoder.

Zoom In Zoom Out Reset image size

Furthermore, nerve activities are consistent with the human anatomy. The median nerve mostly innervates the thumb, index, middle, and half the ring fingers; while the ulnar nerve is with half the ring and the little finger. As shown in figures 7(A) and (C), when flexing the thumb (M1, M9), ENG signals are more prominent in the median nerve and less notable in the ulnar nerve. Opposite observations can be seen with the ring finger (M4, M9) in figures 7(B) and (C). This further reinforces the capability of the Scorpius system in acquiring ENG signals with the necessary information to decode the intended movement of individual fingers.

The performance of the proposed RNN model is compared to other baseline algorithms. They include three 'classic' machine learning algorithms: SVM, RF, MLP; and a deep learning model with a simpler architecture: CNN. Figure 8(A) shows examples of predicted trajectories of 15 DOF produced by each algorithm against the ground-truth movements. In most cases, all algorithms can predict the course motion of each DOF, i.e. whether a finger moves in a specific gesture. However, the two deep learning models, CNN and RNN, always yield a more accurate estimation of the fine trajectory, which is crucial for achieving natural movements.

Zoom In Zoom Out Reset image size

In figure 8(B), for better visualization, the predicted trajectories are mapped onto a 3D model of the prosthetic hand via the MuJoCo platform [70]. We present the mapping of three gestures: thumb flexion (M1), index pinch (M6) and fist/grip (M9) which represent three different classes of motion where one, two and all fingers move respectively. In more complex gestures like M6 and M9, the fine timing and relative relation between multiple DOF become essential to form functional and natural movements. Classic algorithms like SVM, RF, and MLP fail to accomplish this goal even though their raw output trajectories (figure 8(A)) may not look much different from the ground-truth.

Figure 9 reports the comparison of all algorithms using standard quantitative metrics including MSE and VAF. For each DOF, the result shows average performance across all movements. To verify the significance of differences between RNN and the other decoders, we conduct paired t-tests with a Bonferroni correction. The results indicate that the performance differences in MSE and VAF are all statistically significant with $p\lt0.001$. The proposed RNN model consistently outperforms other algorithms including CNN across all 15 DOF, yielding measurable lower MSE and higher VAF scores. It is worth noting that the MSE plot is shown on a log scale. In some cases like D7, D9, and D10, RNN offers 3-4 times lower error than SVM and RF. However, MSE cannot be used to compare the relative performance among different DOF. Some joints like D1-D5 move significantly less than the others resulting in lower MSE baseline. This is due to the limitation of the hand gestures used for training and the implementation of the flex sensors in the data glove. VAF scores offer a more realistic benchmark. In most DOF with substantial movements (D5-D15), the RNN achieves VAF scores of 0.7-0.9 (out of a maximum score of 1). These are equivalent to near-natural movements of individual fingers.

Zoom In Zoom Out Reset image size

The proposed interface opens up a conduit allowing direct access to the nervous system through peripheral nerves. Recorded neural data show characteristics associating with ENG signals originated from bundles of neuron axons in a nerve fascicle. Acquired neural data have rich dynamics in low- and high-frequency bands that suggest the presence of the compound (vCAP) and single-axon (spikes) nerve activities. Those activities are also shown to have distinct spectro-temporal signatures associated with each hand gesture and are consistent with the anatomy/function of the median and ulnar nerve.

The data accessible from this conduit are shown to contain sufficient information contents to enable dexterous control of a prosthetic hand. Motor decoding results show the proposed AI model (RNN) is able to translate the neural patterns into individual finger movements with up to 15 DOF simultaneously. The RNN model outperforms not only classic machine learning algorithms (SVM, RF, MLP) but also the CNN model across all DOF in both performance metrics (MSE, VAF). This implies the motor control signals embedded in the neural data are comprehensive, yet highly complex and intricate. When visualizing through a virtual prosthetic hand, the more fine-detailed prediction of finger trajectories produced by the proposed AI model is shown to be essential to enable lifelike dexterity.

There is an open question that calls for further investigation in future works into the EMG/ENG nature of the acquired neural data. We could not rule out a possibility that volume-conducted EMG from residual muscles in the stump may also be coupled into the neural recordings, masking the ENG signals. While both EMG and ENG recordings can be used to control prosthetic functions, an ENG-based system would be more robust as it can be applied to a wider population of patients with various levels of amputation.

In the current experiment setup, although it is difficult to separate the EMG and ENG components due to their overlapping frequency bandwidth (10–500 Hz), there is evidence to suggest that the acquired data contain an adequate portion of ENG signals:

• (i)  
  The recording channels are physically very closed to each other and strategically located far away from residual muscles. The intrafascicular contacts are 50 µm in diameter and spaced 0.5 mm apart. The four electrode arrays are implanted about 1 cm apart along the nerve. The implantation location is near the far end of the amputated limb. As a result, volume-conducted EMG signals from distant muscles should tend to appear on all recording channels with similar waveform and amplitude, which is not evident in figure 6 and figure 7.
• (ii)  
  Signal characteristics associating with spontaneous single-axon action potentials (i.e. spikes) can be found across all recording channels as shown in figure 6 and the supplementary. These neural spikes can form clusters with typical waveform, inter-spike interval, and firing rate. Some spike clusters have excitatory behavior while others have inhibitory behavior. These suggest that the electrode contacts successfully interface with individual nerve fibers.
• (iii)  
  There is a clear separation between control signals in the median and ulnar nerve. The occurrence of the signals is also consistent with the human hand's anatomy. For example, in figure 7, the signals controlling the flexing of the thumb appear strongly in the median channels, but none can be found in the ulnar channels.
• (iv)  
  The AI decoder can predict the movement of five fingers with high accuracy and dexterity including flexion/extension and abduction/abduction. This suggests that acquired neural data contain control signals associating with the muscles in the hand/palm that are no longer present.
• (v)  
  The patient is asked to purposely contract the residual muscles in the amputated limb but could not reproduce the recorded waveform. We also note that the signals acquired when flexing each finger remain the same regardless of the arm's position.

The future works will involve developing the existing framework into an online paradigm that is capable of real-time decoding neural data and controlling a physical prosthetic hand. This will not only be a definite demonstration of the proposed platform in practice but also one step closer to a comprehensive solution for clinical uses. The incorporation of real-time control will give the essential visual feedback to the amputees, allowing their brain to adapt the neural pattern to the deep neural network's behavior, thus further enhancing the decoding accuracy as well as the perception of the hand. The key challenge will be optimizing the components of the platform to work seamlessly together as well as minimize the processing latency.

Furthermore, we are also working on incorporating electrical stimulation capabilities into the proposed platform. The feasibility of using the same electrode array for stimulation has been explored in [34]. This will allow the system to provide neural feedback to the patient including tactile and proprioceptive responses, which are essential for a realistic neuroprosthesis [71, 72]. These features could be facilitated by the Neuronix chip's support for simultaneous recording and stimulation [35, 47], as well as advances in high-precision, charge-balanced neural stimulators [13, 73].