Multi-modal neural interfacing for prosthetic devices

• US 9,486,332 B2
• Filed: 04/15/2011
• Issued: 11/08/2016
• Est. Priority Date: 04/15/2011
• Status: Active Grant

• Current Assignee: Johns Hopkins University
• Original Assignee: Johns Hopkins University
• Inventors: Harshbarger, Stuart D., Beaty, James D., Vogelstein, R. Jacob, Thakor, Nitish V.
• Primary Examiner(s): Snow, Bruce E
• Application Number:
  US14/110,508
• Publication Number:
  US 20140031952A1
• Time in Patent Office: 2,034 Days
• US Class Current: 1/1
• CPC Class Codes: A61B 5/369 Electroencephalography [EEG] devices for psychotechnics A61B5/16
• CPC Class Codes: A61B 5/398 Electrooculography [EOG], e.g. detecting nystagmus; Electroretinography [ERG]
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: A61F 2/54 Artificial arms or hands or parts thereof
• CPC Class Codes: A61F 2/68 Operating or control means
• CPC Class Codes: A61F 2/72 Bioelectric control, e.g. myoelectric
• CPC Class Codes: A61F 2002/5058 having means for restoring the perception of senses
• CPC Class Codes: A61F 2002/5059 the feeling of position
• CPC Class Codes: A61F 2002/5061 the sense of touch
• CPC Class Codes: A61F 2002/5063 the feeling of temperature, i.e. of warmth or cold
• CPC Class Codes: B25J 9/16 Programme controls programme controls in general G05B19/00, e.g. numerical pogramme controls G05B19/18; recording or playback systems G05B19/42

Abstract

Methods and systems to interface between physiological devices and a prosthetic device, including to receive a plurality of types of physiological activity signals from a user, decode a user movement intent from each of the plurality of signals types, and fuse the movement intents into a joint decision to control moveable elements of the prosthetic device.

Claims

1. A neural interface system comprising:

a prosthetic device;

a plurality of types of sensors configured to receive a plurality of types of physiological activity signals from a prosthetic device user;

a neural interface configured to decode a user movement intent from each of the plurality of signals types, and fuse the movement intents into a joint decision to control moveable elements of the prosthetic device, the neural interface comprising;

a plurality of classifier modules, each classifier module associated with a corresponding one of the signal types and configured to determine a user movement state from signals of the signal type;

a plurality of decoders, each associated with a corresponding one of the signal types and configured to decode a movement intent from signals of the signal type and from one or more of the user movement states; and

a fusion module to fuse movement intents from a plurality of the decoders into the joint movement decision;

wherein the plurality of signal types include a combination of two or more of,

a local field potential (LFP) signal,

a unit activity (spike) signal,

an epidural electrocorticography grid (ECoG) signal,

an electromyography (EMG) signal,

an electroencephalography (EEG) signal, and

an electronystagmography (ENG) signal.

2. The system of claim 1, further including:

a plurality of groups of the classifier modules, each group associated with a corresponding group of control (GOC) of the prosthetic device;

a plurality of groups of the decoders, each group associated with a corresponding one of the GOC;

a plurality of fusion modules, each associated with a corresponding one of the GOCs to fuse the movement intents from decoders of the GOC into a joint decision of the GOC; and

a motion estimator to generate a movement action from joint movement decisions of a plurality of the GOCs.

3. The system of claim 2, wherein:

the prosthetic device includes a prosthetic arm and hand; and

the groups of control include an upper arm group, a wrist group, a hand and finger group, and an endpoint group.

4. The system of claim 2, wherein the plurality of decoders comprise a decoder to process cortical signals, a decoder to process peripheral nerve signals, a decoder to EMG signals, and a decoder to process prosthetic control signals, the system further including:

a cortical multimodal control unit (cMCU), including the decoder to process the cortical signals;

a peripheral nerve multimodal control unit (pMCU), including the decoder to process the peripheral nerve signals; and

a neural fusion unit (NFU), including the fusion module and the decoder to process the EMG and CPC signals.

5. The system of claim 1, wherein at least one of the classifier modules is configured to determine the movement state as one of motionless, pre-movement, and peri-movement.

6. The system of claim 1, wherein the fusion module is configured to perform one or more of decision fusion and data fusion.

7. The system of claim 1, wherein the fusion module is integrated within the decoder.

8. The system of claim 1, further including:

a plurality of pre-processors, each associated with a corresponding one of the signal types to receive and pre-process signals of the signal type and to selectively direct subsets of the signals to one or more of the classifier modules.

9. The system of claim 8, wherein at least one of the pre-processors includes a multi-stage data validation module, each stage to identify signals as one of valid and invalid as a function of a corresponding metric, and wherein the validation module is configured to identify a validation stage at which a signal is identified as invalid.

10. The system of claim 1, wherein:

the neural interface system is configured to receive and incorporate sensory feedback from the prosthetic device into the joint movement decision; and

the sensory feedback includes one or more of velocity, speed, force, direction, position, and temperature information.

11. The system of claim 1, wherein the plurality of sensors comprises one or more relatively non-invasive physiological sensors or one or more relatively invasive physiological sensors, and

wherein the neural interface system includes a plurality of modular and configurable components, including a base configuration to process signals from the one or more relatively non-invasive physiological sensors, and one or more selectively enabled modules to process signals from the one or more relatively invasive physiological sensors.

12. The system of claim 1, wherein:

at least one of the decoders is configured to compute an unnormalized log posterior probability (ULPP) value for each of a plurality of classes of movement in accordance with Bayesian classifiers, determine the movement intent from the ULPP values, and output the movement intent and the ULPP values; and

the fusion module is configured to generate the joint movement decision based at least in part of the ULPP values.