Characterization of bioelectric potentials

• US 6,720,984 B1
• Filed: 06/13/2000
• Issued: 04/13/2004
• Est. Priority Date: 06/13/2000
• Status: Expired due to Fees

• Current Assignee: United States of America as represented by the Administrator of NASA, USA As Represented By The Administrator Of The NASA
• Original Assignee: United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration
• Inventors: Jorgensen, Charles C., Wheeler, Kevin R.
• Primary Examiner(s): Luu, Sy D.
• Application Number:
  US09/606,107
• Time in Patent Office: 1,400 Days
• Field of Search: 345/700-702, 345/863, 345/156, 345/161, 345/179, 382/119, 382/181, 382/186-189, 382/209, 600/300, 600/546, 707/541, 715/541
• US Class Current: 715/863
• CPC Class Codes: A61B 5/389 Electromyography [EMG]
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Method and system for recognizing and characterizing bioelectric potential or electromyographic (EMG) signals associated with at least one of a coarse gesture and a fine gesture that is performed by a person, and use of the bioelectric potentials to enter data and/or commands into an electrical and/or mechanical instrument. As a gesture is performed, bioelectric signals that accompany the gesture are subjected to statistical averaging, within selected time intervals. Hidden Markov model analysis is applied to identify hidden, gesture-related states that are present. A metric is used to compare signals produced by a volitional gesture (not yet identified) with corresponding signals associated with each of a set of reference gestures, and the reference gesture that is “closest” to the volitional gesture is identified. Signals representing the volitional gesture are analyzed and compared with a database of reference gestures to determine if the volitional gesture is likely to be one of the reference gestures. Electronic and/or mechanical commands needed to carry out the gesture may be implemented at an interface to control an instrument. Applications include control of an aircraft, entry of data from a keyboard or other data entry device, and entry of data and commands in extreme environments that interfere with accurate entry.

Claims

1. A method for providing an interface for control of an instrument, the method comprising:

receiving electrical signals at at least two selected, spaced apart locations on a person'"'"'s body that are generated in association with a selected gesture executed by the person, over a selected time interval;

electronically processing the at least two electrical signals to identify at least one signal characteristic that is associated with the selected gesture by a process comprising;

forming a sequence of at least two windows, each having a selected number w1 of samples of the received signals, where the at least two windows overlap each other by a selected number w2 of samples;

.

forming an average of magnitudes of values of the received signals within each of the at least two windows;

determining a sequence of at least two state vectors that represent an average of magnitudes of values of the received signals within at least two selected time segments that are determined with reference to the at least two windows; and

performing a hidden Markov modeling process on the at least two state vectors that are associated with the selected gesture.

2. The method of claim 1, further comprising responding to receipt of a signal having said at least one signal characteristic as if said selected gesture had been performed on said instrument, without requiring that said gesture be represented in a visually perceptible manner.

3. The method of claim 1, wherein said selected gesture is drawn from a group of relatively coarse gestures consisting of:

a stop motion;

making a fist;

making a come motion;

making a thumb up motion;

making a thumb down motion;

moving an arm from a first position to a second position; and

moving said person'"'"'s head in a selected direction.

4. The method of claim 1, wherein said selected gesture is drawn from a group of relatively fine gestures consisting of:

tapping at least one finger;

reaching for and depressing at least one key on a keyboard;

moving a joystick in at least one of the directions forward, backward, right and left;

touching a joystick without movement of the joystick; and

grasping and moving a stylus to a selected location.

5. The method of claim 1, wherein said selected gesture is chosen to be grasping a writing instrument and forming at least one character using cursive writing.

6. The method of claim 1, further comprising displaying at least one result of said electronically processed signals in at least one of a graphical format and an alphanumeric format.

7. A system for providing an interface for control of an instrument, the system comprising a computer that is programmed:

to receive electrical signals at at least two selected, spaced apart locations on the person'"'"'s body that are generated in association with a selected gesture by a person, over a selected time interval; and

to electronically process the at least two electrical signals to identify at least one signal characteristic that is associated with the selected gesture by a process comprising;

forming a sequence of at least two windows, each having a selected number w1 of samples of the received signals, where the at least two windows overlap each other by a selected number w2 of samples;

forming an average of magnitudes of values of the received signals within each of the at least two windows;

determining a sequence of at least two state vectors that represent an average of magnitudes of values of the received signals within selected time segments that are determined with reference to the at least two windows; and

performing a hidden Markov modeling process on the at least two state vectors that are associated with the selected gesture.

8. The system of claim 7, wherein said computer is further programmed to respond to receipt of a signal having said at least one signal characteristic as if said selected gesture had been performed on said instrument, without requiring that said gesture be represented in a visually perceptible manner.

9. The system of claim 7, wherein said computer is further programmed to draw said at least one selected gesture from a group of relatively coarse gestures consisting of:

a stop motion;

making a fist;

making a come motion;

making a thumb up motion;

making a thumb down motion;

moving an arm from a first position to a second position; and

moving said person'"'"'s head in a selected direction.

10. The system of claim 7, wherein said computer is further programmed to draw said selected gesture from a group of relatively fine gestures consisting of:

tapping at least one finger;

reaching for and depressing at least one key on a keyboard;

moving a joystick in at least one of the directions forward, backward, right and left;

touching a joystick without movement of the joystick; and

grasping and moving a stylus to a selected location.

11. The system of claim 7, wherein said computer is further programmed to analyze a gesture that comprises grasping a writing instrument and forming at least one character using cursive writing.

12. The system of claim 7, wherein said computer is further programmed to display at least one result of said electronically processed signals in at least one of a graphical format and an alphanumeric format.

13. A method for identifying a gesture received at an interface for control of an instrument, the method comprising:

receiving electrical signals at at least M selected, spaced apart locations (M≧

2) on a person'"'"'s body that are generated in association with a volitional gesture executed by the person, over a selected time interval;

providing a sequence of averages of magnitudes of the received signals, designated OTA′

(i;

m), determined with reference to selected time partition intervals, tpi≦

t≦

tpi′

, for each of a set of states, numbered q′

=1, 2, . . . , Q′

where Q′

is a selected integer ≧

2, and providing an M-dimensional vector

14. The method of claim 13, further comprising:

interpreting said reference gesture corresponding to said index g=g′

as substantially identical to said volitional gesture only if said metric value f(g′

) is at least equal to a selected threshold value f_thr.

15. The method of claim 13, further comprising choosing said metric value f(g) by a process comprising:

providing G sets of collections of M-dimensional reference vectors {fv(i;

q(i);

g)}, numbered g=1, . . . , G, associated with said reference gestures and corresponding to said collection of vectors {fv′

(i;

q(i))}; and

identifying said metric value f(g) with the value ${\left\{\underset{k=1}{\overset{K}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>w_k·<br><br>{<br><br>{\mathrm{fv}}^{\mathrm{\prime <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>\left(\mathrm{Ik};<br><br>q^{\mathrm{\prime <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>\left(\mathrm{Ik}\right)\right)-\mathrm{fv}<br><br>\text{\hspace{1em}<br><br>}<br><br>\left(\mathrm{Ik};<br><br>q<br><br>\text{\hspace{1em}<br><br>}<br><br>\left(\mathrm{Ik}\right);<br><br>g\right)<br><br>}^r\right\}}^{1<br><br>\text{/}<br><br>r},$

where ∥

x∥

designates a non-negative, real valued norm associated with an M-dimensional vector x, {Ik} (k=1, . . . , K;

K≧

2) is a selected set of values of said index i, r is a selected positive number and w_k is a selected non-negative weighting coefficient.

16. The method of claim 13, further comprising choosing said metric value f(g) by a process comprising:

identifying said metric value f(g) with the value ${\left\{\underset{k=1}{\overset{K}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>w_k^{\mathrm{\prime <br><br>}}·<br><br>{<br><br>{\mathrm{sv}}^{\mathrm{\prime <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>\left(q^{\mathrm{\prime <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>\left(\mathrm{Ik}\right)\right)-\mathrm{sv}<br><br>\text{\hspace{1em}<br><br>}<br><br>\left(q<br><br>\text{\hspace{1em}<br><br>}<br><br>\left(\mathrm{Ik}\right);<br><br>g\right)<br><br>}^{r^{\mathrm{\prime <br><br>}}}\right\}}^{1<br><br>\text{/}<br><br>r^{\mathrm{\prime <br><br>}}},$

where ∥

x∥

designates a non-negative, real valued norm associated with an M-dimensional vector x, {k} (k=1, . . . , K;

K≧

2) is a selected set of values of said index i, r′

is a selected positive number and w′

_k is a selected non-negative weighting coefficient.

17. The method of claim 13, further comprising causing said instrument to respond to receipt of said electrical signals corresponding to said volitional gesture as if said reference gesture corresponding to said index g=g′

had been performed on said instrument.

18. A system for providing an interface for control of an instrument, the system comprising a computer that is programmed:

to receive electrical signals at at least M selected, spaced apart locations (M≧

2) on a person'"'"'s body that are generated in association with a volitional gesture executed by the person, over a selected time interval;

to provide a sequence of averages of magnitudes of the received signals, designated OTA′

(i;

m), determined with reference to selected time partition intervals, tpi≦

t≦

tpi′

, for each of a set of states, numbered q′

=1, 2, . . . , Q′

where Q′

is a selected integer ≧

2, and to provide an M-dimensional vector

19. The system of claim 18, wherein said reference gesture corresponding to said index g=g′

is interpreted as substantially identical to said volitional gesture only if said metric value f(g′

) is at least equal to a selected threshold value f_thr.

20. The system of claim 18, wherein said computer is further programmed to choose said metric value f(g) by a process comprising.

providing G sets of collections of M-dimensional reference vectors {fv(i;

q(i);

g)}, numbered g=1, . . . , G, associated with said reference gestures and corresponding to said collection of vectors {fv′

(i;

q(i))}; and

identifying said metric value f(g) with the value ${\left\{\underset{k=1}{\overset{K}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>w_k·<br><br>{<br><br>{\mathrm{fv}}^{\mathrm{\prime <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>\left(\mathrm{Ik};<br><br>q^{\mathrm{\prime <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>\left(\mathrm{Ik}\right)\right)-\mathrm{fv}<br><br>\text{\hspace{1em}<br><br>}<br><br>\left(\mathrm{Ik};<br><br>q<br><br>\text{\hspace{1em}<br><br>}<br><br>\left(\mathrm{Ik}\right);<br><br>g\right)<br><br>}^r\right\}}^{1<br><br>\text{/}<br><br>r},$

where ∥

x∥

designates a non-negative, real valued norm associated with an M-dimensional vector x, {k} (k=1, . . . , K;

K≧

2) is a selected set of values of said index i, r is a selected positive number and w_k is a selected non-negative weighting coefficient.

21. The system of claim 18, wherein said computer is further programmed to choose said metric value f(g) by a process comprising:

identifying said metric value f(g) with the value ${\left\{\underset{k=1}{\overset{K}{\sum <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>w_k^{\mathrm{\prime <br><br>}}·<br><br>{<br><br>{\mathrm{sv}}^{\mathrm{\prime <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>\left(q^{\mathrm{\prime <br><br>}}<br><br>\text{\hspace{1em}<br><br>}<br><br>\left(\mathrm{Ik}\right)\right)-\mathrm{sv}<br><br>\text{\hspace{1em}<br><br>}<br><br>\left(q<br><br>\text{\hspace{1em}<br><br>}<br><br>\left(\mathrm{Ik}\right);<br><br>g\right)<br><br>}^{r^{\mathrm{\prime <br><br>}}}\right\}}^{1<br><br>\text{/}<br><br>r^{\mathrm{\prime <br><br>}}},$

where ∥

x∥

designates a non-negative, real valued norm associated with an M-dimensional vector x, {Ik} (k=1, . . . , K;

K≧

2) is a selected set of values of said index i, r′

is a selected positive number and w′

_k is a selected non-negative weighting coefficient.

22. The system of claim 18, wherein said computer causes said instrument to respond to receipt of said electrical signals corresponding to said volitional gesture as if said reference gesture corresponding to said index g=g′

had been performed on said instrument.