Brain-body actuated system

• US 5,474,082 A
• Filed: 01/06/1993
• Issued: 12/12/1995
• Est. Priority Date: 01/06/1993
• Status: Expired due to Term

• Current Assignee: Brain Actuated Technologies, Inc.
• Original Assignee: Andrew Junker
• Inventors: Junker, Andrew
• Primary Examiner(s): Kamm, William E.
• Assistant Examiner(s): Jastrzab, Jeffrey R.
• Application Number:
  US08/001,096
• Time in Patent Office: 1,070 Days
• Field of Search: 128/731, 128/732, 128/733, 128/905, 434/236, 434/323, 434/336, 434/350, 273/DIG. 28, 273/438, 273/148 B, 340/825.19, 340/825.24, 345/156, 345/157, 341/20, 341/21, 341/34
• US Class Current: 600/545
• CPC Class Codes: A61B 5/375 using biofeedback
• CPC Class Codes: A61B 5/486 Bio-feedback A61B5/375 takes precedence
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: Y10S 128/905 Feedback to patient of biological signal other than brain electric signal

Abstract

A method and apparatus for assisting a user to control a device in response to a combination of electroencephalographic and electromyographic potentials. The user selects a number of reference frequencies in a range of from 0.5 Hz to 45 Hz defining a like number of control signals. A digital lock-in amplifier is used with a moving average time window filter to produce control signals which are presented to the user. Control system responsiveness is controlled by adjusting the lengths of the moving average time windows. A phase-locked loop is closed around each control signal and is used to track the shifting frequencies of the control signals. The user is able to sense and control changes in the magnitude and frequency of the control signals in the control of the device. By sensing the changes in the magnitude and frequency of the control signals, the user is able to learn a combination of mental and/or physical activities for which changes in the electroencephalographic and electromyographic biopotentials are correlated to control of the device.

Claims

1. A method for controlling at least two functions of a device in response to EEG biopotentials produced by brain activity in a user and EMG biopotentials produced by muscle activity in the user, the method comprising the steps of:

detecting the EEG and EMG biopotentials;

selecting at least two reference frequencies within a predetermined range of selectable frequencies;

producing in response to the EEG and EMG biopotentials, a single input signal representing an aggregate of the EEG and EMG biopotentials, said single input signal changing as a function of changes in either the EEG or the EMG biopotentials;

generating at least two control signals in response to said reference frequencies and said single input signal, each of said control signals being associated with one of said reference frequencies and changing as a function of changes in either the EEG or the EMG biopotentials;

presenting representations of said at least two control signals to the user to enable the user to sense changes in each of said control signals as a function of changes in the brain and muscle activity; and

controlling the at least two functions of the device in response to the changes in each of said control signals, each of the at least two functions being controlled by one of the control signals.

2. The method of claim 1 wherein the step of generating control signals further comprises the steps of:

sampling said input signal at successive sample times to produce sampled input signals over successive sample times; and

periodically computing magnitude and phase values at each of said reference frequencies of said control signals in response to said sampled input signals.

3. The method of claim 2 wherein the step of periodically computing said magnitude and phase values further comprises the steps of:

establishing a base phase value at each of said reference frequencies of said control signals in response to said sampled input signals;

periodically computing time averaged phase values for each of said control signals in response to each of said sampled input signals;

periodically comparing each of said time averaged phase values to said base phase values; and

periodically changing said reference frequencies as a function of comparing said time averaged phase values with said base phase values.

4. The method of claim 3 wherein the device is a music synthesizer and one of the functions is a pitch transposition in the music synthesizer, and the step of controlling the at least two functions of the device further comprises the step of using a control signal representing a change in one of said reference frequencies to control the pitch transposition in the music synthesizer.

5. The method of claim 3 wherein the device is a music synthesizer and one of the functions is left/right panning in the music synthesizer, and the step of controlling the at least two functions of the device further comprises the step of using a control signal representing a change in one of said reference frequencies to control left/right panning in the music synthesizer.

6. The method of claim 3 wherein the device is a music synthesizer and one of the functions is rhythm change in the music synthesizer, and the step of controlling the at least two functions of the device further comprises the step of using a control signal representing a change in one of said reference frequencies to control a rhythm change in the music synthesizer.

7. The method of claim 3 wherein the device is a music synthesizer and one of the functions is volume of the music synthesizer, and the step of controlling the at least two functions of the device further comprises the step of using a control signal representing a change in one of said reference frequencies to control the volume of the music synthesizer.

8. The method of claim 2 wherein the step of periodically computing said magnitude and phase values further comprises the step of averaging said magnitude and phase values for each of said control signals over a user selectable length of time whereby said averaged magnitude and phase values change in response to changes in said input signal.

9. The method of claim 2 wherein said step of periodically computing said magnitude and phase values further comprises the steps of:

periodically computing X and Y quadrature values at each of said reference frequencies of said control signals in response to each of said sampled input signals;

periodically computing time averaged X and Y quadrature values by averaging said X and Y quadrature values in moving average time windows for each control signal; and

periodically computing time averaged magnitude and time averaged phase values for each of the control signals in response to said time averaged X and Y quadrature values.

10. The method of claim 9 wherein said moving average time windows have user selectable window lengths.

11. The method of claim 9 wherein said step of periodically computing said magnitude and phase values further comprises the steps of:

establishing a base phase value at each of said reference frequencies of said control signals in response to one of the sampled input signals;

periodically comparing one of said time averaged phase values of said one of said control signals with said base phase value; and

periodically changing a frequency of one of said reference frequencies associated with said one of said control signals as a function of a difference between said base phase value and said one of said time averaged phase values.

12. The method of claim 11 wherein the step of periodically changing the frequency of said one of said reference frequencies further comprises the steps of:

periodically detecting said one of said time averaged phase values exceeding said base phase value by a predetermined phase change; and

periodically frequency shifting said one of said reference frequencies by a predetermined frequency increment in response to said one of said time averaged phase values exceeding said base phase value by said predetermined phase change.

13. The method of claim 12 wherein the step of presenting said control signals to the user further comprises the step of periodically presenting to the user said time averaged magnitude values and said one of said reference frequencies associated with said one of said control signals.

14. The method of claim 11 wherein the step of controlling the device further comprises the step of controlling the device in response to changes in said frequency of said one of said reference frequencies.

15. The method of claim 9 wherein the step of controlling at least two functions of the device further comprises the steps of:

setting predetermined magnitudes for said control signals; and

using said control signals to control said at least two functions of the device in response to said time averaged magnitudes for said control signals exceeding respective ones of said predetermined magnitudes.

16. The method of claim 1 wherein said predetermined range of selectable reference frequencies is approximately from 0.5 Hz to 500 Hz.

17. The method of claim 1 wherein said control signals range in number from 2 to approximately 50.

18. The method of claim 1 wherein said step of generating control signals further comprises the step of selecting a gain for each of said control signals.

19. The method of claim 1 wherein said step of producing an input signal further comprises:

amplifying and filtering the EEG and EMG biopotentials to produce a filtered input signal having a bandwidth of from approximately 0.5 Hz to 45 Hz; and

producing a digital input signal in response to the filtered input signal.

20. The method of claim 1 wherein said step of detecting the EEG and EMG biopotentials further comprises the step of applying a single set of electrodes to the frontal skull forehead area of the user.

21. An apparatus for controlling at least two functions of a device in response to EEG biopotentials produced by brain activity in a user and EMG biopotentials produced by muscle activity in the user, the apparatus comprising:

a sensor adapted to be applied to the user for producing a single input signal representing an aggregate of the EEG and EMG biopotentials, said single input signal changing as a function of changes in either the EEG or EMG biopotentials;

means for selecting at least two reference frequencies within a predetermined range of selectable frequencies;

signal processing means responsive to said single input signal and said reference frequencies for generating at least two control signals, each of said control signals being associated with one of said reference frequencies and changing as a function of changes in either the EEG or the EMG biopotentials;

means responsive to said control signals for presenting representations of said at least two control signals to the user, thereby enabling the user to sense changes in said control signals as a function of changes in the brain and muscle activity; and

means responsive to said control signals for controlling the at least two functions of the device as a function of the changes in each of said control signals, each of the at least two functions of the device being controlled by one of the control signals.

22. The apparatus of claim 21 wherein said signal processing means further comprises:

means for sampling said input signal at successive sample times to produce a sampled input signal at each sample time;

means responsive to said sampled input signals for periodically producing magnitude and phase values at each of said reference frequencies of said control signals.

23. The apparatus of claim 21 wherein said signal processing means further comprises means for averaging said magnitude and phase values for each of said control signals over a user selectable length of time whereby said averaged magnitude and phase values change in response to changes in said input signal.

24. The apparatus of claim 21 wherein said signal processor further comprises:

digital lock-in amplifier means for periodically computing X and Y quadrature values at each of said reference frequencies of said control signals in response to said sampled input signals;

digital moving average time window filter means for periodically computing averaged X and Y quadrature values in response to the X and Y quadrature values;

digital means for periodically computing magnitude and phase values at each of said reference frequencies of said control signals in response to said averaged X and Y quadrature values.

25. The apparatus of claim 24 wherein said digital moving average time window filter means further comprises means for selecting lengths of moving average time windows.

26. The apparatus of claim 25 wherein said signal processing means further comprises means responsive to said phase values for changing a frequency of one of said reference frequencies as a function of a change of a phase value at said one of said reference frequencies.

27. The apparatus of claim 26 wherein said means for changing said frequency further comprises:

means responsive to said phase values for detecting a change in said one of said phase values in excess of a predetermined phase change; and

means responsive to said means for detecting for frequency shifting said frequency of said one of said reference frequencies by a predetermined frequency increment.

28. The apparatus of claim 27 wherein said means for presenting further comprises means for periodically presenting said magnitude values and said reference frequencies for each of said control signals to the user.

29. The apparatus of claim 21 wherein said predetermined range of frequencies is approximately from 0.5 Hz to 500 Hz.

30. The apparatus of claim 21 wherein said selectable reference frequencies number in a range of from 2 to approximately 50.

31. The apparatus of claim 21 wherein said signal processor further comprises means for selecting a gain for each of said control signals.

32. The apparatus of claim 21 wherein said sensor further comprises:

means for detecting the EEG and EMG biopotentials;

means for amplifying and filtering said EEG and EMG biopotentials to produce a filtered input signal having a bandwidth of from approximately 0.5 Hz to 45 Hz; and

analog to digital converter means responsive to the filter input signal for producing a digital input signal.

33. The apparatus of claim 32 wherein said means for detecting further comprises a single set of electrodes adapted to be applied to the user.

34. The apparatus of claim 33 wherein said single set of electrodes further comprises three signal detection devices.

35. The apparatus of claim 33 wherein said single set of electrodes is adapted to be applied to a frontal skull forehead area of the user.

36. A method of controlling at least two functions of a device in response to biopotentials produced by a user, the method comprising the steps of:

detecting the biopotentials;

producing in response to the biopotentials a single input signal changing as a function of changes in the biopotentials;

generating at least two control signals associated with at least two reference frequencies in response to the single input signal, each of the control signals being associated with one of the reference frequencies and changing as a function of the biopotentials;

presenting representations of the control signals to the user to enable the user to sense changes in each of the control signals as a function of changes in the biopotentials; and

controlling the at least two functions of the device in response to the changes in each of the control signals, each of the at least two functions being controlled by one of the control signals.

37. The method of claim 36 wherein the step of controlling the at least two functions of the device further comprises the steps of:

setting a first predetermined magnitude for a first control signal representing an ON switch state;

setting a second predetermined magnitude for the first control signal representing an OFF switch state;

activating one of the at least two functions of the device in response to said first control signal having a magnitude value equal to said first predetermined magnitude; and

deactivating said one of the at least two functions of the device in response to said second control signal having a magnitude value equal to said second predetermined magnitude.

38. The method of claim 36 wherein the device is a wheelchair and the step of controlling the at least two functions of the device further comprises controlling motion of the wheelchair in at least two directions.

39. The method of claim 36 wherein the device is a cursor on a video display and the step of controlling the at least two functions of the device further comprises controlling motion of the cursor in mutually perpendicular directions.

40. The method of claim 36 wherein the device is a music synthesizer and the step of controlling the at least two functions of the device further comprises controlling at least two functions in the music synthesizer.

41. The method of claim 36 wherein the device is a music synthesizer and one of the functions is selectively turning notes ON and OFF within a melody string produced by the music synthesizer, and the step of controlling the at least two functions of the device further comprises the steps of:

setting a predetermined magnitude value for a first control signal; and

periodically using said first control signal to selectively turn said melody string ON and OFF in response to said first control signal having magnitude values greater and less than said predetermined magnitude value.

42. The method of claim 36 wherein the device is a music synthesizer and one of the functions is selectively turning a harmony string ON and OFF, and the step of controlling the at least two functions of the device further comprises the steps of:

setting a predetermined magnitude value for a first control signal; and

periodically using said first control signal to selectively turn said harmony string ON and OFF in response to said first control signal having magnitude values greater and less than said predetermined magnitude value.

43. The method of claim 36 wherein the device is a music synthesizer and one of the functions is volume, and the step of controlling the at least two functions of the device further comprises the steps of:

setting a predetermined magnitude value for a first control signal; and

periodically using said first control signal to selectively change volume in response to said first control signal having magnitude values greater and less than said predetermined magnitude value.

44. The method of claim 36 wherein the step of controlling the at least two functions of the device further comprises the steps of:

setting a first control signal to be selectively turned ON and OFF in response to a second control signal;

setting a predetermined magnitude value for said second control signal; and

periodically using said second control signal to selectively turn said first control signal ON and OFF in response to said second control signal having magnitude values greater and less than said predetermined magnitude value.

45. An apparatus for controlling at least two functions of an external device in response to biopotentials produced by a user, the apparatus comprising:

a sensor for producing a single input signal in response to detecting the biopotentials produced by the user, the single input signal changing as a function of changes in the biopotentials;

a signal processor operatively connected to the sensor for periodically sampling the single input signal and generating at least two control signals in response to at least two reference frequencies and the single input signal, each of the control signals being associated with one of the reference frequencies and changing as a function of the biopotentials;

an output device connected to the signal processor and providing representations of the control signals to the user to enable the user to sense changes in each of the control signals as a function of changes in the biopotentials; and

a controller operatively connected to the signal processor and adapted to be connected to the external device and controlling the at least two functions of the external device in response to the changes in the control signals, each of the two functions being controlled by one of the control signals.

46. The apparatus of claim 45 wherein the external device is a motor control for a wheel chair and said controller further comprises means for moving said wheelchair in at least two directions.

47. The apparatus of claim 45 wherein the external device is a cursor control for a video display and said controller further comprises means for moving said cursor in mutually perpendicular directions.

48. The apparatus of claim 45 wherein the external device is a music synthesizer and said controller further comprises means for controlling at least two functions in the music synthesizer.

49. The apparatus of claim 45 wherein the controller further comprises:

means for setting predetermined magnitudes for selected ones of said control signals; and

means for periodically providing selected ones of said control signals to control the at least two functions of the external device in response to time averaged magnitudes for said selected ones of said control signals exceeding said predetermined magnitudes.

50. The apparatus of claim 45 wherein the controller further comprises means for periodically providing a control signal to control the external device in response to changing a frequency of said one of said reference frequencies.