Electroencephalogram interface system, electroencephalogram interface apparatus, method, and computer program

• US 8,466,875 B2
• Filed: 08/03/2009
• Issued: 06/18/2013
• Est. Priority Date: 01/25/2008
• Status: Active Grant

• Current Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Nakada, Toru, Morikawa, Koji
• Primary Examiner(s): Leiby, Christopher E
• Application Number:
  US12/534,318
• Publication Number:
  US 20090289895A1
• Time in Patent Office: 1,415 Days
• Field of Search: 345156-158, 600/544
• US Class Current: 345/157
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection

Abstract

An electroencephalogram interface system includes: sections for measuring an electroencephalogram and an eye movement; an output section for presenting on a screen an option related to a device operation; a highlight determination section for, if a predetermined time has elapsed since a rotational angular velocity of the eye movement becomes equal to or less than a threshold value, identifying a region of the screen in which the user is fixing one'"'"'s gaze based on the eye movement, and determining an option to be highlighted; an interface section for highlighting the determined option, and determining an operation of the device based on an event-related potential in the signal based on the timing of highlighting the option; and a timing adjustment section for adjusting a timing of beginning highlighting based on the eye movement after a process of displaying the option on the screen is begun and until the option is displayed on the screen.

Claims

1. An electroencephalogram interface system for controlling an operation of a device by utilizing an electroencephalogram signal of a user, comprising:

an electroencephalogram measurement section for measuring an electroencephalogram signal of the user;

an eye movement measurement section for measuring an eye movement of the user;

an output section for presenting on a screen an interface screen containing an option which is related to an operation of a device;

a highlight determination section for determining whether a predetermined time has elapsed or not since a point in time when a rotational angular velocity of the eye movement becomes equal to or less than a prestored threshold value, and if the predetermined time has elapsed, identifying a region of the interface screen in which the user is fixing his or her gaze based on the eye movement, and determining an option to be highlighted;

an interface section for highlighting the option determined by the highlight determination section, distinguishing a component of an event-related potential contained in the electroencephalogram signal based on the timing of highlighting the option as a starting point, and determining an operation of the device based on the distinguished component; and

a timing adjustment section for adjusting a time period before beginning highlighting of the option, based on the eye movement relative to a position of the option after a process of displaying the interface screen is begun and until the interface screen.

2. The electroencephalogram interface system of claim 1, wherein, if an amount of change in the eye movement becomes equal to or greater than the prestored threshold value, the timing adjustment section causes highlighting of the option to be begun when the amount of change in the eye movement has remained equal to or less than the prestored threshold value for a time which is shorter than the predetermined time after the option on the screen is displayed.

3. The electroencephalogram interface system of claim 1, wherein, if the user fixes his or her gaze in a region of the screen after the option on the screen is displayed, the timing adjustment section measures a fixation time in each fixation region, and, if an average value of the measurement time becomes equal to or greater than the predetermined time, causes the option to be highlighted when an amount of change in the eye movement has remained equal to or less than the prestored threshold value for a time which is longer than the predetermined time.

4. The electroencephalogram interface system of claim 1, wherein the highlight determination section determines an option which is contained in the region of the screen as an option to be highlighted.

5. The electroencephalogram interface system of claim 1, wherein the interface section changes, on the screen, at least one of luminance, hue, and magnitude of the option having been determined by the highlight determination section, thereby highlighting the determined option.

6. The electroencephalogram interface system of claim 1, wherein the interface section changes a method of highlighting depending on the number of options determined by the highlight determination section.

7. The electroencephalogram interface system of claim 6, wherein, if there is a plurality of options determined by the highlight determination section, the plurality of options are highlighted randomly or consecutively with a predetermined highlighting interval.

8. The electroencephalogram interface system of claim 1, wherein the highlight determination section determines no option to be highlighted if no option exists in the region of the screen.

9. The electroencephalogram interface system of claim 1, further comprising an interval adjustment section for adjusting a time interval of highlighting based on the eye movement.

10. The electroencephalogram interface system of claim 9, wherein the interval adjustment section adjusts a time interval of highlighting based on the eye movement after the process of displaying the option on the screen is begun and until the option is displayed on the screen.

11. The electroencephalogram interface system of claim 10, wherein, if an amount of change in the eye movement becomes equal to or greater than the prestored threshold value, the interval adjustment section adjusts the time interval of highlighting to be shorter than a setting value when an amount of change in the eye movement has remained equal to or less than the prestored threshold value for a time which is shorter than the predetermined time after the option on the screen is displayed.

12. The electroencephalogram interface system of claim 10, wherein, if the user fixes his or her gaze in a region of the screen after the option on the screen is displayed, the interval adjustment section measures a fixation time in each fixation region, and, if an average value of the measurement time becomes equal to or greater than the predetermined time, adjusts the time interval of highlighting to be longer than a setting value.

13. The electroencephalogram interface system of claim 1, wherein when an option related to an operation of a device is not displayed, the timing adjustment section adjusts a timing of beginning highlighting, based on the eye movement after a process of displaying the option on the screen is begun and until the option is displayed on the screen.

14. The electroencephalogram interface system of claim 1, wherein when the highlight determination section determines that the rotational angular velocity becomes equal to or greater than the prestored threshold value before the interface screen is displayed, and the rotational angular velocity become equal to or less than the prestored threshold value after the interface screen is displayed, the time adjusting section shortens at least one of the timing before beginning highlighting of the option and an interval between highlighting successive options.

15. The electroencephalogram interface system of claim 1, wherein when the highlight determination section determines that the rotational angular velocity becomes equal to or greater than the prestored threshold value after the interface screen is displayed, the time adjusting section lengthens at least one of the timing before beginning highlighting of the optoin and an interval between highlighting successive options.

16. The electroencephalogram interface system of claim 1, wherein the output section presents on the interface screen a content displaying region and a menu displaying region, and the interface section begins highlighting the option when the highlighting determination section determines that the user is fixing his or her gaze on the menu displaying region for the predetermined time.

17. An electroencephalogram interface apparatus for use in an electroencephalogram interface system for presenting on a screen an interface screen containing an option which is related to an operation of a device, and controlling an operation of the device by utilizing an eye movement and an electroencephalogram signal of the user, wherein,

the electroencephalogram interface system includes an electroencephalogram measurement section for measuring an electroencephalogram signal of the user, and an eye movement measurement section for measuring an eye movement of the user,

the electroencephalogram interface apparatus comprising:

a highlight determination section for determining whether a predetermined time has elapsed or not since a point in time when a rotational angular velocity of the eye movement received from the eye movement measurement section becomes equal to or less than a prestored threshold value, and if the predetermined time has elapsed, identifying a region of the interface screen in which the user is fixing his or her gaze based on the eye movement, and determining an option to be highlighted;

an interface section for highlighting the option determined by the highlight determination section, distinguishing a component of an event-related potential contained in the electroencephalogram signal received from the electroencephalogram measurement section based on the timing of highlighting the option as a starting point, and determining an operation of the device based on the distinguished component; and

a timing adjustment section for adjusting a time period before beginning highlighting of the option, based on the eye movement relative to a position of the option after a process of displaying the interface screen is begun and until the interface screen is displayed.

18. A method for use in an electroencephalogram interface system for controlling an operation of a device by utilizing an electroencephalogram signal of a user, comprising the steps of:

measuring an electroencephalogram signal of the user;

measuring an eye movement of the user;

presenting on a screen an interface screen containing an option which is related to an operation of a device;

determining whether a predetermined time has elapsed or not since a point in time when a rotational angular velocity of the eye movement becomes equal to or less than a prestored threshold value;

if the predetermined time has elapsed, identifying a region of the interface screen in which the user is fixing his or her gaze based on the eye movement, and determining an option to be highlighted;

highlighting the determined option, and distinguishing a component of an event-related potential contained in the electroencephalogram signal based on the timing of highlighting the option as a starting point;

determining an operation of the device based on the distinguished component; and

adjusting a time period before beginning highlighting of the option based on the eye movement relative to a position of the option after a process of displaying the interface screen is begun and until the interface screen is displayed.

19. A computer program stored on a non-transitory computer-readable medium and executed by an electroencephalogram interface apparatus,

the electroencephalogram interface apparatus being incorporated in an electroencephalogram interface system for controlling an operation of a device by utilizing an electroencephalogram signal of a user,

the electroencephalogram interface system including

an electroencephalogram measurement section for measuring an electroencephalogram signal of the user,

an eye movement measurement section for measuring an eye movement of the user, and

an output section for presenting on a screen an interface screen containing an option which is related to an operation of a device, wherein

the computer program causes a computer of the electroencephalogram interface apparatus to execute the steps of:

determining whether a predetermined time has elapsed or not since a point in time when a rotational angular velocity of the eye movement measured by the eye movement measurement section becomes equal to or less than a prestored threshold value;

if the predetermined time has elapsed, identifying a region of the interface screen in which the user is fixing his or her gaze based on the eye movement, and determining an option to be highlighted;

highlighting the determined option, and distinguishing a component of an event-related potential contained in the electroencephalogram signal based on the timing of highlighting the option as a starting point;

determining an operation of the device based on the distinguished component; and

adjusting a time period before beginning highlighting of the option based on the eye movement relative to a position of the option after a process of displaying the interface screen is begun and until the interface screen is displayed.