Gaze detection apparatus and its method as well as information display apparatus

• US 5,912,721 A
• Filed: 03/12/1997
• Issued: 06/15/1999
• Est. Priority Date: 03/13/1996
• Status: Expired due to Fees

• Current Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Original Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Inventors: Okamoto, Yasukazu, Yamaguchi, Osamu, Fukui, Kazuhiro
• Primary Examiner(s): Dang, Hung Xuan
• Application Number:
  US08/816,453
• Time in Patent Office: 825 Days
• Field of Search: 351/210, 351/209, 351/206, 351/205, 351/200, 351/211, 351/221, 351/246, 351/247
• US Class Current: 351/210
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

Apparatus disclosed herein includes an information presentation section 120 for performing presentation of information selected, a gaze-point detecting section 130 for estimating the user'"'"'s gaze point, a display change section 150 for changing the display method of a display portion of information being selected by the user, a prediction section 140 for predicting information to be selected by the user, an intention transmission section 160 for transmitting to the system determination of information which the user wants to input, and a result output section 170 for storing therein the selected information and for sending forth it to another device or apparatus, thereby enabling easy and rapid selection of information by use of the user'"'"'s gaze-point information.

Claims

1. A gaze detection apparatus for detecting a gaze point of a user who makes use of a system or apparatus operable by human'"'"'s manual operation, characterized by comprising:

an image input means for sensing an image of said user;

eye detection means for detecting eyes of said user from said image as sensed by said image input means;

pupil detection means for obtaining positions of pupils in said eyes detected by said eye detection means;

three or more light emission means for emitting light toward said user;

reference point detection means for detecting respective reflecting positions of illumination light of said three or more light emission means on the eyes of said user; and

gaze point detection means for detecting the gaze point of said user based on the three or more reflecting positions detected by said reference point detection means and the positions of said pupils as detected by said pupil detection means.

2. The gaze detection apparatus according to claim 1, characterized in that said eye detection means compares said image sensed by said image input means with the three or more reflecting positions detected by said reference point detection means thereby determining as eyes the nearby region of said reflecting positions in said image.

3. The gaze detection apparatus according to claim 1, characterized in that said gaze point detection means detects three or more reflecting positions to be detected by said reference point detection means and said pupil positions to be detected by said pupil detection means with the light emission position of each said three or more light emission means being as a reference.

4. A gaze detection method for detecting a gaze point of a user who makes use of a system or apparatus operable by human'"'"'s manual operation, characterized by comprising:

an image input step for sensing an image of said user;

an eye detection step for detecting eyes of said user from said image as sensed by said image input means;

a pupil detection step for obtaining positions of pupils in said eyes detected at said eye detection step;

a reference point detection step for detecting on the eyes of said user respective reflecting positions of illumination light of three or more light emission means for emitting light toward said user; and

a gaze point detection step for detecting a gaze point of said user based on the three or more reflecting positions detected at said reference point detection step and the positions of said pupils detected at said pupil detection step.

5. The gaze detection method according to claim 4, characterized in that said eye detection step compares said image sensed at said image input step with the three or more reflecting positions detected at said reference point detection step thereby determining as eyes the nearby region of said reflecting positions in said image.

6. The gaze detection method according to claim 4, characterized in that said gaze point detection step detects three or more reflecting positions to be detected at said reference point detection step and said pupil positions to be detected at said pupil detection step with the light emission position of each said three or more light emission means being as a reference.

7. A gaze detection apparatus for detecting a gaze point of a user who makes use of a system or apparatus operable by human'"'"'s manual operation, characterized by comprising:

image input means for inputting an image of said user;

face image extraction means for extracting a face image of said user from said image as input by said image input means;

eye'"'"'s inner corner detection means for detecting a position of an inner corner of at least one eye from said face image extracted by said face extraction means;

pupil detection means for detecting a position of a pupil in said eye from the face image extracted by said face extraction means; and

gaze point detection means for obtaining a vector extending from the position of said inner corner of the eye detected by said eye'"'"'s inner corner detection means to the position of said pupil detected by said pupil detection means, and for making use of resultant vector to detect the gaze point of said user.

8. A gaze detection apparatus for detecting a gaze point of a user who makes use of a system or apparatus operable by human'"'"'s manual operation, characterized by comprising:

image input means for inputting an image of said user;

face image extraction means for extracting a face image of said user from said image as input by said image input means;

eye detection means for detecting a position of an inner or outer corner of at least one eye from said face image extracted by said face extraction means;

mouth/nose detection means for detecting a position of a mouth or a nostril from the face image extracted by said face extraction means;

face direction calculation means for calculating a face direction plane connecting between inner corner positions or outer corner positions of both eyes detected by said eye detection means and the position of the mouth or nostril detected by said mouth/nose detection means; and

gaze point detection means for detecting a gaze point of said user based on a perpendicular vector of said face direction plane calculated by said face direction plane calculation means.

9. An information display apparatus for displaying information, characterized by comprising:

gaze point detection means for detecting a gaze point of a user of said information display apparatus;

prediction means for predicting information as required by said user based on said gaze point detected by said gaze point detection means; and

information display means for performing displaying of the information predicted by said prediction means.

10. The information display apparatus according to claim 9, characterized in that said prediction means comprises:

prediction change means for, upon movement of said gaze point detected by said gaze point detection means, repredicting information as necessary for said user based on said gaze point moved; and

display change means for instructing said information display means so as to display the information repredicted by said prediction change means.

11. The information display apparatus according to claim 9, characterized in that where said gaze point detected by said gaze point detection means is within said information display means, said prediction means predicts information necessary for said user based on such steadily looked information, and that where said gaze point detected by said gaze point detection means is out of said information display means, said prediction means predicts information necessary for said user based on that steadily looked portion.

12. A gaze detection method for detecting a gaze point of a user who makes use of a system or apparatus operable by human'"'"'s manual operation, characterized by comprising:

an image input step for inputting an image of said user;

a face image extraction step for extracting a face image of said user from said image as input at said image input step;

an eye'"'"'s inner corner detection step for detecting a position of an inner corner of at least one eye from said face image extracted at said face extraction step;

a pupil detection step for detecting a position of a pupil in said eye from the face image extracted at said face extraction step; and

a gaze point detection step for obtaining a vector extending from the position of said inner corner of the eye detected at said eye'"'"'s inner corner detection step to the position of said pupil detected at said pupil detection step, and for using resultant vector to detect the gaze point of said user.

13. A gaze detection method for detecting a gaze point of a user who makes use of a system or apparatus operable by human'"'"'s manual operation, characterized by comprising:

an image input step for inputting an image of said user;

a face image extraction step for extracting a face image of said user from said image as input at said image input step;

an eye detection step for detecting a position of an inner or outer corner of at least one eye from said face image extracted at said face extraction step;

a mouth/nose detection step for detecting a position of a mouth or a nostril from the face image extracted at said face extraction step;

a face direction calculation step for calculating a face direction plane connecting between inner corner positions or outer corner positions of both eyes detected at said eye detection step and the position of the mouth or nostrils detected at said mouth/nose detection step; and

a gaze point detection step for detecting a gaze point of said user based on a perpendicular vector of said face direction plane calculated at said face direction plane calculation step.

14. An information display method for displaying information, characterized by comprising:

a gaze point detection step for detecting a gaze point of a user of said information display apparatus;

a prediction step for predicting information as required by said user based on said gaze point detected at said gaze point detection step; and

an information display step for performing displaying of the information predicted at said prediction step.

15. The information display method according to claim 14, characterized in that said prediction step comprises:

a prediction change step for, upon movement of said gaze point detected at said gaze point detection step, repredicting information necessary for said user based on said gaze point moved; and

a display change step for instructing said information display means so as to display the information repredicted at said prediction change step.

16. The information display method according to claim 14, characterized in that where said gaze point detected at said gaze point detection step is within said information display means, said prediction step predicts information necessary for said user based on the steadily looked information thereof, and that where said gaze point detected at said gaze point detection step is out of said information display means, said prediction step predicts information necessary for said user based on that steadily looked portion.

17. An information display apparatus for displaying information, characterized by comprising:

distance measurement means for obtaining a distance between said information display apparatus and a user thereof;

gaze point detection means for detecting a gaze point of said user by selecting a detection method of gaze point in accordance with the distance measured by said distance measurement means;

prediction means for predicting information as required by said user based on said gaze point detected by said gaze point detection means; and

information display means for performing displaying of the information predicted by said prediction means.

18. An information display method for displaying information by use of information display means, said method comprising:

a distance measurement step for obtaining a distance between said information display means and a user thereof;

a gaze point detection step for detecting a gaze point of said user by selecting a detection method of gaze point in accordance with the distance measured at said distance measurement step;

a prediction step for predicting information as required by said user based on said gaze point detected at said gaze point detection step; and

an information display step for performing displaying of the information as predicted at said prediction step.