Gaze position detection apparatus and method

• US 6,246,779 B1
• Filed: 12/11/1998
• Issued: 06/12/2001
• Est. Priority Date: 12/12/1997
• Status: Expired due to Term

• Current Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Original Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Inventors: Yamaguchi, Osamu, Fukui, Kazuhiro
• Primary Examiner(s): Bella, Matthew C.
• Assistant Examiner(s): CHOOBIN, BARRY
• Application Number:
  US09/209,266
• Time in Patent Office: 914 Days
• Field of Search: 382/103, 382/115, 382/118, 382/100, 351/221, 424/133.1, 600/558, 396/51
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

A gaze position detection apparatus. A dictionary section previously stores a plurality of dictionary patterns representing a user'"'"'s image including pupils. An image input section inputs an image including the user'"'"'s pupils. A feature point extraction section extracts at least one feature point from a face area on the input image. A pattern extraction section geometrically transforms the input image according to a relative position of the feature point on the input image, and extracts a pattern including the user s pupils from the transformed image. A gaze position determination section compares the extracted pattern with the plurality of dictionary patterns, and determines the user'"'"'s gaze position according to the dictionary pattern matched with the extracted pattern.

Claims

1. A gaze position detection apparatus, comprising:

dictionary means for storing a plurality of dictionary patterns representing a user'"'"'s face image including pupils, each dictionary pattern corresponding to each of a plurality of indexes on a display, an image for each dictionary pattern being input at a predetermined camera position while the user is gazing at each index on the display, the image being geometrically transformed as the dictionary pattern according to a relative position of a feature point of the user'"'"'s face area on the image;

image input means for inputting an image including the user'"'"'s pupils at the predetermined camera position in the user'"'"'s operation mode;

feature point extraction means for extracting at least one feature point from a face area on the input image;

pattern extraction means for geometrically transforming the input image according to a relative position of the feature point on the input image, and for extracting a pattern including the user'"'"'s pupils from the transformed image; and

gaze position determination means for comparing the extracted pattern with each of the plurality of dictionary patterns, and for determining the users gaze position as one index on the display according to the dictionary pattern matched with the extracted pattern.

2. The gaze position detection apparatus according to claim 1, wherein the image for the dictionary pattern is previously inputted by said image input means fixedly located while the user is gazing at each index on the display.

3. The gaze position detection apparatus according to claim 2, further comprising a dictionary management means for extracting a basis coordinate of the user'"'"'s face area as the feature point, calculating the relative position of the basis coordinate of the user'"'"'s face area on the image for the dictionary pattern, geometrically transforming the image according to the relative position, and extracting the face area and at least one of the right and left pupils from the transformed image as one of the dictionary patterns.

4. The gaze position detection apparatus according to claim 3, wherein said dictionary management means manages correspondence between each of the plurality of dictionary patterns and each of a plurality of indexes on the display.

5. The gaze position detection apparatus according to claim 1,

wherein said feature point extraction means is for extracting a basis coordinate of the face area as the feature point, and wherein the basis coordinate is a point calculated from at least one of the right and left pupils and the two nostrils on the face area.

6. The gaze position detection apparatus according to claim 5, wherein said pattern extraction means calculates the relative position between the basis coordinate of the face area and a center position on the input image, and executes Affine-transformation to the input image as a quantity corresponding to the relative position.

7. The gaze position detection apparatus according to claim 6,

wherein said pattern extraction means rotates the input image about an angle corresponding to the relative position, and parallelly moves the basis coordinate on the rotated image a distance corresponding to the relative position.

8. The gaze position detection apparatus according to claim 7,

wherein said pattern extraction means extracts the user'"'"'s face area and right and left pupil areas from the rotated and parallel-moved image as the extracted pattern to compare with the plurality of dictionary patterns.

9. The gaze position detection apparatus according to claim 1,

wherein said pattern extraction means partially converts image intensities of pixels included in the input image according to the relative position.

10. The gaze position detection apparatus according to claim 4,

further comprising a cursor control means for positioning a cursor at the index corresponding to the dictionary pattern matched with the extracted pattern as the user'"'"'s gaze position.

11. The gaze position detection apparatus according to claim 1, an input position of the image of said image input means is the same as an input position of the plurality of the dictionary patterns of said image input means.

12. The gaze position detection apparatus according to claim 5,

wherein the basis coordinate is a center point between the right and left pupils and the two nostrils on the face area or a center point between the right and left pupils.

13. A gaze position detected method, comprising the steps of:

storing a plurality of dictionary patterns representing a user'"'"'s face image including pupils, each dictionary pattern corresponding to each of a plurality of indexes on a display, an image for each dictionary pattern being previously input at a predetermined camera position while the user is gazing at each index on the display, the image being geometrically transformed as the dictionary pattern according to a relative position of a feature point of the user'"'"'s face area on the image;

inputting an image including the user'"'"'s pupils through an image input unit at the predetermined camera position in the user'"'"'s operation mode;

extracting at least one feature point from a face area on the input image;

geometrically transforming the input image according to a relative position of the feature point on the input image;

extracting a pattern including the user'"'"'s pupils from the transformed image;

comparing the extracted pattern with each of the plurality of dictionary patterns; and

determining the user'"'"'s gaze position as one index on the display according to the dictionary pattern matched with the extracted pattern.

14. The gaze position detection method according to claim 13, further comprising the step of:

previously inputting the image for the dictionary pattern through the image input unit fixedly located while the user is gazing at each index on the display.

15. The gaze position detection method according to claim 14, further comprising the steps of:

extracting a basis coordinate of the user'"'"'s face area as the feature point;

calculating the relative position of the basis coordinate of the user'"'"'s face area on the image for the dictionary pattern;

geometrically transforming the image according to the relative position; and

extracting the face area and at least one of the right and left pupils from the transformed image as one of the dictionary patterns.

16. The gaze position detection method according to claim 15, further comprising the step of:

managing correspondence between each of the plurality of dictionary patterns and each of a plurality of indexes on the display.

17. The gaze position detection method according to claim 13, further comprising the step of:

extracting a basis coordinate of the face area as the feature point, the basis coordinate being a point calculated from at least one of the right and left pupils and the two nostrils on the face area.

18. The gaze position detection method according to claim 17, further comprising the steps of:

calculating the relative position between the basis coordinate of the face area and a center position on the input image; and

executing Affine-transformation to the input image as a quantity corresponding to the relative position.

19. The gaze position detection method according to claim 18, further comprising the steps of:

rotating the input image about an angle corresponding to the relative position; and

parallelly moving the basis coordinate on the rotated image a distance corresponding to the relative position.

20. The gaze position detection method according to claim 19, further comprising the step of:

extracting the user'"'"'s face area and right and left pupil areas from the rotated and parallel-moved image as the extracted pattern to compare with the plurality of dictionary patterns.

21. The gaze position detection method according to claim 13, further comprising the step of:

partially converting image intensities of pixels included in the input image according to the relative position.

22. The gaze position detection method according to claim 16, further comprising the step of:

positioning a cursor at the index corresponding to the dictionary pattern matched with the extracted pattern as the user'"'"'s gaze position.

23. The gaze position detection method according to claim 13, an input position of the image of the image input unit is the same as an input position of the plurality of the dictionary patterns of the image input unit.

24. The gaze position detection method according to claim 17,

wherein the basis coordinate is a center point between the right and left pupils and the-two nostrils on the face area or a center point between the right and left pupils.

25. A computer-readable memory, comprising:

instruction means for causing a computer to store a plurality of dictionary patterns representing a user'"'"'s face image including pupils, each dictionary pattern corresponding to each of a plurality of indexes on a display, an image for each dictionary pattern being previously input at a predetermined camera position while the user is gazing at each index on the display, the image being geometrically transformed as the dictionary pattern according to a relative position of a feature point of the user'"'"'s face area on the image;

instruction means for causing a computer to input an image including the user'"'"'s pupils through an image input unit at the predetermined camera position in the users operation mode;

instruction means for causing a computer to extract at least one feature point from a face area on the input image;

instruction means for causing a computer to geometrically transform the input image according to a relative position of the feature point on the input image;

instruction means for causing a computer to extract a pattern including the user'"'"'s pupils from the transformed image;

instruction means for causing a computer to compare the extracted pattern with each of the plurality of dictionary patterns; and

instruction means for causing a computer to determine the user'"'"'s gaze position as one index on the display according to the dictionary pattern matched with the extracted pattern.