LINE-OF-SIGHT DETECTION APPARATUS AND METHOD THEREOF

• US 20120189160A1
• Filed: 07/28/2011
• Published: 07/26/2012
• Est. Priority Date: 08/03/2010
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Kaneda, Yuji, Matsugu, Masakazu, Mori, Katsuhiko
• Granted Patent: US 8,891,819 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

A line-of-sight detection apparatus includes a detection unit configured to detect a face from image data, a first extraction unit configured to extract a feature amount corresponding to a direction of the face from the image data, a calculation unit configured to calculate a line-of-sight reliability of each of a right eye and a left eye based on the face, a selection unit configured to select an eye according to the line-of-sight reliability, a second extraction unit configured to extract a feature amount of an eye region of the selected eye from the image data, and an estimation unit configured to estimate a line of sight of the face based on the feature amount corresponding to the face direction and the feature amount of the eye region.

Claims

1. A line-of-sight detection apparatus comprising:

a detection unit configured to detect a face from image data;

a first extraction unit configured to extract a feature amount corresponding to a direction of the face from the image data;

a calculation unit configured to calculate a line-of-sight reliability of each of a right eye and a left eye based on the face;

a selection unit configured to select an eye according to the line-of-sight reliability;

a second extraction unit configured to extract a feature amount of an eye region of the selected eye from the image data; and

an estimation unit configured to estimate a line of sight of the face based on the feature amount corresponding to the face direction and the feature amount of the eye region.

2. The line-of-sight detection apparatus according to claim 1, wherein the calculation unit calculates the line-of-sight reliability as a binary value representing whether the line of sight is reliable.

3. The line-of-sight detection apparatus according to claim 1, wherein the calculation unit calculates the line-of-sight reliability based on a spatial arrangement relation of organs of the face.

4. The line-of-sight detection apparatus according to claim 1, wherein the calculation unit calculates the line-of-sight reliability based on a distribution of organs of the face.

5. The line-of-sight detection apparatus according to claim 1, wherein the calculation unit calculates the line-of-sight reliability based on a size of an eye or a pupil of the face.

6. The line-of-sight detection apparatus according to claim 1, wherein the calculation unit calculates the line-of-sight reliability based on the feature amount corresponding to the face direction extracted by the first extraction unit.

7. The line-of-sight detection apparatus according to claim 1, wherein the feature amount is at least one of an edge, luminance, color, frequency, and histogram thereof.

8. An imaging apparatus comprising:

the line-of-sight detection apparatus according to claim 1; and

an imaging unit,

wherein the detection unit detects the face from image data acquired by the imaging unit,

wherein the estimation unit estimates whether the line of sight is directed in a direction of the imaging apparatus, and

wherein the imaging unit records the image data when the line of sight is estimated to be directed in the direction of the imaging apparatus.

9. A line-of-sight detection apparatus comprising:

a detection unit configured to detect a face from image data;

a first extraction unit configured to extract a feature amount corresponding to a direction of the face from the image data;

a second extraction unit configured to extract a feature amount of a right eye region and a feature amount of a left eye region of the face from the image data;

a first estimation unit configured to estimate a first line-of-sight based on the feature amount corresponding to the face direction and the feature amount of the right eye region, and to estimate a second line-of-sight based on the feature amount corresponding to the face direction and the feature amount of the left eye region; and

a second estimation unit configured to estimate a line of sight of the face based on the first line-of-sight and the second line-of-sight.

10. The line-of-sight detection apparatus according to claim 9, further comprising a calculation unit configured to calculate a reliability of the first line-of-sight and a reliability of the second line-of-sight based on the face detected by the detection unit,

wherein the second estimation unit estimates the line of sight of the face by putting weight on the first line-of-sight according to the first line-of-sight reliability and putting weight on the second line-of-sight according to the second line-of-sight reliability.

11. An imaging apparatus comprising:

the line-of-sight detection apparatus according to claim 9; and

an imaging unit,

wherein the detection unit detects the face from image data acquired by the imaging unit,

wherein the second estimation unit estimates whether the line of sight is directed in a direction of the imaging apparatus, and

wherein the imaging unit records the image data when the line of sight is estimated to be directed in the direction of the imaging apparatus.

12. A line-of-sight detection method comprising:

detecting a face from image data;

extracting a feature amount corresponding to a direction of the face from the image data;

calculating a line-of-sight reliability of each of a right eye and a left eye based on the face;

selecting an eye according to the line-of-sight reliability;

extracting a feature amount of an eye region of the selected eye from the image data; and

estimating a line of sight of the face based on the feature amount corresponding to the face direction and the feature amount of the eye region.

13. A storage medium storing a program that causes a computer to execute the line-of-sight detection method according to claim 12.

14. A line-of-sight detection method comprising:

detecting a face from image data;

extracting a feature amount corresponding to a direction of the face from the image data;

extracting a feature amount of a right eye region and a feature amount of a left eye region of the face from the image data;

estimating a first line-of-sight based on the feature amount corresponding to the face direction and the feature amount of the right eye region, and estimating a second line-of-sight based on the feature amount corresponding to the face direction and the feature amount of the left eye region; and

estimating a line of sight of the face based on the first line-of-sight and the second line-of-sight.

15. A storage medium storing a program that causes a computer to execute the line-of-sight detection method according to claim 14.