Method, apparatus, and program for detecting sightlines

• US 20070230797A1
• Filed: 03/29/2007
• Published: 10/04/2007
• Est. Priority Date: 03/30/2006
• Status: Abandoned Application

• Current Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Original Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Inventors: Hisanaga, Ryuji
• Application Number:
  US11/730,126
• Publication Number:
  US 20070230797A1
• Time in Patent Office: Days
• US Class Current: 382/195
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships

Abstract

Detection of sightlines of faces within images is performed efficiently. A facial image is detected from within an entire image. A plurality of eye characteristic points and facial characteristic points are extracted from the detected facial image. Thereafter, eye features and facial features are generated, based on the extracted eye characteristic points and facial characteristic points. A characteristic vector that has the eye features and facial features as vector components is generated. A sightline is detected employing the generated characteristic vector.

Claims

1. A sightline detecting method, comprising the steps of:

detecting a facial image from within an entire image;

extracting a plurality of eye characteristic points from within eyes of the detected facial image;

extracting a plurality of facial characteristic points from facial parts that constitute a face within the facial image;

generating eye features that indicate the gazing direction of the eyes, employing the plurality of extracted eye characteristic points;

generating facial features that indicate the facing direction of the face, employing the plurality of extracted facial characteristic points; and

detecting a sightline, employing the generated eye features and the generated facial features.

2. A sightline detecting apparatus, comprising:

detecting means, for detecting a facial image from within an entire image;

characteristic point extracting means, for extracting a plurality of eye characteristic points from within eyes of the detected facial image, and for extracting a plurality of facial characteristic points from facial parts that constitute a face within the facial image;

feature generating means, for generating eye features that indicate the gazing direction of the eyes, employing the plurality of extracted eye characteristic points, and for generating facial features that indicate the facing direction of the face, employing the plurality of extracted facial characteristic points; and

sightline detecting means, for detecting a sightline, employing the generated eye features and the generated facial features.

3. A sightline detecting apparatus as defined in claim 2, wherein the sightline detecting means detects the sightline by:

generating characteristic vectors having the eye features and the facial features as vector components; and

employing the characteristic vectors to perform pattern classification.

4. A sightline detecting apparatus as defined in claim 3, wherein:

the sightline detecting means has performed machine learning to classify the characteristic vectors into a class of forward facing sightlines and a class of sightlines facing other directions.

5. A sightline detecting apparatus as defined in claim 2, wherein the feature generating means:

calculates the distances between each of the eye characteristic points;

generates the ratios of the calculated distances as the eye features;

calculates the distances between each of the facial characteristic points; and

generates the ratios of the calculated distances as the facial features.

6. A sightline detecting apparatus as defined in claim 2, wherein:

the eye characteristic points are extracted from the pupils, the inner corners, and the outer corners of the eyes; and

the facial characteristic points are extracted from the nose and the lips of the face.

7. A sightline detecting apparatus as defined in claim 2, wherein the face detecting means comprises:

partial image generating means, for generating a plurality of partial images by scanning a subwindow, which is a frame surrounding a set number of pixels; and

face classifiers, for performing final discrimination regarding whether the plurality of partial images represent faces, employing discrimination results of a plurality of weak classifiers.

8. A sightline detecting apparatus as defined in claim 7, wherein:

the face detecting means comprises a plurality of face classifiers corresponding to forward facing faces, faces in profile, and inclined faces; and

a plurality of sightline detecting means are provided corresponding to the forward facing faces, faces in profile, and incline faces detected by the face detecting means.

9. A program that causes a computer to execute a sightline method, comprising the procedures of:

detecting a facial image from within an entire image;

extracting a plurality of eye characteristic points from within eyes of the detected facial image;

extracting a plurality of facial characteristic points from facial parts that constitute a face within the facial image;

generating eye features that indicate the gazing direction of the eyes, employing the plurality of extracted eye characteristic points;

generating facial features that indicate the facing direction of the face, employing the plurality of extracted facial characteristic points; and

detecting a sightline, employing the generated eye features and the generated facial features.

10. A computer readable program having the program of claim 9 recorded therein.