System for classifying an individual's gaze direction

• US 6,154,559 A
• Filed: 10/01/1998
• Issued: 11/28/2000
• Est. Priority Date: 10/01/1998
• Status: Expired due to Term

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Information Technology Center America, Inc. (Mitsubishi Electric Corporation)
• Inventors: Beardsley, Paul Anthony
• Primary Examiner(s): Mancuso, Joseph
• Assistant Examiner(s): Bali, Vikkram
• Application Number:
  US09/164,734
• Time in Patent Office: 789 Days
• Field of Search: 382/103, 382/117, 382/118, 382/100, 382/159, 382/104, 382/168, 382/170, 345/7, 345/8, 345/326, 345/146, 345/327, 340/907, 340/908, 340/576, 340/575, 340/573.1, 348/113, 348/77, 348/78, 348/118, 348/169, 348/172
• US Class Current: 382/103
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61B 5/1176 Recognition of faces
• CPC Class Codes: A61B 5/7264 Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems neural networks per se G06N3/00; expert systems per se G06N5/00
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

A system is provided to classify the gaze direction of an individual obseng a number of surrounding objects. The system utilizes a qualitative approach in which frequently occurring head poses of the individual are automatically identified and labelled according to their association with the surrounding objects. In conjunction with processing of eye pose, this enables the classification of gaze direction. In one embodiment, each observed head pose of the individual is automatically associated with a bin in a "pose-space histogram". This histogram records the frequency of different head poses over an extended period of time. Given observations of a car driver, for example, the pose-space histogram develops peaks over time corresponding to the frequently viewed directions of toward the dashboard, toward the mirrors, toward the side window, and straight-ahead. Each peak is labelled using a qualitative description of the environment around the individual, such as the approximate relative directions of dashboard, mirrors, side window, and straight-ahead in the car example. The labelled histogram is then used to classify the head pose of the individual in all subsequent images. This head pose processing is augmented with eye pose processing, enabling the system to rapidly classify gaze direction without accurate a priori information about the calibration of the camera utilized to view the individual, without accurate a priori 3D measurements of the geometry of the environment around the individual, and without any need to compute accurate 3D metric measurements of the individual'"'"'s location, head pose or eye direction at run-time.

Claims

1. A system to classify the gaze direction of an individual observing a number of surrounding objects, comprising:

means for observing the head pose of an individual;

means for generating a number of templates corresponding to different gaze directions of said individual as said individual looks in different gaze directions;

means for matching an image of said individual with the template which is most similar in appearance;

means utilizing said matched templates for generating a pose-space histogram recording the frequency of different head poses over a predetermined period of time, said histogram recording as peaks over time frequently viewed directions towards said number of surrounding objects, with each peak labelled utilizing a qualitative description of the environment around said individual;

means coupled to said head pose observing means and said pose-space histogram for determining that peak which most closely corresponds to the current gaze direction of said individual, whereby gaze direction is classified without direct measurement of head position or eye direction.

2. The system of claim 1 and further including means for augmenting said classification to permit more rapid classification of gaze including means for establishing eye pose.

3. The system of claim 2, wherein said means for establishing eye pose includes means for determining eyelid droop.

4. The system of claim 3, wherein said means for determining eyelid droop includes means for identifying the area in the image occupied by the eye using color, means for classifying the shape of the segmented eye using an equivalent rectangle and, means for determining the ratio of the height of said rectangle to the width thereof, whereby said ratio determines eyelid droop and thus whether the eye is looking straight-ahead or in a downward direction.

5. The system of claim 1 wherein said means for matching images with a template includes cross-correlation of an image of said individual with all of said templates in order to identify the most similar template.

6. The system of claim 1 and further including means for storing said templates during an initialization phase such that said templates are available during subsequent real-time classification, thereby eliminating time-consuming template generation during classification.

7. The system of claim 1 and further including means for culling out templates which are a poor match for the image of said individual.

8. The system of claim 7 wherein said culling means includes means for segmenting the skin portion of said image using color to provide a unique shape signature corresponding to said segmented skin area made available by said color segmentation, means for generating shape signatures for each of said templates, means for matching the shape signature of said image with shape signatures of said templates and means for eliminating from consideration those templates having shape signatures exhibiting poor correspondence.

9. The system of claim 8 wherein said means for segmenting the skin portion of said image using color to provide a unique shape signature corresponding to said segmented skin area includes means for generating 1D projections and moments of segmented skin areas, thus to provide said signature.