METHOD AND APPARATUS FOR EARLY DETECTION OF DYNAMIC ATTENTIVE STATES FOR PROVIDING AN INATTENTIVE WARNING

• US 20140210978A1
• Filed: 01/25/2013
• Published: 07/31/2014
• Est. Priority Date: 01/25/2013
• Status: Active Grant

• Current Assignee: Central Michigan University, Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Gunaratne, Pujitha, Backs, Richard W.
• Granted Patent: US 8,847,771 B2
• Time in Patent Office: Days
• US Class Current: 348/77
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: A61B 5/1103 Detecting eye twinkling
• CPC Class Codes: A61B 5/1128 using image analysis A61B5/1127 takes precedence
• CPC Class Codes: A61B 5/163 by tracking eye movement, gaze, or pupil change
• CPC Class Codes: A61B 5/18 for vehicle drivers or machine operators
• CPC Class Codes: A61B 5/6893 Cars
• CPC Class Codes: B60K 28/066 actuating a signalling device B60K28/063 takes precedence
• CPC Class Codes: G06V 20/597 Recognising the driver's state or behaviour, e.g. attention or drowsiness
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G08B 21/0476 Cameras to detect unsafe condition, e.g. video cameras
• CPC Class Codes: G08G 1/165 for passive traffic, e.g. including static obstacles, trees
• CPC Class Codes: G08G 1/166 for active traffic, e.g. moving vehicles, pedestrians, bikes
• CPC Class Codes: Y02T 10/84 Data processing systems or methods, management, administration

Abstract

A method and apparatus for determining an inattentive state of an operator of a vehicle and for providing information to the operator of the vehicle by obtaining face images of the operator of the vehicle, obtaining images of an environment of the vehicle, determining one or more areas of interest in the environment of the vehicle based on the images of the environment, obtaining, from a relevance and priority database, relevance and priority values corresponding to the one or more areas of interest, determining a probability of attention of the operator of the vehicle to the one or more areas of interest based on the images of the environment and the relevance and priority values, determining an attention deficiency based on the determined probability of attention and the face images, and providing the information to the operator of the vehicle based on the determined attention deficiency.

Claims

1. A method for determining an inattentive state of an operator of a vehicle and for providing information to the operator of the vehicle, the method comprising:

obtaining, via a first camera, face images of the operator of the vehicle;

obtaining, via a second camera, images of an environment of the vehicle;

determining one or more areas of interest in the environment of the vehicle based on the images of the environment of the vehicle;

obtaining, from a relevance and priority database, relevance and priority values corresponding to the one or more areas of interest;

determining a probability of attention of the operator of the vehicle to the one or more areas of interest based on the images of the environment of the vehicle and the relevance and priority values;

determining an attention deficiency based on the determined probability of attention and eye movements of the operator of the vehicle, the eye movements being based on the face images; and

providing, via a warning/guidance device, the information to the operator of the vehicle based on the determined attention deficiency.

2. The method of claim 1, wherein the one or more areas of interest includes at least one of a pedestrian, a cyclist, a traffic light, a traffic sign, an animal, an object, or another vehicle.

3. The method of claim 1, wherein the step of determining the probability of attention includes:

determining linear distances of the one or more areas of interest from the vehicle;

determining saliencies of the one or more areas of interest;

detecting event frequencies within the one or more areas of interest; and

determining the probability of attention of the operator of the vehicle to the one or more areas of interest based on the linear distances, the saliencies, the event frequencies, and the relevance and priority values.

4. The method of claim 1, wherein the step of determining the attention deficiency includes:

detecting facial features of the operator of the vehicle by feature point extraction algorithms based on the face images, the facial features including one or more of eye corners, upper eyelids, or lower eyelids;

determining an iris location based on the detected facial features;

determining gaze pointers based on the iris location and the images of the environment of the vehicle;

determining gaze fixation dwell times based on the gaze pointers;

determining an observed attention level based on gaze fixation dwell times; and

determining the attention deficiency based on the observed attention level and the probability of attention.

5. The method of claim 3, wherein the step of determining the linear distances includes:

setting a linear distance of an area of interest to zero when an angle between a line of sight of the operator of the vehicle and the line of sight corresponding to the area of interest is greater than or equal to 90 degrees; and

setting the linear distance of the area of interest to a distance between the area of interest and the vehicle divided by a cosine of the angle between the line of sight of the operator of the vehicle and the line of sight corresponding to the area of interest when said angle is smaller than 90 degrees.

6. The method of claim 1, wherein the step of providing the information includes:

providing, via the warning/guidance device, to the operator of the vehicle, based on the determined attention deficiency, one or more of a failed-to-look warning, a look-but-fail-to-see warning, an audio/visual warning, or guidance information.

7. An apparatus for determining an inattentive state of an operator of a vehicle and for providing information to the operator of the vehicle, the apparatus comprising:

a first camera which takes face images of the operator of the vehicle;

a second camera which takes images of an environment of the vehicle;

a relevance and priority database storing a plurality of relevance and priority values corresponding to a plurality of areas of interest;

a warning/guidance device which provides the information to the operator of the vehicle; and

a controller configured to

obtain, via the first camera, the face images of the operator of the vehicle;

obtain, via the second camera, the images of the environment of the vehicle;

determine one or more areas of interest in the environment of the vehicle based on the images of the environment of the vehicle;

obtain, from the relevance and priority database, relevance and priority values corresponding to the one or more areas of interest;

determine a probability of attention of the operator of the vehicle to the one or more areas of interest based on the images of the environment of the vehicle and the relevance and priority values;

determine an attention deficiency based on the determined probability of attention and eye movements of the operator of the vehicle, the eye movements being based on the face images; and

provide, via the warning/guidance device, the information to the operator of the vehicle based on the determined attention deficiency.

8. A non-transitory computer-readable medium storing a program that, when executed by a processor, causes the processor to perform a method for determining an inattentive state of an operator of a vehicle and for providing information to the operator of the vehicle, the method comprising comprising:

obtaining, via a first camera, face images of the operator of the vehicle;

obtaining, via a second camera, images of an environment of the vehicle;

determining one or more areas of interest in the environment of the vehicle based on the images of the environment of the vehicle;

obtaining, from a relevance and priority database, relevance and priority values corresponding to the one or more areas of interest;

determining a probability of attention of the operator of the vehicle to the one or more areas of interest based on the images of the environment of the vehicle and the relevance and priority values;

determining an attention deficiency based on the determined probability of attention and eye movements of the operator of the vehicle, the eye movements being based on the face images; and

providing, via a warning/guidance device, the information to the operator of the vehicle based on the determined attention deficiency.