Eye tracking method based on correlation and detected eye movement

• US 7,331,671 B2
• Filed: 03/29/2004
• Issued: 02/19/2008
• Est. Priority Date: 03/29/2004
• Status: Active Grant

• Current Assignee: Aptiv Technologies AG (Aptiv PLC)
• Original Assignee: Delphi Technologies, Inc. (BorgWarner Incorporated)
• Inventors: Hammoud, Riad I.
• Primary Examiner(s): Sugarman; Scott J
• Assistant Examiner(s): PINKNEY, DAWAYNE
• Application Number:
  US10/813,192
• Publication Number:
  US 20050213792A1
• Time in Patent Office: 1,422 Days
• Field of Search: 351/205, 351208-210
• US Class Current: 351/210
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

An eye tracking method tracks a subject'"'"'s eye template by correlation between successive video frames, and periodically updates the eye template based on detected characteristic eye or eyelid movement such as blinking, eyelash movement and iris movement. In the absence of eyelid motion detection, a state vector corresponding to the center of the subject'"'"'s eye is determined by a correlation technique, and when eyelid motion is detected, the state vector is determined based on the location of the detected motion.

Claims

1. A method of tracking movement of a subject'"'"'s eye between first and second successively generated video images after a position of the subject'"'"'s eye in said first video image has been identified, comprising the steps of:

defining a state vector for the first video image corresponding to the identified position of the subject'"'"'s eye;

defining an eye template in said second video image based on said state vector, and defining a search window comprising said eye template and a portion of the second video image surrounding said eye template;

forming a difference image corresponding to differences between said search window and a corresponding portion of said first video image;

identifying at least one eye movement candidate region in the difference image;

determining a centroid of the eye movement candidate region and extracting a patch from the search window based on the determined centroid;

when the extracted search window patch has the appearance of an eye, identifying eyelid motion between the first and second video images and updating the state vector for the second video image according to the determined centroid; and

when the extracted search window patch does not have the appearance of an eye, identifying a lack of eyelid motion between the first and second video images, and updating the state vector for the second video image according to the following steps;

computing correlation values based on a comparison of said eye template with different regions of said search window, and selecting a first region for which the computed correlation value is highest;

establishing an eye model defining image characteristics of the subject'"'"'s eye and a non-eye model defining image characteristics of facial features other than the subject'"'"'s eye;

computing deviations of the search window regions from said eye model, and selecting a second region for which the computed deviation is lowest;

updating the state vector for the second video image according to a center of the first selected region if said first selected region is determined to be more reliable than said second selected region; and

updating the state vector for the second video image according to a center of the second selected region if said second selected region is determined to be more reliable than said first selected region.

2. The method of claim 1, including the steps of:

establishing an eye model defining image characteristics of the subject'"'"'s eye and a non-eye model defining image characteristics of facial features other than the subject'"'"'s eye;

computing deviations of the extracted search window patch from said eye model and said non-eye model; and

determining that the extracted search window patch has the appearance of an eye when the deviation of the extracted search window patch from the non-eye model is greater than the deviation of the the extracted search window patch from the eye model.

3. The method of claim 1, including the steps of:

establishing an eye model defining image characteristics of the subject'"'"'s eye and a non-eye model defining image characteristics of facial features other than the subject'"'"'s eye;

computing deviations of the extracted search window patch from said eye model and said non-eye model; and

determining that the extracted search window patch does not have the appearance of an eye when the deviation of the extracted search window patch from the eye model is greater than the deviation of the extracted search window patch from the non-eye model.

4. The method of claim 1, including the step of:

determining that said first selected region is more reliable than said second selected region when the correction value corresponding to said first selected region exceeds a correlation threshold, and the deviation of said second selected region from said eye model is greater than a deviation of the second selected region from said non-eye model.

5. The method of claim 1, including the step of:

determining that said second selected region is more reliable than said first selected region when the correction value corresponding to said first selected region is less than a correlation threshold, and the deviation of said second selected region from said eye model is less than a deviation of the second selected region from said non-eye model.

6. The method of claim 1, including the steps of:

computing a first variance of search window patches surrounding a center of the first selected region, and a second variance of search window patches surrounding a center of the second selected region;

determining that said first selected region is more reliable than said second selected region when the first variance exceeds the second variance; and

determining that said second selected region is more reliable than said first selected region when the second variance exceeds the first variance.