System and method for eye-tracking and blink detection

• US 7,676,063 B2
• Filed: 03/22/2005
• Issued: 03/09/2010
• Est. Priority Date: 03/22/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Cohen, Michael, Wang, Jue
• Primary Examiner(s): Mehta; Bhavesh M
• Assistant Examiner(s): Rush; Eric
• Application Number:
  US11/087,281
• Publication Number:
  US 20060215017A1
• Time in Patent Office: 1,813 Days
• Field of Search: 382/100, 382/103, 382/117, 382168-172
• US Class Current: 382/103
• CPC Class Codes: H04N 7/148 Interfacing a video terminal to a particular transmission medium, e.g. ISDN

Abstract

A real-time low frame-rate video compression system and method that allows the user to perform face-to-face communication through an extremely low bandwidth network. The system and method employs novel eye tracking and blink detection techniques in order to select images for transmission. Experimental results show that the system is superior to more traditional video codecs for low bit-rate face-to-face communication.

Claims

1. A process for tracking eyes and detecting eye blinks, comprising the process actions of:

using a computing device for,

defining eye templates for a person depicted in a video frame;

inputting a video frame of the person'"'"'s face;

using a face detector to find a face box which surrounds the face of the person;

searching the upper part of the face box for eyes using feature based matching to match image patches for each eye in the video frame to the eye templates to locate the eyes, wherein the feature based matching comprises the process actions of;

for each image patch,

computing a grayscale image corresponding to the image patch;

creating horizontal and vertical edge maps of the image patch;

summing columns of pixels for the grayscale image and the vertical edge map to project the image patch to the horizontal axis to create two one dimensional (1D) signals;

summing rows of pixels for the horizontal edge map to project the image patch to the vertical axis to produce one 1D signal; and

computing the similarity between the eye template and the image patch as a weighted sum of the correlations between corresponding 1D signals, wherein the similarity is determined by a signal correlation function S(A,B) of

$S<br><br>\left(A,B\right)=\underset{i=1}{\overset{L}{\sum <br><br>}}<br><br>\left(a_i<br><br>b_i\right)/\sqrt{\underset{i=1}{\overset{L}{\sum <br><br>}}<br><br>a_i^2<br><br>\underset{i=1}{\overset{L}{\sum <br><br>}}<br><br>b_i^2}$

where L is the length of the signal and signals A and B are two arrays, A=a₁, a₂, . . . , a_L, B=b₁, b₂, . . . , b_L where a_i and b_i are elements in the two arrays.

2. The process of claim 1 wherein the three 1D signals for the left eye template T_L are denoted as G_i^TL,i=1, . . . , X_L for the grayscale image, H_i^TL, i=1, . . . , Y_L for the horizontal edge map, and V_i^TL, i=1, . . . , X_L for the vertical edge map, and where X_L and Y_L are the width and height of the template, for a candidate image patch I, three corresponding signals are denoted as G_i^I, H_i^I and V_i^I and the correlation S(T_L,I) between the two image patches is computed as

S(T_L,I)=w_G·

S(G^TL,G^I)+w_H·

S(H^TL,H^I)+w_V·

S(V^TL,V^I)

where w_G, w_H and w_v are predefined weights.

3. The process of claim 1 wherein the three 1D signals for the right eye template T_R are denoted as G_i^TL,i=1, . . . , X_R for the grayscale image, H_i^TL,i=1, . . . , Y_R for the horizontal edge map, and V_i^TL,i=1, . . . , X_R for the vertical edge map, and where X_R and Y_R are the width and height of the template, for a candidate image patch I, three corresponding signals are denoted as G_i^I, H_i^I and V_i^I and the correlation S(T_R,I) between the two image patches is computed as

S(T_R,I)=w_G·

S(G^TR,G^I)+w_H·

S(H^TR,H^I)+w_V·

S(V^TR,V^I)

where w_G, w_H and w_v are predefined weights.

4. The process of claim 1 wherein defining an eye template comprises the process actions of:

manually indicating the pupil positions on the first frame of the video sequence in which the person is depicted with wide open eyes; and

extracting two image patches at the pupil positions as templates, one for each eye.

5. The process of claim 1 wherein said face detector only scans in the neighborhood of the face location in a previous frame to find the face box.

6. The process of claim 1 wherein blinking is detected when the correlation values drop significantly.

7. The process of claim 3 wherein eye closing is detected when the correlation values for a given frame drop below a given threshold.

8. The process of claim 7 wherein the threshold is set to be 0.6.

9. A computer-readable storage medium having computer-executable instructions stored thereon for performing the process recited in claim 1.

10. A system for detecting eye blinks, comprising:

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

define eye templates for a person depicted in a video frame;

input a sequence of video frames at least some of which containing image frames of the person;

use a face detector to find a face;

if a face is found, search the face for eyes using said eye templates and feature based matching, wherein image patches for each frame in the video sequence are extracted and compared to the eye templates wherein the feature based matching comprises the sub-modules to;

for each extracted image patch,

compute a grayscale image corresponding to the image patch;

create horizontal and vertical edge maps of the image patch;

sum columns of pixels of the grayscale image and the vertical edge map to project the image patch to the horizontal axis to create two one dimensional (1D) signals;

sum rows of pixels for the horizontal edge map to project the image patch to the vertical axis to produce one 1D signal; and

compute the similarity between the eye template and the image patch as a weighted sum of the correlations between corresponding 1D signals, wherein the similarity is determined by a signal correlation function S(A,B) of

$S<br><br>\left(A,B\right)=\underset{i=1}{\overset{L}{\sum <br><br>}}<br><br>\left(a_i<br><br>b_i\right)/\sqrt{\underset{i=1}{\overset{L}{\sum <br><br>}}<br><br>a_i^2<br><br>\underset{i=1}{\overset{L}{\sum <br><br>}}<br><br>b_i^2}$

where L is the length of the signal and signals A and B are two arrays, A=a₁, a₂, . . . , a_L, B=b₁, b₂, . . . , b_L where a_i and b_i are elements in the two arrays.

11. The system of claim 10 wherein the module to define eye templates comprises sub-modules to:

manually indicate the pupil positions on the first frame of the video sequence in which the person is depicted with wide open eyes; and

extract two image patches at the pupil positions as templates, one for each eye.

12. The system of claim 10 wherein blinking is detected when the correlation values drop significantly.