Automatic direct gaze detection based on pupil symmetry

• US 7,819,525 B2
• Filed: 02/15/2009
• Issued: 10/26/2010
• Est. Priority Date: 02/15/2009
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Connell, Jonathan H. II
• Primary Examiner(s): MAI, HUY KIM
• Application Number:
  US12/371,607
• Publication Number:
  US 20100208207A1
• Time in Patent Office: 618 Days
• Field of Search: 351/204, 351/205, 351/209, 351/210, 351/246, 382/103, 382/117
• US Class Current: 351/204
• CPC Class Codes: A61B 3/112 for measuring diameter of pupils
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

The present invention provides a system and methods for direct gaze detection based on pupil symmetry. The system and methods work by first finding the user'"'"'s eyes in a video camera image and measuring the symmetry of the pupil region. If this pupil region is close to round, the invention determines that the user is looking nearly directly or close to directly at the camera, or has direct gaze at the camera. This technique handles both eye motions and head motions (e.g., the gaze may shift with or without the head being turned).

Claims

1. A method for direct gaze detection based on pupil symmetry comprising:

finding one or both of a user'"'"'s pupil regions of eyes of the user in a video camera image from a video camera;

measuring a symmetry of a pupil region;

determining if the pupil region is close to round; and

if the pupil region is close to round, determining that the user is looking at the video camera.

2. The method as defined in claim 1 further comprising utilizing the video camera to find the user'"'"'s pupil regions in the video camera image positioned away from the user'"'"'s eyes.

3. The method as defined in claim 2 further comprising positioning the video camera several feet from the user'"'"'s eyes.

4. The method as defined in claim 3 further comprising capturing video camera images of the user'"'"'s pupil regions and processing the images.

5. The method as defined in claim 1 further comprising finding the user'"'"'s pupil regions in the image using a red eye effect.

6. The method as defined in claim 5 wherein the camera has an optical axis, the method further comprising shining infrared light from a first set of infrared light-emitting diodes (LEDs) near the optical axis of the camera and acquiring a first image of the user'"'"'s pupil regions.

7. The method as defined in claim 6 further comprising acquiring a second image by shining a second infrared light from a second set of LEDs, the second set of LEDs being further displaced from the optical axis of the camera than the first set of infrared LEDs.

8. The method as defined in claim 7 further comprising computing a pixel-by-pixel difference of the first acquired image and the second acquired image for identifying possible retroreflectors.

9. The method as defined in claim 8 further comprising identifying a pair of detected pupil regions that are approximately the same shape and size and displaced horizontally in a manner indicative of a pair of eyes.

10. The method as defined in claim 9 further comprising examining at least one pupil region for symmetry by thresholding bright spots and computing direction and axis measurements of an equivalent area ellipse based on moments of a binary region.

11. The method as defined in claim 1 further comprising finding the pupil regions by template matching to a model eye.

12. The method as defined in claim 1 further comprising finding the pupil regions by looking for time differences in a video stream that are induced by eye blink motions.

13. A system for direct gaze detection based upon pupil symmetry, the system comprising:

a video camera for finding eyes and pupil regions of a user, the video camera having an optical axis;

a first set of infrared light-emitting diodes (LEDs) for shining infrared light into the user'"'"'s eyes so that the video camera may acquire a first image of the user'"'"'s pupil regions;

a second set of infrared LEDs for shining infrared light into the user'"'"'s eyes so that the video camera may acquire a second image of the user'"'"'s pupil regions; and

a processing unit for computing a pixel-by-pixel difference of the first acquired image and the second acquired image for identifying possible retroreflectors.

14. The system as defined in claim 13 further comprising a memory unit for storing a model eye template wherein the processing unit further finds pupil regions by comparing the acquired images with the model eye template.

15. A computer program product embodied in a computer readable medium for operating in a system comprising a processing unit, a memory, a bus, and input/output (I/O) interfaces, for implementing a method for direct gaze detection of a user into a video camera, the method comprising:

finding one or both of a user'"'"'s pupil regions of eyes of the user in a video camera image from a video camera;

measuring a symmetry of at least pupil region;

determining if the at least one pupil region is close to round; and

if the at least one pupil region is close to round, determining that the user is looking at the video camera.

16. The computer program product as defined in claim 15 wherein the camera has an optical axis and further wherein the method further comprises shining infrared light from a first set of infrared light-emitting diodes (LEDs) near the optical axis of the camera and acquiring a first image of the user'"'"'s pupil regions, acquiring a second image by shining a second infrared light from a second set of LEDs, the second set of LEDs being further displaced from the optical axis of the camera than the first set of infrared LEDs and computing a pixel-by-pixel difference of the first acquired image and the second acquired image for identifying possible retroreflectors.

17. The computer program product as defined in claim 16 wherein the method further comprises identifying a pair of detected pupil regions that are approximately the same shape and size and displaced horizontally in a manner indicative of a pair of eyes.

18. The computer program product as defined in claim 17 wherein the method further comprises examining at least one pupil region for symmetry by thresholding bright spots and computing direction and axis measurements of an equivalent area ellipse based on moments of a binary region.

19. A method for deploying computing infrastructure comprising integrating computer-readable code into a computing system, wherein the code in combination with the computing system is capable of performing a process for detecting a direct gaze of a user into a video camera, the process comprising:

finding one or both of a user'"'"'s pupil regions of eyes of the user in a video camera image from a video camera;

measuring a symmetry of at least pupil region;

determining if the at least one pupil region is close to round; and

if the at least one pupil region is close to round, determining that the user is looking at the video camera.

20. The method for deploying computing infrastructure as defined in claim 19 wherein the camera has an optical axis and further wherein the process further comprises shining infrared light from a first set of infrared light-emitting diodes (LEDs) near the optical axis of the camera and acquiring a first image of the user'"'"'s pupil regions, acquiring a second image by shining a second infrared light from a second set of LEDs, the second set of LEDs being further displaced from the optical axis of the camera than the first set of infrared LEDs and computing a pixel-by-pixel difference of the first acquired image and the second acquired image for identifying possible retroreflectors.