Method and apparatus for determining eye contact

• US 6,393,136 B1
• Filed: 01/04/1999
• Issued: 05/21/2002
• Est. Priority Date: 01/04/1999
• Status: Expired due to Term

• Current Assignee: Tobii Technology AB
• Original Assignee: International Business Machines Corporation
• Inventors: Amir, Arnon, Zhai, Shumin, Flickner, Myron Dale, Morimoto, Carlos Hitoshi, Koons, David Bruce
• Primary Examiner(s): Boudreau, Leo
• Assistant Examiner(s): CHOOBIN, BARRY
• Application Number:
  US09/225,476
• Time in Patent Office: 1,233 Days
• Field of Search: 382/103, 382/166, 382/173, 382/118, 348/15, 348/169, 351/226, 351/210
• US Class Current: 382/103
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

A method and apparatus determine when a subject is looking at a specific target area by estimating a divergence angle between (1) the direction in which the subject is looking and (2) the direction from the subject directly to the target area. This technique accesses whether the subject is looking at a particular area. The invention may further condition this determination according to the subject'"'"'s distance from the target area, because there is less tolerance for divergent angles when the subject is farther away. In one embodiment, the divergence angle is estimated using the position of a glint of light in the subject'"'"'s pupil. The glint is created by a light source located in the target area. If the glint is sufficiently central to the pupil, with the camera and light source being near the target area, the subject is looking at the target area. At long distances, when the glint is not sufficiently discernable from the pupil, another technique may be employed to estimate divergence angle. Namely, the plane of the subject'"'"'s face is computed, and analyzed with respect to a vector between the subject'"'"'s face and the target area. If the plane is substantially normal to the vector, the subject is looking at the target area.

Claims

1. A method for determining whether a subject is looking at a target area, the subject having a face including an eye with a pupil, the method comprising the operations of:

if distance between the subject and the target area is less than a predetermined range, performing operations comprising;

operating a light source to create a glint upon the subject'"'"'s eye;

using a light sensor to detect the glint and measure glint position;

receiving a machine-readable representation of the pupil;

analyzing relative position of the glint with respect to the pupil to determine whether the relative position satisfies predetermined criteria; and

only if the relative position satisfies the predetermined criteria, providing an eye contact output declaring that the subject is looking at the target area;

otherwise, if distance between the subject and the target area is less than the predetermined range, performing operations comprising;

receiving a machine-readable input representing three-dimensional positions of multiple points on the face;

using the points to compute a face plane describing an average orientation of the face;

determining alignment between the face plane and the target area; and

if the alignment between the face plane and the target area meets predetermined criteria, providing an eye contact output declaring that the subject is looking at the target area.

2. The method of claim 1, further comprising the operations of estimating the distance between the target area and the subject.

3. The method of claim 1, the operations further comprising deeming the distance between the target area and the subject to be less than a predetermined threshold whenever the glint and pupil are sufficiently distinguishable.

4. A method for conditionally processing voice input, comprising:

operating a light source to create a glint upon the subject'"'"'s eye;

using a light sensor to detect the glint and measure glint position;

receiving a machine-readable representation of the pupil;

analyzing relative position of the glint with respect to the pupil to determine whether the relative position satisfies predetermined criteria;

only if the relative position satisfies the predetermined criteria, providing an eye contact output declaring that the subject is looking at the target area;

responsive to issuance of voice input, consulting the eye contact output to determine whether the subject is looking in the target area; and

ignoring the voice input unless the subject is looking in the target area when issuing the voice input.

5. A method for conditionally processing voice input comprising:

receiving a machine-readable input representing three-dimensional position of multiple points on the face;

using the points to compute a face plane describing an average orientation of the face;

determining alignment between the face plane and the target area;

if the alignment between the face plane and the target area meets predetermined criteria, providing an eye contact output declaring that the subject is looking at the target area;

responsive to issuance of voice input, consulting the eye contact output to determine whether the subject is looking in the target area; and

ignoring the voice input unless the subject is looking in the target area when issuing the voice input.

6. A signal-bearing medium tangibly embodying a program of machine-readable instructions executable by a digital data processing apparatus to perform operations for determining whether a subject is looking at a target area, the subject having a face including an eye with a pupil, the operations comprising:

if distance between the subject and the target area is less than a predetermined range, performing operations comprising;

operating a light source to create a glint upon the subject'"'"'s eye;

using a light sensor to detect the glint and measure glint position;

receiving a machine-readable representation of the pupil;

analyzing relative position of the glint with respect to the pupil to determine whether the relative position satisfies predetermined criteria; and

only if the relative position satisfies the predetermined criteria, providing an eye contact output declaring that the subject is looking at the target area;

otherwise, if distance between the subject and the target area is less than the predetermined range, performing operations comprising;

receiving a machine-readable input representing three-dimensional positions of multiple points on the face;

using the points to compute a face plane describing an average orientation of the face;

determining alignment between the face plane and the target area; and

if the alignment between the face plane and the target area meets predetermined criteria, providing an eye contact output declaring that the subject is looking at the target area.

7. The medium of claim 6, the operations further comprising estimating the distance between the target area and the subject.

8. The medium of claim 6, the operations further comprising deeming the distance between the target area and the subject to be less than a predetermined threshold whenever the glint and pupil are sufficiently distinguishable.

9. A signal-bearing medium tangibly embodying a program of machine-readable instructions executable by a digital data processing apparatus to perform operations for conditionally processing voice input according to whether a subject having a face is looking at a target area, the operations comprising:

operating a light source to create a glint upon the subject'"'"'s eye;

using a light sensor to detect the glint and measure glint position;

receiving a machine-readable representation of the pupil;

analyzing relative position of the glint with respect to the pupil to determine whether the relative position satisfies predetermined criteria;

only if the relative position satisfies the predetermined criteria, providing an eye contact output declaring that the subject is looking at the target area;

responsive to issuance of voice input, consulting the eye contact output to determine whether the subject is looking in the target area; and

ignoring the voice input unless the subject is looking in the target area when issuing the voice input.

10. A signal-bearing medium tangibly embodying a program of machine-readable instructions executable by a digital data processing apparatus to perform operations for conditionally processing voice input according to whether a subject having a face is looking at a target area, the operations comprising:

receiving a machine-readable input representing three-dimensional position of multiple points on the face;

using the points to compute a face plane describing an average orientation of the face;

determining alignment between the face plane and the target area;

if the alignment between the face plane and the target area meets predetermined criteria, providing an eye contact output declaring that the subject is looking at the target area;

responsive to issuance of voice input, consulting the eye contact output to determine whether the subject is looking in the target area; and

ignoring the voice input unless the subject is looking in the target area when issuing the voice input.

11. A logic circuit configured to perform operations for determining whether a subject is looking at a target area, the subject having a face including an eye with a pupil, the operations comprising:

if distance between the subject and the target area is less than a predetermined range, performing operations comprising;

operating a light source to create a glint upon the subject'"'"'s eye;

using a light sensor to detect the glint and measure glint position;

receiving a machine-readable representation of the pupil;

analyzing relative position of the glint with respect to the pupil to determine whether the relative position satisfies predetermined criteria; and

only if the relative position satisfies the predetermined criteria, providing an eye contact output declaring that the subject is looking at the target area;

otherwise, if distance between the subject and the target area is less than the predetermined range, performing operations comprising;

receiving a machine-readable input representing three-dimensional position of multiple points on the face;

using the points to compute a face plane describing an average orientation of the face;

determining alignment between the face plane and the target area; and

if the alignment between the face plane and the target area meets predetermined criteria, providing an eye contact output declaring that the subject is looking at the target area.

12. A logic circuit configured to perform operations for conditionally processing voice input according to whether a subject having a face is looking at a target area, the operations comprising:

operating a light source to create a glint upon the subject'"'"'s eye;

using a light sensor to detect the glint and measure glint position;

receiving a machine-readable representation of the pupil;

analyzing relative position of the glint with respect to the pupil to determine whether the relative position satisfies predetermined criteria;

only if the relative position satisfies the predetermined criteria, providing an eye contact output declaring that the subject is looking at the target area;

responsive to issuance of voice input, consulting the eye contact output to determine whether the subject is looking in the target area; and

ignoring the voice input unless the subject is looking in the target area when issuing the voice input.

13. A logic circuit configured to perform operations for conditionally processing voice input according to whether a subject having a face is looking at a target area, the operations comprising:

receiving a machine-readable input representing three-dimensional position of multiple points on the face;

using the points to compute a face plane describing an average orientation of the face;

determining alignment between the face plane and the target area;

if the alignment between the face plane and the target area meets predetermined criteria, providing an eye contact output declaring that the subject is looking at the target area;

responsive to issuance of voice input, consulting the eye contact output to determine whether the subject is looking in the target area; and

ignoring the voice input unless the subject is looking in the target area when issuing the voice input.

14. A computer-driven eye contact sensing system, comprising:

a light source;

a light sensor; and

an eye contact estimating apparatus, coupled to the light sensor and configured to perform operations for determining whether a subject is looking at a target area, the subject having a face including an eye with a pupil, the operations comprising;

if distance between the subject and the target area is less than a predetermined range, performing operations comprising;

operating a light source to create a glint upon the subject'"'"'s eye;

using a light sensor to detect the glint and measure glint position;

receiving a machine-readable representation of the pupil;

analyzing relative position of the glint with respect to the pupil to determine whether the relative position satisfies predetermined criteria; and

only if the relative position satisfies the predetermined criteria, providing an eye contact output declaring that the subject is looking at the target area;

otherwise, if distance between the subject and the target area is less than the predetermined range, performing operations comprising;

receiving a machine-readable input representing three-dimensional positions of multiple points on the face;

using the points to compute a face plane describing an average orientation of the face;

determining alignment between the face plane and the target area; and

if the alignment between the face plane and the target area meets, predetermined criteria, providing an eye contact output declaring that the subject is looking at the target area.

15. A computer-driven eye contact sensing system, comprising:

a light source;

a light sensor; and

a data processing apparatus, coupled to the light sensor and programmed to conditionally process voice input according to whether a subject having a face is looking at a target area by performing operations comprising;

operating a light source to create a glint upon the subject'"'"'s eye;

using a light sensor to detect the glint and measure glint position;

receiving a machine-readable representation of the pupil;

analyzing relative position of the glint with respect to the pupil to determine whether the relative position satisfies predetermined criteria;

only if the relative position satisfies the predetermined criteria, providing an eye contact output declaring that the subject is looking at the target area;

responsive to issuance of voice input, consulting the eye contact output to determine whether the subject is looking in the target area; and

ignoring the voice input unless the subject is looking in the target area when issuing the voice input.

16. The system of claim 15, the data processing apparatus comprising a digital data processing computer.

17. The system of claim 15, the data processing apparatus comprising a logic circuit.

18. The system of claim 15, the light sensor comprising a camera.

19. A computer-driven eye contact sensing system, comprising:

a light source;

a light sensor; and

a data processing apparatus, coupled to the light sensor and programmed to conditionally process voice input according to whether a subject having a face is looking at a target area by performing operations comprising;

receiving a machine-readable input representing three-dimensional position of multiple points on the face;

using the points to compute a face plane describing an average orientation of the face;

determining alignment between the face plane and the target area; and

if the alignment between the face plane and the target area meets predetermined criteria, providing an eye contact output declaring that the subject is looking at the target area;

responsive to issuance of voice input, consulting the eye contact output to determine whether the subject is looking in the target area; and

ignoring the voice input unless the subject is looking in the target area when issuing the voice input.

20. The system of claim 19, the data processing apparatus comprising a digital data processing computer.

21. The system of claim 19, the data processing apparatus comprising a logic circuit.

22. The system of claim 19, the light sensor comprising a camera.