Machine vision system and method for estimating and tracking facial pose

• US 7,756,294 B2
• Filed: 05/05/2006
• Issued: 07/13/2010
• Est. Priority Date: 12/31/2001
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Toyama, Kentaro
• Primary Examiner(s): Werner; Brian P
• Application Number:
  US11/382,012
• Publication Number:
  US 20060193494A1
• Time in Patent Office: 1,530 Days
• Field of Search: 382/103, 382/118, 345/158
• US Class Current: 382/103
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

System and method for estimating and tracking an orientation of a user'"'"'s face by combining head tracking and face detection techniques. The orientation of the face, or facial pose, can be expressed in terms of pitch, roll and yaw of the user'"'"'s head. Facial pose information can be used, for example, to ascertain in which direction the user is looking. In general, the facial pose estimation method obtains a position of the head and a position of the face and compares the two to obtain the facial pose. In particular, a camera is used to obtain an image containing a user'"'"'s head. Any movement of the user'"'"'s head is tracked and the head position is determined. A face then is detected on the head and the face position is determined. The head and face positions then are compared.

Claims

1. An attention detection system for determining whether a computer application should execute an action, comprising:

an image captured by a camera that contains the user'"'"'s head and face;

a display device for providing visual information to the user;

a yaw center line defined as a line in an x-z yaw plane that bisects the user'"'"'s head into two approximately equal parts;

a deviation angle that indicates an angle between a center of the camera and a center of the user'"'"'s face;

a threshold angle that is compared to the deviation angle to determine whether the user is facing the display device; and

a facial pose estimation system that processes the image and uses the threshold angle to determine whether a user is gazing at the display device and instructs the computer application whether to execute an action based on the determination.

2. The attention detection system of claim 1, further comprising a head tracker that tracks a head position of the user'"'"'s head in the image.

3. The attention detection system of claim 2, further comprising a face detection system that detects a face position of the user'"'"'s face in the image.

4. The attention detection system of claim 3, further comprising a position comparator that receives the head position and the face position and determines a facial pose of the user by comparing the head and face positions.

5. The attention detection system of claim 1, wherein the deviation angle is in an x-z yaw plane that indicates a yaw angle between the center of the camera and the center of the user'"'"'s face.

6. The attention detection system of claim 5, wherein the threshold angle is in the x-z yaw plane.

7. The attention detection system of claim 1, wherein the deviation angle is in a y-z pitch plane that indicates a pitch angle between the center of the camera and the center of the user'"'"'s face.

8. The attention detection system of claim 7, wherein the threshold angle is in the y-z pitch plane.

9. The attention detection system of claim 1, further comprising:

a center of the user'"'"'s face in the x-z yaw plane; and

a yaw deviation corresponding to a pixel distance between the yaw center line and the center of the user'"'"'s face in the x-z yaw plane.

10. The attention detection system of claim 9, further comprising a yaw threshold that is compared to the yaw deviation to determine whether the user is facing the display device and, if so, then instructs the computer application to execute the action.

11. A computer-implemented attention detection system for determining where a user'"'"'s attention is focused, comprising:

at least one monitor that provides the user with visual information;

a camera disposed on the monitor that captures images of the user including a head of the user;

a yaw center line defined as a line in an x-z yaw plane that bisects the user'"'"'s head into two approximately equal parts;

a deviation angle that indicates an angle between a center of the camera and the yaw center line;

a threshold angle that is compared to the deviation angle to determine the facial pose information; and

a facial pose estimation system that receives and uses the captured images and yaw center line to output the facial pose information containing an estimate of a facial pose of the user to determine where the user'"'"'s attention is focused.

12. The computer-implemented attention detection system as set forth in claim 11, further comprising a screen saver application in communication with the monitor that receives the facial pose information and uses the information to decide whether to start and terminate the screen saver application on the monitor.

13. The computer-implemented attention detection system as set forth in claim 11, further comprising a plurality monitors and a monitor application that uses the facial pose information to determine which of the plurality of monitors the user is observing.

14. The computer-implemented attention detection system as set forth in claim 11, further comprising an instant messaging application that receives the facial pose information and uses the information to decide whether to display a message to the user on the monitor.

15. The computer-implemented attention detection system as set forth in claim 11, wherein the deviation angle is in the x-z yaw plane and indicates a yaw angle between a center of the camera and the yaw center line.

16. The computer-implemented attention detection system as set forth in claim 15, wherein the threshold angle is in the x-z yaw plane.

17. A computer-implemented method for determining where a user'"'"'s attention is focused in relation to a computer display device, comprising:

capturing an image of the user'"'"'s head and face using a camera;

calculating a centroid of the user'"'"'s head and a centroid of the user'"'"'s face;

computing a deviation angle using the centroid of the user'"'"'s head and the centroid of the user'"'"'s face;

comparing the deviation angle and a threshold angle using the computer to determine where the user'"'"'s attention is focused in relative to a computer display device; and

instructing a computer application whether to execute an action based on where the user'"'"'s attention is focused.

18. The computer-implemented method as set forth in claim 17, wherein comparing further comprises:

calculating the deviation angle as a deviation between the centroid of the user'"'"'s face and the centroid of the user'"'"'s head.

19. The computer-implemented method as set forth in claim 18, further comprising:

determining that deviation angle is greater than the threshold angle and that the user'"'"'s attention is not focused on the computer display device; and

causing the computer application to perform background computation while the user'"'"'s attention is not focused on the computer display device.

20. The computer-implemented method as set forth in claim 18, further comprising:

determining that deviation angle is less than the threshold angle and that the user'"'"'s attention is focused on the computer display device; and

causing the computer application to display a message to the user on the computer display device.