Multi-camera head pose tracking

• US 8,339,459 B2
• Filed: 09/16/2009
• Issued: 12/25/2012
• Est. Priority Date: 09/16/2009
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, Sankaranarayanan, Aswin, Zhang, Qing, Liu, Zicheng, Cai, Qin
• Primary Examiner(s): SALAD, ABDULLAHI ELMI
• Application Number:
  US12/561,154
• Publication Number:
  US 20110063403A1
• Time in Patent Office: 1,196 Days
• Field of Search: 348/169, 348/14.1, 382/103
• US Class Current: 348/169
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display

Abstract

Techniques and technologies for tracking a face with a plurality of cameras wherein a geometry between the cameras is initially unknown. One disclosed method includes detecting a head with two of the cameras and registering a head model with the image of the head (as detected by one of the cameras). The method also includes back projecting the other detected face image to the head model and determining a head pose from the back-projected head image. Furthermore, the determined geometry is used to track the face with at least one of the cameras.

Claims

1. A method comprising:

detecting a head with a first camera of a plurality of cameras, the head as detected with the first camera being captured in a first detected head image associated with the first camera;

registering a head model with the first detected head image;

detecting the head with a second camera of the plurality of cameras, the head as detected with the second camera being captured in a second detected head image associated with the second camera, a geometry between the cameras being uncalibrated;

back-projecting the second detected head image to the head model;

determining the geometry between the cameras; and

determining a head pose of the head from the second detected head image as back-projected to the head model.

2. The method of claim 1 wherein the registering the head model includes rescaling or re-orienting the head model.

3. The method of claim 1 further comprising determining a first key frame of the first detected head image, the first key frame being associated with the first camera, the first key frame being a constraint in the determining the head pose.

4. The method of claim 3 further comprising associating the first key frame associated with the first camera with a corresponding frame of the second detected head image associated with the second camera to establish a second key frame from the corresponding frame, the second key frame being associated with the second camera.

5. The method of claim 4 further comprising:

detecting a loss of tracking of the head by one of the cameras;

detecting the head with a third camera of the plurality of cameras, the head as detected with the third camera being captured in a third detected head image associated with the third camera;

comparing the third detected head image with the registered head model; and

responsive to the comparing of the third detected head image and the registered head model, establishing a third key frame associated with the third camera.

6. The method of claim 5 wherein the loss of tracking occurs as the head leaves a field of view of the one of the cameras.

7. The method of claim 1 further comprising using knowledge of a smoothness of movement of the head as detected by the first camera and of the geometry between the cameras to predict a movement of the head as detected by the second camera.

8. The method of claim 1 further comprising detecting a movement of the first camera and invalidating a determined geometry related to the first and second cameras.

9. The method of claim 8 further comprising re-determining the head pose by repeating the:

detecting the head with the first detected head image;

registering the head model; and

back-projecting a third detected head image to the head model.

10. The method of claim 9 further comprising using knowledge of a geometry related to the second camera and a third camera and the geometry related to the first and the second cameras to re-calibrate the geometry between the moved first camera and the second camera.

11. The method of claim 1 further comprising:

detecting a movement of the head with the first camera;

predicting a movement of the head as seen from a position of the second camera using the movement of the head detected by the first camera; and

tracking the head with the second camera using the predicted movement of the head.

12. The method of claim 11 wherein the predicted movement of the head will bring the head into a field of view of the second camera.

13. A computing device comprising;

a display device;

a processor in communication with the display device and a first camera and a second camera of a plurality of cameras; and

a memory in communication with the processor and for storing images captured by the cameras and a head model, the memory storing processor readable instructions which when executed cause the processor to;

detect a head in a first image captured by the first camera,

register the head model with the detected head in the first image captured by the first camera,

detect the head in a second image captured by the second camera, a geometry between the first and second cameras being at least initially uncalibrated,

back project the detected head in the second image to the head model;

determine the geometry between the cameras;

determine a head pose of the head from the detected head in the second image as back-projected to the head model; and

display the head on the display device while tracking the head pose.

14. The computing device of claim 13 wherein the cameras are installed in a teleconferencing facility.

15. The computing device of claim 13 wherein the computing device is a portion of a telepresence based system.

16. The computing device of claim 13 wherein one of the cameras is a handheld camera.

17. The computing device of claim 13 wherein the first camera has a first field of view oriented along a first axis and the second camera has a second field of view oriented along a second axis, the first and second axis of the first and second cameras respectively intersecting at about an expected location of the head.

18. The computing device of claim 17 wherein the processor executable instructions further cause the processor to:

detect an approximate frontal view of the head in the first image captured by the first camera;

detect an approximate side view of the head in the second image captured by the second camera; and

determine an angle between the first and second axis of the first and second cameras.

19. The computing device of claim 18 wherein the head model is a generic head model.

20. A memory including processor executable instructions for a method of tracking a head and which when executed cause the processor to:

detect a head with a first camera of a plurality of cameras, the head as detected with the first camera being captured in a first detected head image associated with the first camera;

register a head model with the first detected head image by rescaling and re-orienting the head model to fit the first detected head image;

detect the head with a second camera of the plurality of cameras, the head as detected with the second camera being captured in a second detected head image associated with the second camera, a distance and an orientation between the cameras being at least initially uncalibrated;

back-project the second detected head image to the head model;

determine the distance and the orientation between the cameras;

determine a head pose of the head from the second detected head image as back-projected to the head model;

determine a first key frame of the first detected head image, the first key frame being associated with the first camera; and

associate the first key frame associated with the first camera with a corresponding frame of the second detected head image associated with the second camera to establish a second key frame from the corresponding frame, the second key frame being associated with the second camera.