Real-time head pose tracking with online face template reconstruction

• US 9,672,412 B2
• Filed: 06/24/2014
• Issued: 06/06/2017
• Est. Priority Date: 06/24/2014
• Status: Active Grant

• Current Assignee: Chinese University of Hong Kong
• Original Assignee: Chinese University of Hong Kong
• Inventors: Ngan, King Ngi, Li, Songnan
• Primary Examiner(s): Lefkowitz, Sumati
• Assistant Examiner(s): Wang, Carol
• Application Number:
  US14/313,639
• Publication Number:
  US 20150371080A1
• Time in Patent Office: 1,078 Days
• Field of Search: None
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images

Abstract

Provided are methods and apparatus for tracking a head pose with online face template reconstruction. The method comprises the steps of retrieving a plurality of frames of images of the user; comparing each of the retrieved frames with a predetermined face template to determine one or more head poses that are monitored successfully and obtain head pose information of the determined one or more head poses; and reconstructing, during the step of comparing, the face template from the obtained head pose information; wherein the reconstructed face template is compared with subsequently retrieved images such that the head poses of the user are tracked in time.

Claims

1. A computer-implemented method for tracking a head pose of a user, comprising:

retrieving a plurality of frames of images of the user;

comparing each of the retrieved frames with a predetermined face template to determine one or more head poses that are monitored successfully and obtain head pose information of the determined one or more head poses;

selecting a plurality of frames for reconstructing the face template by;

checking, from the determined one or more head poses, a head pose is located in a preset effective fitting zone;

selecting the frame corresponding to the checked head pose as a frame to be selected; and

updating the effective fitting zone according to the checking results;

wherein the steps of checking, selecting and updating are repeated until no head pose is located in the updated effective fitting zone, such that all frames are selected; and

reconstructing, during the step of comparing, the face template from the obtained head pose information;

wherein the reconstructed face template is compared with subsequently retrieved images such that the head poses of the user are tracked in time.

2. The method of claim 1, wherein, the method further comprises, before the step of reconstructing:

selecting a frame in which a head pose is firstly monitored successfully, so as to reconstruct the face template from the head pose information of the selected frame.

3. The method of claim 2, wherein the head pose information comprises rotation information and translation information, and the step of reconstructing comprises:

assigning a weight for each point of the face template, wherein a larger weight is assigned to a facial feature point of the face template; and

determining a face changing magnitude of the face template by using the rotation information and the translation information of the selected frame and the assigned weights to reconstruct the face template.

4. The method of claim 1, wherein the head pose information comprises rotation information and translation information, the step of reconstructing comprises:

assigning a weight for each point of the face template, wherein a larger weight is assigned to a facial feature point of the face template;

determining, for each of the selected frames, a face changing magnitude of the face template by using the rotation information, the translation information and the assigned weights, respectively; and

accumulating the determined magnitudes from the selected plurality of frames to reconstruct the face template.

5. The method of claim 1, wherein the step of comparing further comprises, for each frame:

searching, for each point of the face template, a matching point in the retrieved image to form a matching point pair;

calculating matching errors for the searched matching point pairs; and

estimating head pose of the next frame by minimizing sum of the calculated matching errors, such that the head pose the current frame is monitored.

6. The method of claim 5, wherein, for each frame,

if the sum of the matching errors is smaller than a first threshold, it is determined that the head pose of the current frame is monitored successfully, such that the current frame is selected for reconstructing the face template;

and

if the sum of the matching errors is larger than a second threshold, it is determined that the head pose of the current frame is not monitored successfully, such that the head pose of the current frame is initialized,

wherein the second threshold is larger than the first threshold.

7. The method of claim 1, further comprising:

counting the number of the head poses that are not monitored successfully;

wherein, when the number successively counted exceeds a threshold, the step of reconstructing is evoked.

8. The method of claim 1, wherein the comparing is implemented by iterative closest point (ICP) algorithm.

9. The method of claim 1, wherein a depth image is captured by a RGB-Depth camera.

10. The method of claim 9, further comprising:

detecting the user is located in an image capturing area for the RGB-Depth camera so as to evoke the step of retrieving.

11. An apparatus for tracking a head pose of a user, comprising:

a retriever configured to retrieve a plurality of frames of images of the user;

a comparator configured to compare each of the retrieved frames with a predetermined face template to determine one or more head poses that are monitored successfully and obtain head pose information of the determined one or more head poses;

a selector configured to select a plurality of frames for reconstructing the face template by;

checking, from the determined one or more head poses, a head pose is located in a preset effective fitting zone;

selecting the frame corresponding to the checked head pose as a frame to be selected; and

updating the effective fitting zone according to the checking results;

wherein the steps of checking, selecting and updating are repeated until no head pose is located in the updated effective fitting zone, such that all frames are selected; and

a reconstructor configured to reconstruct the face template from the obtained head pose information;

wherein the reconstructed face template is compared by the comparator with images retrieved subsequently by the retriever such that the head poses of the user are tracked in time.

12. The apparatus of claim 11, further comprising:

a selector configured to select a frame in which a head pose is firstly monitored successfully, such that the face template is reconstructed from the head pose information of the selected frame.

13. The apparatus of claim 11, wherein the head pose information comprises rotation information and translation information, and

the reconstructor is further configured to:

assign a weight for each point of the face template, wherein a larger weight is assigned to a facial feature point of the face template;

determine, for each of the selected frames, a face changing magnitude of the face template by using the rotation information, the translation information and the assigned weights, respectively; and

accumulate the determined magnitudes from the selected plurality of frames to reconstruct the face template.

14. The apparatus of claim 11, wherein the comparator is further configured to, for each frame:

search, for each point of the face template, a matching point in the retrieved image to form a matching point pair;

calculate matching errors for the searched matching point pairs; and

estimate head pose of the next frame by minimizing sum of the calculated matching errors, such that the head pose is monitored.

15. The apparatus of claim 14, wherein the comparator is further configured to:

determine the head pose of the current frame as a head pose which is monitored successfully, if the sum of the matching errors is smaller than a first threshold;

or

determine the head pose of the current frame as a head pose which is not monitored successfully, if the sum of the matching errors is larger than a second threshold,

wherein the second threshold is larger than the first threshold.

16. The apparatus of claim 11, further comprising:

a counter configured to count the number of the head pose that is not monitored successfully; and

an evoker configured to evoke the reconstructor to reconstruct the face template,

wherein, when the number successively counted by the counter exceeds a threshold, the reconstructor is evoked by the evoker.

17. The apparatus of claim 11, wherein the comparator is implemented by performing iterative closest point (ICP) algorithm.

18. The apparatus of claim 11, wherein the retriever is a RGB-Depth camera and the apparatus further comprises:

a detector configured to detect a user is located in an image capturing area for the RGB-Depth camera; and

an evoker configured to evoke the retriever to retrieve an image when the user is detected by detector to be located in the image capturing area.

19. A computer-implemented method for tracking a head pose of a user, comprising:

detecting that a user is located in an image capturing area for a RGB-Depth camera;

retrieving a plurality of frames of images of the user in response to the detecting;

comparing each of the retrieved frames with a predetermined face template to determine one or more head poses that are monitored successfully and obtain rotation information and translation information of the determined one or more head poses, such that a plurality of frames for reconstructing the face template is selected by;

checking, from the determined one or more head poses, a head pose is located in a preset effective fitting zone;

selecting the frame corresponding to the checked head pose as a frame to be selected; and

updating the effective fitting zone according to the checking results;

wherein the steps of checking, selecting and updating are repeated until no head pose is located in the updated effective fitting zone, such that all frames are selected;

reconstructing, during the step of comparing, the face template from the obtained rotation information and translation information of the selected plurality of frames;

wherein the reconstructed face template is compared with subsequently retrieved images such that the head poses of the user are tracked in time.