Object orientation estimation

• US 9,727,776 B2
• Filed: 05/27/2014
• Issued: 08/08/2017
• Est. Priority Date: 05/27/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hulten, Geoffrey J., Dedhia, Bhaven P., Eshet, Yaron
• Primary Examiner(s): Patel, Jayesh A
• Application Number:
  US14/288,287
• Publication Number:
  US 20150348269A1
• Time in Patent Office: 1,169 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

The description relates to estimating object orientation. One example includes determining a first estimate of object orientation using a first technique and image data. In this example, a second estimate of the object orientation can be determined using a second technique and the image data. The first estimate can be corrected with the second estimate to generate a corrected object orientation estimate which can be output.

Claims

1. A method performed by a computing device, the method comprising:

differentiating an object from other portions of an image, the image comprising two-dimensional image data and depth data;

determining landmarks of the differentiated object;

determining a first estimate of object orientation of the differentiated object using a first technique that uses the depth data in determining a geometric model of the differentiated object;

determining a second estimate of the object orientation of the differentiated object using a second technique with the landmarks of the differentiated object;

correcting the first estimate using the second estimate to generate a corrected object orientation estimate; and

outputting the corrected object orientation estimate.

2. The method of claim 1, wherein the geometric model includes a plane of the differentiated object.

3. The method of claim 1, wherein the first technique comprises determining the landmarks and an up vector of the differentiated object.

4. The method of claim 1, wherein the first estimate comprises estimates of angles of rotation relative to multiple axes of rotation of the differentiated object.

5. The method of claim 4, wherein the multiple axes of rotation of the differentiated object correspond to pitch, roll, and yaw.

6. The method of claim 4, wherein the second estimate of the object orientation includes another estimate of at least one of the angles of rotation.

7. The method of claim 6, wherein the multiple axes of rotation of the differentiated object correspond to pitch, roll, and yaw and the second estimate corresponds to yaw.

8. The method of claim 1, wherein the landmarks are determined from the two-dimensional image data.

9. The method of claim 1, wherein the depth data includes time-of-flight depth data.

10. The method of claim 1, further comprising assigning a weight to the second estimate according to a distance of the differentiated object from a camera that collects the depth data, wherein the weight is used to generate the corrected object orientation estimate.

11. The method of claim 1, wherein the correcting includes determining an extent to which the first estimate is consistent with the second estimate.

12. The method of claim 1, wherein the differentiated object is a body part of a user, a handheld object, a vehicle, or a computing device.

13. A system, comprising:

a processing device; and

a storage device storing computer-executable instructions which, when executed by the processing device, cause the processing device to;

obtain an image comprising depth data;

differentiate an object from other portions of the image;

determine landmarks of the differentiated object;

determine a first estimate of object orientation of the differentiated object using a first technique that uses the depth data to determine a geometric model of the differentiated object;

determine a second estimate of the object orientation of the differentiated object using a second technique that considers the landmarks of the differentiated object; and

correct the first estimate using the second estimate to generate a corrected object orientation estimate.

14. The system of claim 13, wherein the image further comprises two-dimensional image data and the landmarks are determined from the two-dimensional image data.

15. The system of claim 13, wherein an overall accuracy of the corrected object orientation estimate is greater than a first accuracy of the first estimate or a second accuracy of the second estimate.

16. The system of claim 13, wherein the second technique is an iterative technique.

17. The system of claim 16, wherein the computer-executable instructions further cause the processing device to adjust a number of iterations of the iterative technique according to available computational resources of the system.

18. A computer-executable memory device or storage device storing computer-executable instructions that, when executed by one or more processing devices, cause the one or more processing devices to perform acts comprising:

detecting an object in an image;

determining landmarks of the detected object;

obtaining a first estimate of object orientation of the detected object, the first estimate being provided by a first technique that determines a geometric model of the detected object;

obtaining a second estimate of the object orientation of the detected object, the second estimate being provided by a second technique that considers the landmarks of the detected object;

correcting the first estimate using the second estimate, based at least in part on an extent to which the first estimate is consistent with the second estimate, to generate a corrected object orientation estimate; and

outputting the corrected object orientation estimate.

19. The computer-executable memory device or storage device of claim 18, wherein the detected object is a head of a user.

20. The computer-executable memory device or storage device of claim 19, wherein the geometric model is a facial plane of the head.

21. The computer-executable memory device or storage device of claim 20, wherein the first estimate of the object orientation is a rotation angle estimate determined using the facial plane of the head.