METHODS AND SYSTEMS OF PERFORMING OBJECT POSE ESTIMATION

• US 20180137644A1
• Filed: 04/24/2017
• Published: 05/17/2018
• Est. Priority Date: 11/11/2016
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Rad, Mahdi, Oberweger, Markus, Lepetit, Vincent
• Granted Patent: US 10,235,771 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/242 by image rotation, e.g. by 90 degrees
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects

Abstract

Techniques are provided for estimating a three-dimensional pose of an object. An image including the object can be obtained, and a plurality of two-dimensional (2D) projections of a three-dimensional bounding (3D) box of the object in the image can be determined. The plurality of 2D projections of the 3D bounding box can be determined by applying a trained regressor to the image. The trained regressor is trained to predict two-dimensional projections of the 3D bounding box of the object in a plurality of poses, based on a plurality of training images. The three-dimensional pose of the object is estimated using the plurality of 2D projections of the 3D bounding box.

Claims

1. A method of estimating a three-dimensional pose of an object, the method comprising:

obtaining an image including the object;

determining a plurality of two-dimensional projections of a three-dimensional bounding box of the object, the plurality of two-dimensional projections of the three-dimensional bounding box being determined by applying a trained regressor to the image, wherein the trained regressor is trained to predict two-dimensional projections of the three-dimensional bounding box of the object in a plurality of poses; and

estimating the three-dimensional pose of the object using the plurality of two-dimensional projections of the three-dimensional bounding box.

2. The method of claim 1, wherein the regressor is a convolutional neural network.

3. The method of claim 1, further comprising determining an image patch in the image, wherein the image patch is centered on the object, and wherein the plurality of two-dimensional projections of the three-dimensional bounding box are determined by applying the trained regressor to the image patch.

4. The method of claim 3, wherein determining the image patch includes:

segmenting the image into an object area map of the image, the object area map including a grid of object area blocks;

determining a respective probability for each of the object area blocks in the grid, wherein a probability of an object area block indicates a probability that the object is located in the object area block; and

selecting an object area block with a highest determined probability from the grid of object area blocks.

5. The method of claim 4, wherein determining the image patch further includes:

segmenting the selected object area block into a plurality of object center blocks;

determining a respective probability for each of the object center blocks in the plurality of object center blocks, a probability of an object center block indicating a probability that a center of the object is located in the object center block;

selecting an object center block with a highest determined probability from the plurality of object center blocks;

determining a center of the object in the selected object center block; and

generating the image patch by cropping the image, wherein the cropped image is centered on the center of the object.

6. The method of claim 1, wherein estimating the three-dimensional pose of the three-dimensional object using the plurality of two-dimensional projections of the three-dimensional bounding box includes:

estimating a pose of a calibrated camera for the image, wherein the pose of the calibrated camera for the image is estimated using three-dimensional locations of the three-dimensional bounding box for the object in the image and the plurality of two-dimensional projections of the three-dimensional bounding box of the object in the image, wherein the plurality of two-dimensional projections include projections of the three-dimensional locations of the three-dimensional bounding box.

7. The method of claim 1, wherein the plurality of two-dimensional projections of the three-dimensional bounding box of the object include projections of corners of the three-dimensional bounding box.

8. The method of claim 1, wherein the image includes a red-green-blue (RGB) image.

9. The method of claim 1, wherein the image does not include depth information.

10. The method of claim 1, wherein the determined three-dimensional pose includes a three-dimensional orientation of the object and a three-dimensional translation of the object.

11. An apparatus for estimating a three-dimensional pose of an object, comprising:

a memory configured to store an image including the object; and

a processor configured to;

obtain the image including the object;

determine a plurality of two-dimensional projections of a three-dimensional bounding box of the object, the plurality of two-dimensional projections of the three-dimensional bounding box being determined by applying a trained regressor to the image, wherein the trained regressor is trained to predict two-dimensional projections of the three-dimensional bounding box of the object in a plurality of poses; and

estimate the three-dimensional pose of the object using the plurality of two-dimensional projections of the three-dimensional bounding box.

12. The apparatus of claim 11, wherein the regressor includes a convolutional neural network.

13. The apparatus of claim 11, wherein the processor is further configured to determine an image patch in the image, wherein the image patch is centered on the object, and wherein the plurality of two-dimensional projections of the three-dimensional bounding box are determined by applying the trained regressor to the image patch.

14. The apparatus of claim 13, wherein determining the image patch includes:

segmenting the image into an object area map of the image, the object area map including a grid of object area blocks;

determining a respective probability for each of the object area blocks in the grid, wherein a probability of an object area block indicates a probability that the object is located in the object area block; and

selecting an object area block with a highest determined probability from the grid of object area blocks.

15. The apparatus of claim 14, wherein determining the image patch further includes:

segmenting the selected object area block into a plurality of object center blocks;

determining a respective probability for each of the object center blocks in the plurality of object center blocks, a probability of an object center block indicating a probability that a center of the object is located in the object center block;

selecting an object center block with a highest determined probability from the plurality of object center blocks;

determining a center of the object in the selected object center block; and

generating the image patch by cropping the image, wherein the cropped image is centered on the center of the object.

16. The apparatus of claim 11, wherein estimating the three-dimensional pose of the three-dimensional object using the plurality of two-dimensional projections of the three-dimensional bounding box includes:

estimating a pose of a calibrated camera for the image, wherein the pose of the calibrated camera for the image is estimated using three-dimensional locations of the three-dimensional bounding box for the object in the image and the plurality of two-dimensional projections of the three-dimensional bounding box of the object in the image, wherein the plurality of two-dimensional projections include projections of the three-dimensional locations of the three-dimensional bounding box.

17. The apparatus of claim 11, wherein the plurality of two-dimensional projections of the three-dimensional bounding box of the object include projections of corners of the three-dimensional bounding box.

18. The apparatus of claim 11, wherein the image includes a red-green-blue (RGB) image.

19. The apparatus of claim 11, wherein the image does not include depth information.

20. The apparatus of claim 11, wherein the determined three-dimensional pose includes a three-dimensional orientation of the object and a three-dimensional translation of the object.

21. A non-transitory computer readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processor to:

obtain an image including an object;

determine a plurality of two-dimensional projections of a three-dimensional bounding box of the object, the plurality of two-dimensional projections of the three-dimensional bounding box being determined by applying a trained regressor to the image, wherein the trained regressor is trained to predict two-dimensional projections of the three-dimensional bounding box of the object in a plurality of poses; and

estimate a three-dimensional pose of the object using the plurality of two-dimensional projections of the three-dimensional bounding box.

22. The non-transitory computer readable medium of claim 21, wherein the regressor includes a convolutional neural network.

23. The non-transitory computer readable medium of claim 21, further comprising instructions that, when executed by the one or more processors, cause the one or more processors to determine an image patch in the image, wherein the image patch is centered on the object, and wherein the plurality of two-dimensional projections of the three-dimensional bounding box are determined by applying the trained regressor to the image patch.

24. The non-transitory computer readable medium of claim 23, wherein determining the image patch includes:

segmenting the image into an object area map of the image, the object area map including a grid of object area blocks;

determining a respective probability for each of the object area blocks in the grid, wherein a probability of an object area block indicates a probability that the object is located in the object area block; and

selecting an object area block with a highest determined probability from the grid of object area blocks.

25. The non-transitory computer readable medium of claim 24, wherein determining the image patch further includes:

segmenting the selected object area block into a plurality of object center blocks;

determining a respective probability for each of the object center blocks in the plurality of object center blocks, a probability of an object center block indicating a probability that a center of the object is located in the object center block;

selecting an object center block with a highest determined probability from the plurality of object center blocks;

determining a center of the object in the selected object center block; and

generating the image patch by cropping the image, wherein the cropped image is centered on the center of the object.

26. The non-transitory computer readable medium of claim 21, wherein estimating the three-dimensional pose of the three-dimensional object using the plurality of two-dimensional projections of the three-dimensional bounding box includes:

estimating a pose of a calibrated camera for the image, wherein the pose of the calibrated camera for the image is estimated using three-dimensional locations of the three-dimensional bounding box for the object in the image and the plurality of two-dimensional projections of the three-dimensional bounding box of the object in the image, wherein the plurality of two-dimensional projections include projections of the three-dimensional locations of the three-dimensional bounding box.

27. The non-transitory computer readable medium of claim 21, wherein the plurality of two-dimensional projections of the three-dimensional bounding box of the object include projections of corners of the three-dimensional bounding box.

28. The non-transitory computer readable medium of claim 21, wherein the image includes a red-green-blue (RGB) image.

29. The non-transitory computer readable medium of claim 21, wherein the image does not include depth information.

30. The non-transitory computer readable medium of claim 21, wherein the determined three-dimensional pose includes a three-dimensional orientation of the object and a three-dimensional translation of the object.