Systems and methods for articulated pose estimation

• US 10,535,155 B2
• Filed: 10/24/2017
• Issued: 01/14/2020
• Est. Priority Date: 10/24/2017
• Status: Active Grant

• Current Assignee: Carnegie Mellon University, Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Original Assignee: Toyota Motor Engineering & Manufacturing North America Incorporated (Toyota Motor Corporation)
• Inventors: Cao, Zhe, Zhu, Qi, Sheikh, Yaser, Chelian, Suhas E.
• Primary Examiner(s): Coleman, Stephen P
• Application Number:
  US15/791,986
• Publication Number:
  US 20190122380A1
• Time in Patent Office: 812 Days
• Field of Search: 382103
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 30/19173 Classification techniques

Abstract

Systems and methods for articulated pose estimation are provided. Some embodiments include training a convolutional neural network for object pose estimation, which includes receiving a two-dimensional training image of an articulated object that has a plurality of components and identifying, from the two-dimensional training image, at least one key point for each of the plurality of components. Some embodiments also include testing the accuracy of the object pose estimation, which includes visualizing a three or more dimensional pose of each of the plurality of components of the articulated object from a two-dimensional testing image and providing data related to the visualization for output.

Claims

1. A method for articulated pose estimation comprising:

training a convolutional neural network for object pose estimation, which includes at least the following;

receiving a two-dimensional training image of an articulated object, wherein the articulated object includes a plurality of components; and

identifying, from the two-dimensional training image, at least one key point for each of the plurality of components; and

testing accuracy of the object pose estimation, which includes at least the following;

rendering a three or more dimensional pose of each of the plurality of components of the articulated object from a two-dimensional testing image;

solving the three or more dimensional pose from the two-dimensional location of the at least one key point to render the three or more dimensional pose of each of the plurality of components of the articulated object; and

providing data related to the rendering for output.

2. The method of claim 1, wherein testing accuracy of the object pose estimation further comprises:

rendering a validation object model of the articulated object in the three or more dimensional pose;

comparing the validation object model to a predetermined rendering of the articulated object to determine whether the validation object model meets a predetermined accuracy threshold; and

in response to determining that the validation object model does not meet the predetermined accuracy threshold, re-implementing training.

3. The method of claim 1, wherein training further comprises generating an object model of the plurality of components of the articulated object.

4. The method of claim 3, wherein receiving the two-dimensional training image includes at least one of the following:

receiving the two-dimensional training image of the articulated object in an environment from a remote computing device and rendering the object model into an existing image.

5. The method of claim 1, wherein identifying the at least one key point for each of the plurality of components includes rendering a heat map to indicate a level of confidence associated with the respective at least one key point.

6. The method of claim 1, wherein testing the object pose estimation further includes:

receiving the two-dimensional testing image that includes the articulated object; and

identifying a two-dimensional location of the at least one key point for each of the plurality of components.

7. The method of claim 6, wherein testing accuracy of the object pose estimation further comprises:

implementing a perspective-n point (PnP) module to solve the three or more dimensional pose from the two-dimensional location of the at least one key point; and

eliminating at least one false key point match.

8. A system for articulated pose estimation comprising:

a processor;

a convolutional neural network; and

a memory component that stores training logic and testing logic, wherein the training logic, when executed by the processor, causes the system to perform at least the following;

receive an object model of an articulated object, wherein the object model includes at least one key point for the articulated object;

receive a training image of the articulated object; and

cause the convolutional neural network to identify, from the training image and the object model, a two-dimensional location of the at least one key point in the training image;

wherein the testing logic, when executed by the processor, causes the system to perform at least the following;

receive a two-dimensional testing image that includes the articulated object;

deploy the convolutional neural network to determine the two-dimensional location of the at least one key point in the two-dimensional testing image;

render a three or more dimensional estimated pose of the articulated object from the two-dimensional location of the at least one key point to determine accuracy of the three or more dimensional estimated pose of the articulated object; and

provide data related to the accuracy for display.

9. The system of claim 8, wherein receiving the two-dimensional testing image includes at least one of the following:

receiving the two-dimensional testing image of the articulated object in an environment from a remote computing device and rendering the object model into an existing image.

10. The system of claim 8, wherein identifying the at least one key point includes rendering a heat map to indicate a level of confidence associated with the respective at least one key point.

11. The system of claim 8, wherein the testing logic further causes the system to implement a perspective-n point (PnP) module to solve the three or more dimensional pose from the two-dimensional location of the at least one key point.

12. The system of claim 8, wherein determining the three or more dimensional pose of the articulated object includes determining a six-dimensional pose of the articulated object.

13. The system of claim 8, wherein determining the at least one key point includes receiving input from a user to identify the two-dimensional location of the at least one key point.

14. The system of claim 8, wherein the testing logic further causes the system to perform at least the following:

rendering a validation object model of the articulated object in the three or more dimensional pose;

comparing the validation object model to a predetermined rendering of the articulated object to determine whether the validation object model meets a predetermined accuracy threshold; and

in response to determining that the validation object model does not meet the predetermined accuracy threshold, re-implementing training.

15. A system for articulated pose estimation comprising:

a robot device;

a processor; and

a memory component that stores training logic, testing logic, and convolutional neural network logic, wherein the training logic, when executed by the processor causes the system to perform at least the following;

cause the convolutional neural network logic to identify, from a two-dimensional training image, a two-dimensional location of at least one key point on an articulated object;

wherein the testing logic, when executed by the processor, causes the system to perform at least the following;

receive a two-dimensional testing image of the articulated object;

deploy the convolutional neural network logic to determine, from the two-dimensional testing image, the two-dimensional location of the at least one key point;

solve a three or more dimensional estimated pose of the articulated object from the two-dimensional location of the at least one key point in the two-dimensional training image;

determine accuracy of the three or more dimensional estimated pose; and

provide data related to the accuracy for display.

16. The system of claim 15, wherein receiving the two-dimensional training image includes at least one of the following:

receiving the two-dimensional training image of the articulated object in an environment from a remote computing device and rendering an object model of the articulated object into an existing image.

17. The system of claim 15, wherein identifying the at least one key point includes rendering a heat map of the at least one key point to indicate a level of confidence associated with the respective at least one key point.

18. The system of claim 15, wherein the testing logic further causes the system to implement a perspective-n point (PnP) module to solve the three or more dimensional pose from the two-dimensional location of the at least one key point.

19. The system of claim 15, wherein determining the three or more dimensional pose of the articulated object includes determining a six-dimensional pose of the articulated object.

20. The system of claim 15, wherein the testing logic further causes the system to perform at least the following:

render a validation object model of the articulated object in the three or more dimensional pose;

compare the validation object model to a predetermined rendering of the articulated object to determine whether the validation object model meets a predetermined accuracy threshold; and

in response to determining that the validation object model does not meet the predetermined accuracy threshold, re-implement training.