System and method of three-dimensional pose estimation

• US 7,957,583 B2
• Filed: 08/02/2007
• Issued: 06/07/2011
• Est. Priority Date: 08/02/2007
• Status: Active Grant

• Current Assignee: Roboticvisiontech LLC
• Original Assignee: Roboticvisiontech LLC
• Inventors: Boca, Remus F., Pescaru, Simona Liliana, Beis, Jeffrey Scott
• Primary Examiner(s): Chawan; Sheela C
• Application Number:
  US11/833,187
• Publication Number:
  US 20090033655A1
• Time in Patent Office: 1,405 Days
• Field of Search: 382/154, 382/153, 382/131, 382/100, 382/242, 382/238, 382/203, 382/232, 382/128, 382/181, 382/190, 382/195, 345/419, 345/163, 700/245, 359/9, 359/22, 359/23, 348/E13.014, 348/E13.019, 348/E13.023, 348/E13.025, 348/E13.033, 348/E13.04, 348/E13.044, 348/E13.059, 348/E13.067, 348/E13.071, 348/E13.072, 348/E13.073, 348/E13.038
• US Class Current: 382/154
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/35315 Projection, two, three section views
• CPC Class Codes: G05B 2219/40543 Identification and location, position of components, objects
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30164 Workpiece; Machine component
• CPC Class Codes: G06T 7/75 involving models

Abstract

A system and method for identifying objects using a machine-vision based system are disclosed. Briefly described, one embodiment is a method that captures a first image of at least one object with an image capture device, processes the first captured image to find an object region based on a reference two-dimensional model and determines a three-dimensional pose estimation based on a reference three-dimensional model that corresponds to the reference two-dimensional model and a runtime three-dimensional representation of the object region where a point-to-point relationship between the reference three-dimensional models of the object and the runtime three-dimensional representation of the object region is not necessarily previously known. Thus, two-dimensional information or data is used to segment an image and three-dimensional information or data used to perform three-dimensional pose estimation on a segment of the image.

Claims

1. A method of object pose estimation using machine-vision, comprising:

identifying an object region of an image on which pose estimation is being performed based on a correspondence between at least a portion of a representation of an object in the image and at least a corresponding one of a plurality of reference two-dimensional models of the object, the object region being a portion of the image that contains the representation of at least a portion of the object; and

determining a three-dimensional pose of the object based on at least one of a plurality of reference three-dimensional models of the object and a runtime three-dimensional representation of the object region where a point-to-point relationship between the reference three-dimensional models of the object and the runtime three-dimensional representation of the object region is not necessarily previously known.

2. The method of claim 1 wherein identifying an object region of an image includes comparing each of a number of features in a digital representation of the image with a number of features in the corresponding one of the reference two-dimensional models.

3. The method of claim 2 wherein identifying an object region of an image includes comparing each of the number of features in the digital representation of the image to a number of features in successive ones of the reference two-dimensional models until a match is found within a defined tolerance or no match is found among all of the reference two-dimensional models of a set of the reference two-dimensional models.

4. The method of claim 3 wherein comparing a number of features in digital representation of the image to a number of features in successive ones of the reference two-dimensional models includes comparing a representation of at least one of an edge, a point, or an image patch in the digital representation of the image to a representation at least one of an edge, a point, or an image patch in the reference two-dimensional model.

5. The method of claim 1 wherein identifying an object region of an image includes computationally performing at least one of correlation based pattern matching, blob analysis, or geometric pattern matching.

6. The method of claim 1, further comprising:

identifying the at least one of the plurality of reference three-dimensional models of the object based on the at least corresponding one of the reference two-dimensional models of the object.

7. The method of claim 6 wherein identifying at least one of the plurality of reference three-dimensional models of the object based on the at least corresponding one of the reference two-dimensional models of the object and a runtime three-dimensional representation of the object region where a point-to-point relationship between the reference three-dimensional models of the object and the runtime three-dimensional representation of the object region is not necessarily previously known includes identifying at least one of the reference three-dimensional models based on a stored relationship between the corresponding one of the reference two-dimensional models and the at least one of the reference three-dimensional models.

8. The method of claim 7 wherein determining a three-dimensional pose of the object based on at least one of a plurality of reference three-dimensional models of the object and a runtime three-dimensional representation of the object region where a point-to-point relationship between the reference three-dimensional models of the object and the runtime three-dimensional representation of the object region is not necessarily previously known includes performing a registration between the at least one of the reference three-dimensional models and a digital runtime three-dimensional representation of the object region of the image.

9. The method of claim 8 wherein the digital runtime three-dimensional representation of the object region of the image is a runtime three-dimensional model of the object region of the image and wherein performing a registration between the at least one of the reference three-dimensional models and a digital runtime three-dimensional representation of the object region of the image includes executing an error minimization algorithm to minimize error between the at least one of the reference three-dimensional models and the runtime three-dimensional model of the object region of the image.

10. The method of claim 9 wherein executing an error minimization algorithm includes executing an iterative closest point algorithm.

11. The method of claim 8, further comprising:

extracting three-dimensional information from the object region of the image.

12. The method of claim 11, further comprising:

forming the runtime three-dimensional model of the object region of the image from the runtime three-dimensional information extracted from the object region of the image.

13. The method of claim 8, further comprising:

providing an indication that the three-dimensional pose of the object has not been found if an outcome of the registration is unsuccessful.

14. The method of claim 1 wherein determining a three-dimensional pose of the object based on at least one of a plurality of reference three-dimensional models of the object and a runtime three-dimensional representation of the object region where a point-to-point relationship between the reference three-dimensional models of the object and the runtime three-dimensional representation of the object region is not necessarily previously known includes performing a registration between a set of dense three-dimensional data stored as the at least one reference three-dimensional model and a set of dense three-dimensional data stored as the runtime three-dimensional representation of the object region of the image.

15. The method of claim 1, further comprising:

capturing the image.

16. The method of claim 15 wherein the capturing an image, the identifying an object region of the image and the determining a three-dimensional pose of the object all occur during a runtime mode that follows a training mode.

17. The method of claim 16, further comprising:

acquiring the reference two-dimensional models during the training mode.

18. The method of claim 17 wherein acquiring the reference two-dimensional models during the training mode includes at least one of accessing an existing computer model of the object or sensing data from a representative object.

19. The method of claim 17, further comprising:

acquiring the reference three-dimensional models during the training mode.

20. The method of claim 19 wherein acquiring the reference three-dimensional models during the training mode includes acquiring information using at least one of a dense stereo sensor system, a laser triangulation system, a laser time of fight system or an ultrasound transducer.

21. The method of claim 19 wherein acquiring the reference three-dimensional models during the training mode includes identifying a portion of the image that contains a digital representation of at least part of the object.

22. The method of claim 21 wherein identifying a portion of the image that contains a digital representation of at least part of the object is performed either manually or automatically.

23. A non-transitory computer readable medium that stores instructions for causing a computer to perform object pose estimation using machine-vision, by:

identifying an object region of an image based on a correspondence between at least a portion of a representation of an object in the object region of the image and at least a corresponding one of a plurality of reference two-dimensional models of the object, the object region being a portion of the image that contains the representation of at least a portion of the object; and

determining a three-dimensional pose of the object based on at least one of a plurality of reference three-dimensional models of the object and a runtime three-dimensional representation of the object region where a point-to-point relationship between the reference three-dimensional models of the object and the runtime three-dimensional representation of the object region is not necessarily previously known.

24. The non-transitory computer readable medium of claim 23 wherein the instructions cause the computer to perform object pose estimation using machine-vision, further by:

extracting two-dimensional information from the object region of the image during a runtime to form a runtime two-dimensional digital model of the object region of the image.

25. The non-transitory computer readable medium of claim 24 wherein identifying an object region of an image includes comparing each of the number of features in the runtime two-dimensional digital model of the object region of the image to a number of features in successive ones of the reference two-dimensional models until a match within a defined tolerance is found or no match is found among all of the reference two-dimensional models of a set of the reference two-dimensional models.

26. The non-transitory computer readable medium of claim 24 wherein the instructions cause the computer to perform object pose estimation using machine-vision, further by:

extracting three-dimensional information from the object region of the image during the runtime to form a runtime three-dimensional model of the object region of the image.

27. The non-transitory computer readable medium of claim 26 wherein the instructions cause the computer to perform object pose estimation using machine-vision, further by:

identifying the at least one of the plurality of reference three-dimensional models of the object based on a stored relationship between the corresponding one of the reference two-dimensional models and the at least one of the reference three-dimensional models.

28. The non-transitory computer readable medium of claim 26 wherein determining a three-dimensional pose of the object based on at least one of a plurality of reference three-dimensional models of the object and a runtime three-dimensional representation of the object region where a point-to-point relationship between the reference three-dimensional models of the object and the runtime three-dimensional representation of the object region is not necessarily previously known includes performing a registration between the at least one of the reference three-dimensional models and the runtime three-dimensional model of the object region of the image.

29. The non-transitory computer readable medium of claim 23 wherein the identifying an object region of the image and the determining a three-dimensional pose of the object all occur during a runtime mode following a training mode.

30. The non-transitory computer readable medium of claim 29 wherein the instructions cause the computer to perform object pose estimation using machine-vision, further by:

acquiring the reference two-dimensional models during the training mode; and

acquiring the reference three-dimensional models during the training mode.

31. A system to perform three-dimensional pose estimation, the system comprising:

at least one sensor;

at least one processor; and

at least one memory storing processor executable instructions that cause the at least one processor to segment an image captured by the at least one sensor into a number of object regions based at least in part on a correspondence between at least a portion of a representation of an object in the object region of the image and at least a corresponding one of a plurality of reference two-dimensional models of the object and to cause the at least one processor to determine a three-dimensional pose of the object based on at least one of a plurality of reference three-dimensional models of the object that is related to the corresponding one of the plurality of reference two-dimensional models of the object and a runtime three-dimensional representation of the object region where a point-to-point relationship between the reference three-dimensional models of the object and the runtime three-dimensional representation of the object region is not necessarily previously known.

32. The system of claim 31 wherein the instructions further cause at least one processor to identify the at least one of the plurality of reference three-dimensional models of the object based on a set of stored relationships between a plurality of reference two-dimensional models and the plurality of reference three-dimensional models of the object.

33. The system of claim 31 wherein the at least one sensor includes at least one of an imager mounted for movement, a stereo pair of cameras, and a laser.

34. The system of claim 31 wherein the at least one sensor includes at least one imager mounted for movement with respect to the object.

35. The system of claim 31 wherein the at least one sensor includes at least one imager and at least one of a laser or a set of structured lighting.

36. The system of claim 31 wherein the at least one sensor includes at least one stereo pair of cameras.

37. The system of claim 31 wherein the instructions further cause the at least one processor to provide drive signals to drive a robotic member based on the determined three-dimensional pose estimation.