System and method for finding correspondence between cameras in a three-dimensional vision system

• US 9,124,873 B2
• Filed: 10/24/2013
• Issued: 09/01/2015
• Est. Priority Date: 12/08/2010
• Status: Active Grant

• Current Assignee: Cognex Corporation
• Original Assignee: Cognex Corporation
• Inventors: Liu, Lifeng, Wallack, Aaron S., Marrion, Cyril C. Jr., Michael, David J.
• Primary Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US14/062,718
• Publication Number:
  US 20140118500A1
• Time in Patent Office: 677 Days
• Field of Search: 382/154, 382/190, 382/285, 382/291, 382/295, 382/296, 382/305, 382/312, 348/46, 348/47, 345/419, 345/653, 345/654, 345/662, 700/245, 700/259
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10056 Microscopic image
• CPC Class Codes: G06T 2207/30164 Workpiece; Machine component
• CPC Class Codes: G06T 2207/30208 Marker matrix
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00

Abstract

This invention provides a system and method for determining correspondence between camera assemblies in a 3D vision system implementation having a plurality of cameras arranged at different orientations with respect to a scene involving microscopic and near microscopic objects under manufacture moved by a manipulator, so as to acquire contemporaneous images of a runtime object and determine the pose of the object for the purpose of guiding manipulator motion. At least one of the camera assemblies includes a non-perspective lens. The searched 2D object features of the acquired non-perspective image, corresponding to trained object features in the non-perspective camera assembly can be combined with the searched 2D object features in images of other camera assemblies, based on their trained object features to generate a set of 3D features and thereby determine a 3D pose of the object.

Claims

1. A vision system that determines 3D pose of an object under manufacture having a size in the microscopic or near-microscopic range comprising:

a plurality of cameras in which a lens of at least one of the cameras is optically connected to a non-perspective microscope lens to acquire a non-perspective image, the cameras being respectively oriented with a discrete field of view so as to contemporaneously acquire a plurality of images of the object, respectively, at a known position with respect to the field of view;

a search process, operatively connected to the cameras, that searches for 2D feature points in at least some of the acquired images, including the non-perspective image;

an association process that associates the 2D feature points with known 3D model points;

a pose determination process that computes the 3D pose based upon the 2D feature points and known 3D model points; and

a manipulator control that moves the object to a predetermined location based upon the determined 3D pose.

2. The vision system as set forth in claim 1 wherein each of the cameras is a non-perspective camera with a respective non-perspective lens.

3. The vision system as set forth in claim 2 wherein the non-perspective lens for at least one of the cameras is a telecentric lens.

4. The vision system as set forth in claim 2 wherein the lens for a first camera of the plurality of cameras is a non-perspective lens and the lens for a second camera of the plurality of cameras is a non-perspective lens, and further comprising a trained model for use by the second camera to search for the 2D features, the trained model being based upon an affine transform of a first model with at least one reference point in the image acquired by the first camera.

5. The vision system as set forth in claim 4 wherein the affine transform is based upon at least a subset of intrinsics and extrinsics of the first camera and the second camera.

6. The vision system as set forth in claim 5 wherein the search process is constructed and arranged to acquire a second image with the second camera and to search for 2D features in the second image within a predetermined range based upon the trained model.

7. The vision system as set forth in claim 1 wherein the device is a robot manipulator that moves the object from a source location to a receiving location in which the object is assembled to another component.

8. The vision system as set forth in claim 1 wherein the pose determination process includes a triangulation process based upon light rays extending between respective of the cameras and feature points on the object.

9. The vision system as set forth in claim 8 wherein the triangulation process is constructed and arranged to employ least squares fit to determine points of intersection of the light rays.

10. A method for determining 3D pose of an object under manufacture having a size in the microscopic or near-microscopic range comprising the steps of:

locating a plurality of cameras wherein a lens of at least one of the cameras optically connected to a non-perspective microscope lens, the cameras being respectively oriented with a discrete field of view so as to contemporaneously acquire a plurality of images of the object at a known position with respect to the field of view, including at least one non-perspective image;

searching for 2D feature points in at least some of the images of the plurality of images including the non-perspective image;

associating the 2D feature points with known 3D model points;

computing the 3D pose based upon the 2D feature points and known 3D model points; and

moving the object to a predetermined location with a manipulator based upon the computed 3D pose.

11. The method as set forth in claim 10 wherein the non-perspective lenses is a telecentric lens.

12. The method as set forth in claim 10 wherein the lens for a first camera of the plurality of cameras is a non-perspective lens and the lens for a second camera of the plurality of cameras is a non-perspective lens, and further comprising searching for 2D features with the second camera using a trained model, the trained model being based upon an affine transform of a first model with at least one reference point in the image acquired by the first camera.

13. The method as set forth in claim 10 further comprising computing the affine transform based upon at least a subset of intrinsics and extrinsics of the first camera and the second camera.

14. The method as set forth in claim 13 further comprising acquiring a second image with the second camera and searching for 2D features in the second image within a predetermined range based upon the trained model.

15. The method as set forth in claim 10 wherein the step of moving comprises moving the object with a robot manipulator from a pick location to a place location having another component, and assembling the object to the other component.

16. A system for guiding a manipulator that releasably holds an object sized in the microscopic or near-microscopic range comprising:

a plurality of cameras in which at least one of the cameras is optically connected to a non-perspective microscope lens to acquire a non-perspective image, the cameras being respectively oriented with a discrete field of view so as to contemporaneously acquire a plurality of images of the object in a location in which the object is undergoing a manufacturing task, respectively, at an alignment location with respect to the field of view;

a search process, operatively connected to the cameras, that searches for 2D feature points in at least some of the acquired images, including the non-perspective image;

an association process that associates the 2D feature points with known 3D model points;

a pose determination process that computes the 3D pose based upon the 2D feature points and known 3D model points; and

a manipulator control that moves the manipulator to locate the object at the alignment location based upon the determined 3D pose.