System and method for tying together machine vision coordinate spaces in a guided assembly environment

• US 10,290,118 B2
• Filed: 07/29/2016
• Issued: 05/14/2019
• Est. Priority Date: 08/06/2015
• Status: Active Grant

• Current Assignee: Cognex Corporation
• Original Assignee: Cognex Corporation
• Inventors: Shivaram, Guruprasad, Marrion, Jr., Cyril C., Liu, Lifeng, Li, Tuotuo
• Primary Examiner(s): Thomas, Mia M
• Application Number:
  US15/223,436
• Publication Number:
  US 20170132807A1
• Time in Patent Office: 1,019 Days
• CPC Class Codes: B25J 9/1692 Calibration of manipulator
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/39039 Two cameras detect same reference on workpiece to define its position in space
• CPC Class Codes: G06T 2207/30164 Workpiece; Machine component
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: Y10S 901/47 Optical

Abstract

This invention provides a system and method that ties the coordinate spaces at the two locations together during calibration time using features on a runtime workpiece instead of a calibration target. Three possible scenarios are contemplated: wherein the same workpiece features are imaged and identified at both locations; wherein the imaged features of the runtime workpiece differ at each location (with a CAD or measured workpiece rendition available); and wherein the first location containing a motion stage has been calibrated to the motion stage using hand-eye calibration and the second location is hand-eye calibrated to the same motion stage by transferring the runtime part back and forth between locations. Illustratively, the quality of the first two techniques can be improved by running multiple runtime workpieces each with a different pose, extracting and accumulating such features at each location; and then using the accumulated features to tie the two coordinate spaces.

Claims

1. A method for calibrating a vision system in an environment in which a first workpiece at a first location is transferred by a manipulator to a second location, wherein an operation performed on the first workpiece relies upon tying together coordinate spaces of the first location and the second location, the method comprising the steps of:

arranging at least one vision system camera to image the first workpiece when positioned at the first location and to image the first workpiece when positioned at the second location;

calibrating at least one vision system camera with respect to the first location to derive first calibration data which defines a first coordinate space and at least one vision system camera with respect to the second location to derive second calibration data which defines a second coordinate space;

identifying features of at least the first workpiece at the first location from a first image of the first workpiece;

based on the identified features in the first image locating the first workpiece with respect to the first coordinate space relative to the first location;

gripping and moving, with the manipulator, at least one time, the first workpiece to a predetermined manipulator position at the second location;

acquiring a second image of the first workpiece at the second location; and

based upon the identified features in the second image, locating the first workpiece with respect to the second coordinate space relative to the second location and tying together the first coordinate space and the second coordinate space.

2. The method as set forth in claim 1 wherein the identified features in the first image are the same as the identified features in the second image, and further comprising (a) mapping locations of the identified features in the first image with respect to the first calibration data, (b) mapping locations of the identified features in the second image with respect to the second calibration data, and (c) computing a transform mapping the mapped features at the second location to the mapped features at the first location.

3. The method as set forth in claim 1 wherein some of the identified features in the first image differ from the identified features in the second image, and further comprising (a) mapping locations of the identified features in the first image with respect to the first calibration data, (b) computing a transform relative to a stored specification of feature locations of the first workpiece, (c) mapping locations of the identified features in the second image with respect to the second calibration data, (d) using the transform computed in step (b) to derive locations of the identified features from the second image in the first coordinate space when the first workpiece is located at the first location, and (e) computing a transform mapping the mapped features at the second location to the corresponding transformed features at the first location.

4. The method as set forth in claim 3 wherein the stored specification is based upon a CAD model of the first workpiece.

5. The method as set forth in claim 3 wherein the stored specification is based upon a measured model of the first workpiece.

6. The method as set forth in claim 1 further comprising (a) moving the first workpiece iteratively with a motion rendering device at either the first location or the second location to a plurality of different poses, (b) identifying features at each of the poses at each of the first location and the second location and (c) accumulating the identified feature information to enhance accuracy, wherein the first workpiece is either the same workpiece or is one of a plurality of discrete workpieces.

7. The method as set forth in claim 1 further comprising providing a mapping from an image coordinate system to a calibration coordinate system at the first location, and wherein the mapping is unity.

8. The method as set forth in claim 1 wherein the second location has a second workpiece into which the first workpiece is placed into engagement in a desired alignment with the second workpiece.

9. The method as set forth in claim 8 wherein the second workpiece is a part, a container or a framework for further processing of the first workpiece.

10. The method as set forth in claim 1 wherein the operation is at least one of an alignment operation with respect to another object, a printing operation on the first workpiece, and an application operation on the first workpiece.

11. The method as set forth in claim 10 wherein the operation is performed at least in part at a location remote from the first location and the second location.

12. A method for calibrating a vision system in an environment in which a first workpiece at a first location is transferred by a manipulator to a second location, wherein an operation performed on the first workpiece relies upon tying together coordinate spaces of the first location and the second location, the method comprising the steps of:

(a) arranging at least one vision system camera to image the first workpiece at the first location and to image the second location;

(b) hand-eye calibrating at least one vision system camera with respect to the first location to derive first calibration data;

(c) positioning the first workpiece at the first location;

(d) moving the first workpiece from the first location to the second location;

(e) acquiring an image and locating features on the first workpiece;

(f) moving the first workpiece to the first location from the second location and changing a pose of the first workpiece at the first location by moving the motion rendering device to a new known pose;

(g) iterating steps (d-f) until feature location and other data relevant to hand-eye calibration is accumulated; and

(h) using the accumulated data to hand-eye calibrate at least one vision system camera with respect to the second location, and tying together the first coordinate space and the second coordinate space by the common coordinate space relative to the motion rendering device obtained from the hand-eye calibration.

13. The method as set forth in claim 12 wherein the second location has a second workpiece on the motion rendering device into which the first workpiece is placed into engagement in a desired alignment with the second workpiece.

14. The method as set forth in claim 13 wherein the second workpiece is a part, a container or a framework for further processing of the first workpiece.

15. The method as set forth in claim 12 wherein the operation is at least one of an alignment operation with respect to another object, a printing operation on the first workpiece, and an application operation on the first workpiece.

16. A system for calibrating a vision system in an environment in which a first workpiece at a first location is transferred by a manipulator to a second location, wherein an operation performed on the first workpiece relies upon tying together coordinate spaces of the first location and the second location, comprising:

at least one vision system camera arranged to image the first workpiece when positioned at the first location and to image the first workpiece when positioned at the second location;

a calibration process that calibrates at least one vision system camera with respect to the first location to derive first calibration data and the at least one vision system camera with respect to the second location to derive second calibration data;

a feature extraction process that identifies features of at least the first workpiece at the first location from a first image of the first workpiece, and based on the identified features in the first image, that locates the first workpiece with respect to a first coordinate space relative to the first location, and based upon the identified features in a second image at a second location, that locates the first workpiece with respect to a second coordinate space relative to the second location; and

a calibration process that ties together the first coordinate space and the second coordinate space.

17. The system as set forth in claim 16 wherein the second location has a second workpiece into which the first workpiece is placed into engagement in a desired alignment with the second workpiece.

18. The system as set forth in claim 16 wherein some of the identified features in the first image differ from the identified features in the second image, and wherein the calibration process is constructed and arranged to (a) map locations of the identified features in the first image with respect to the first calibration data, (b) compute a transform relative to a stored specification of feature locations of the first workpiece, (c) map locations of the identified features in the second image with respect to the second calibration data, (d) using the transform computed in step (b) derive locations of the identified features from the second image in the first coordinate space when the first workpiece is located at the first location, and (e) compute a transform mapping the mapped features at the second location to the corresponding transformed features at the first location.

19. The system as set forth in claim 18 wherein the stored specification is based upon either a CAD model of the first workpiece or a measured model of the first workpiece.

20. The system as set forth in claim 16 wherein the second location has a second workpiece into which the first workpiece is placed into engagement in a desired alignment with the second workpiece.