System and method for robust calibration between a machine vision system and a robot

• US 9,393,694 B2
• Filed: 05/14/2010
• Issued: 07/19/2016
• Est. Priority Date: 05/14/2010
• Status: Active Grant

• Current Assignee: Cognex Corporation
• Original Assignee: Cognex Corporation
• Inventors: Wallack, Aaron S., Liu, Lifeng, Ye, Xiangyun
• Primary Examiner(s): DULANEY, KATHLEEN YUAN
• Application Number:
  US12/780,119
• Publication Number:
  US 20110280472A1
• Time in Patent Office: 2,258 Days
• Field of Search: 382/153, 382/154
• US Class Current: 1/1
• CPC Class Codes: B25J 9/1607 Calculation of inertia, jacobian matrixes and inverses
• CPC Class Codes: B25J 9/1692 Calibration of manipulator
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/39008 Fixed camera detects reference pattern held by end effector
• CPC Class Codes: G05B 2219/39022 Transform between measuring and manipulator coordinate system
• CPC Class Codes: G05B 2219/39045 Camera on end effector detects reference pattern
• CPC Class Codes: G05B 2219/39057 Hand eye calibration, eye, camera on hand, end effector
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: Y10S 901/09 Closed loop, sensor feedback controls arm movement

Abstract

A system and method for robustly calibrating a vision system and a robot is provided. The system and method enables a plurality of cameras to be calibrated into a robot base coordinate system to enable a machine vision/robot control system to accurately identify the location of objects of interest within robot base coordinates.

Claims

1. A method for determining calibration between a machine vision system and a robot, the method comprising:

obtaining, using a camera fixed in space, an initial image of a calibration object fixed to an effector of the robot as the robot occupies an initial pose within a workplace for the robot, and a subsequent image of the calibration object as the robot occupies a subsequent pose within the workplace for the robot, the initial pose and the subsequent pose are different, the calibration object comprising a first object feature and a second object feature, the first object feature and the second object feature located at a fixed, known distance relative to one another, the robot having a robot coordinate system, the camera having a camera coordinate system, the effector having an effector coordinate system, the calibration object having an object coordinate system;

identifying a camera-robot transform between the camera coordinate system and the robot coordinate system;

identifying an object-effector transform between the object coordinate system and the effector coordinate system;

locating a first initial image feature and a second initial image feature in the initial image and a first subsequent image feature and a second subsequent image feature in the subsequent image, the first initial image feature and the first subsequent image feature corresponding to the first object feature and the second initial image feature and the second subsequent image feature corresponding to the second object feature;

calculating, using the fixed, known distance, the initial pose, the subsequent pose, the camera-robot transform, and the object-effector transform, a predicted first initial image feature and a predicted second initial image feature for the initial image and a predicted first subsequent image feature and a predicted second subsequent image feature for the subsequent image, the predicted first initial image feature and the predicted first subsequent image feature corresponding to the first object feature and the predicted second initial image feature and the predicted second subsequent image feature corresponding to the second object feature;

minimizing, by varying the camera-robot transform or the object-effector transform, a discrepancy between at least one of the first initial image feature and the predicted first initial image feature, the second initial image feature and the predicted second initial image feature, the first subsequent image feature and the predicted first subsequent image feature, and the second subsequent image feature and the predicted second subsequent image feature, thereby producing optimized transforms; and

calibrating the machine vision system and the robot using the optimized transforms.

2. The method of claim 1 wherein the calibration object comprises a calibration plate.

3. The method of claim 1 wherein a difference between the initial pose and the subsequent pose comprises rotations around at least two non-parallel axes.

4. The method of claim 3 wherein the rotations comprise rotation of at least five degrees.

5. The method of claim 1 further comprising pre-processing a set of obtained images to remove outliers, where the pre-processing further comprises:

determining a set of outliers;

eliminating, in response to determining a set of outliers, one or more images from the set of obtained images having the outliers to produce a reduced set of obtained images; and

re-ordering the reduced set of obtained images to obtain suitable motions.

6. The method of claim 5 wherein the suitable motions comprises changes between a particular pair of the reduced set of obtained images.

7. The method of claim 1 wherein the transforms permit six degrees of freedom.

8. A method for determining calibration between a machine vision system and a robot, the method comprising:

obtaining, using a primary camera fixed in space, a primary initial image of a calibration object fixed to an effector of the robot as the robot occupies a primary initial pose within a workplace for the robot, and a primary subsequent image of the calibration object as the robot occupies a primary subsequent pose within the workplace for the robot, the primary initial pose and the primary subsequent pose are different, the calibration object comprising a first object feature and a second object feature, the first object feature and the second object feature located at a fixed, known distance relative to one another, the robot having a robot coordinate system, the primary camera having a primary camera coordinate system, the effector having an effector coordinate system, the calibration object having an object coordinate system;

obtaining, using a secondary camera fixed in space, a secondary initial image of the calibration object as the robot occupies a secondary initial pose within the workplace for the robot, and a secondary subsequent image of the calibration object as the robot occupies a secondary subsequent pose within the workplace for the robot, the secondary initial pose and the secondary subsequent pose are different, the secondary camera having a secondary camera coordinate system,

identifying camera-robot transforms between each of the primary and secondary camera coordinate systems and the robot coordinate system;

identifying an object-effector transform between the object coordinate system and the effector coordinate system;

locating a first primary initial image feature and a second primary initial image feature in the primary initial image, a first primary subsequent image feature and a second primary subsequent image feature in the primary subsequent image, a first secondary initial image feature and a second secondary initial image feature in the secondary initial image, and a first secondary subsequent image feature, and a second secondary subsequent image feature in the secondary subsequent image, the first primary initial image feature, the first primary subsequent image feature, the first secondary initial image feature, and the first secondary subsequent image feature corresponding to the first object feature, and the second primary initial image feature, the second primary subsequent image feature, the second secondary initial image feature, and the second secondary subsequent image feature corresponding to the second object feature;

calculating, using the fixed, known distance, the primary initial pose, the primary subsequent pose, the secondary initial pose, the secondary subsequent pose, the camera-robot transforms, and the object-effector transform, a predicted first primary initial image feature and a predicted second primary initial feature for the primary initial image, a predicted first primary subsequent image feature and a predicted second primary subsequent image feature for the primary subsequent image, a predicted first secondary initial image feature and a predicted second secondary initial feature for the secondary initial image, a predicted first secondary subsequent image feature and a predicted second secondary subsequent image feature for the secondary subsequent image, the predicted first primary initial image feature, the predicted first primary subsequent image feature, the predicted first secondary initial image feature, and the predicted first secondary subsequent image feature corresponding to the first object feature, and the predicted second primary initial image feature, the predicted second primary subsequent image feature, the predicted second secondary initial image feature, and the predicted second secondary subsequent image feature corresponding to the second object feature;

minimizing, by varying the camera-robot transforms and the object-effector transform, a discrepancy between at least one of the first primary initial image feature and the predicted first primary initial image feature, the second primary initial image feature and the predicted second primary initial image feature, the first primary subsequent image feature and the predicted first primary subsequent image feature, and the second primary subsequent image feature and the predicted second primary subsequent image feature, the first secondary initial image feature and the predicted first secondary initial image feature, the second secondary initial image feature and the predicted second secondary initial image feature, the first secondary subsequent image feature and the predicted first secondary subsequent image feature, and the second secondary subsequent image feature and the predicted second secondary subsequent image feature, thereby producing optimized transforms; and

calibrating the machine vision system and the robot using the optimized transforms.

9. The method of claim 8 wherein the calibration object comprises a calibration plate.

10. The method of claim 8 wherein a difference between the primary initial pose and the primary subsequent pose and a difference between the secondary initial pose and the secondary subsequent pose each comprises rotations around at least two non-parallel axes.

11. The method of claim 10 wherein the rotations comprise rotation of at least five degrees.

12. The method of claim 8 further comprising pre-processing a set of obtained images to remove outliers, where the pre-processing further comprises:

determining a set of outliers;

eliminating, in response to determining a set of outliers, one or more images from the set of obtained images having the outliers to produce a reduced set of obtained images; and

re-ordering the reduced set of obtained images to obtain suitable motions.

13. The method of claim 12 wherein the suitable motions comprises changes between a particular pair of the reduced set of obtained images.

14. The method of claim 8 wherein identifying camera robot transforms identifies the transforms for each of the primary and secondary cameras simultaneously.

15. The method of claim 8 wherein the transforms permit six degrees of freedom.

16. A method for determining calibration between a machine vision system and a robot, the method comprising:

obtaining, using two or more cameras fixed in space, a plurality of images of a calibration object fixed to an effector of the robot as the robot moves the calibration object to a plurality of different poses within the workplace for the robot, the calibration object comprising a first object feature and a second object feature, the first object feature and the second object feature located at a fixed, known distance relative to one another, the robot having a robot coordinate system, the two or more cameras each having a camera coordinate system, the effector having an effector coordinate system, the calibration object having an object coordinate system;

identifying a camera-robot transform between each of the two or more camera coordinate systems and the robot coordinate system;

identifying an object-effector transform between the object coordinate system and the effector coordinate system;

identifying image features in the plurality of images, the image features corresponding to the first object feature and the second object feature;

calculating, using the fixed, known distance, the plurality of poses, the two or more camera-robot transforms, and the object-effector transforms, predicted features in the plurality of images, the predicted features corresponding to the first object feature and the second object feature;

minimizing, by simultaneously varying the two or more camera-robot transforms and the object-effector transform, a discrepancy between the predicted features and the image features, thereby producing optimized transforms;

calibrating the machine vision system and the robot using the optimized transforms.