Method and apparatus for single camera 3D vision guided robotics

• US 20030144765A1
• Filed: 05/24/2002
• Published: 07/31/2003
• Est. Priority Date: 01/31/2002
• Status: Active Grant

• Current Assignee: Roboticvisiontech LLC
• Original Assignee: Braintech Canada, Inc.
• Inventors: Pescaru, Simona, Habibi, Babak
• Granted Patent: US 6,816,755 B2
• Time in Patent Office: Days
• US Class Current: 700/259
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G05B 2219/39057 Hand eye calibration, eye, camera on hand, end effector
• CPC Class Codes: G05B 2219/39393 Camera detects projected image, compare with reference image, position end effector
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 2207/30164 Workpiece; Machine component
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19

Abstract

A method of three-dimensional handling of an object by a robot using a tool and one camera mounted on the robot is disclosed in which at least six target features which are normal features of the object are selected on the object.

Claims

1. A method of three-dimensional handling of an object by a robot using a tool and one camera mounted on the robot, comprising:

i) calibrating the camera by finding

a) the camera intrinsic parameters;

b) the position of the camera relative to the tool of the robot (“

hand-eye”

calibration);

c) the position of the camera in a space rigid to the place where the object will be trained (“

Training Space”

);

ii) teaching the object features by

a) putting the object in the “

Training Space” and

capturing an image of the object with the robot in the calibration position where the “

Camera→

Training Space”

transformation was calculated;

b) selecting at least 6 visible features from the image;

c) calculating the 3D position of each feature in “

Training Space”

;

d) defining an “

Object Space”

aligned with the “

Training Space”

but connected to the object and transposing the 3D coordinates of the features into the “

Object Space”

;

e) computing the “

Object Space→

Camera”

transformation using the 3D position of the features inside the “

Object Space” and

the positions of the features in the image;

f) defining an “

Object Frame”

inside “

Object Space”

to be used for teaching the intended operation path;

g) computing the Object Frame position and orientation in “

Tool Frame”

using the transformation from “

Object Frame→

Camera” and

“

Camera→

Tool”

;

h) sending the “

Object Frame”

to the robot;

i) training the intended operation path relative to the “

Object Frame”

using the robot;

iii) carrying out object finding and positioning by

a) positioning the robot in a predefined position above the bin containing the object and capturing an image of the object;

b) if an insufficient number of selected features are in the field of view, moving the robot until at least 6 features can be located;

c) with the positions of features from the image and their corresponding positions in “

Object Space”

as calculated in the training step, computing the object location as the transformation between the “

Object Space” and

“

Camera Space”

;

d) using the said transformation to calculate the movement of the robot to position the camera so that it appears orthogonal to the object;

e) moving the robot to the position calculated in step d);

f) finding the “

Object Space→

Camera Space”

transformation in the same way as in step c);

g) computing the object frame memorized at training using the found transformation and “

Camera→

Tool”

transformation;

h) sending the computed “

Object Frame”

to the robot; and

i) using the “

Tool”

position to define the frame in “

Robot Space” and

performing the intended operation path on the object inside the “

Robot Space”

.

3. The method of claim 1 wherein said 6 visible features comprise at least one anchor feature and at least 5 other visible features, and the position and orientation of said anchor feature is used to compute the position of the remaining set of selected features.

4. The method of claim 3 wherein the position and orientation of the anchor features is first used to estimate the position of the remaining selected features, and the actual position of the remaining selected features is then searched for and determined.

9. The method of claim 1 wherein step ii) h) and i) are carried out by allowing the user to select the desired operation path points interactively and sending to the robot the coordinates of the selected points which are computed using the object'"'"'s current position and orientation.

11. The method of claim 1 wherein in step ii) i) the intended operation path points are sourced from an offline robot programming software application.

13. The method of claim 1 wherein the coordinates of the “

Object Frame”

points, operation path points or other points of interest are transformed using the transformation from tool to robot base and all the coordinates are sent to robot in the robot base coordinate frame.

16. The method of claim 1 wherein the determination of object location in step iii d) is carried out using any of the following algorithms:

i) 3D pose estimation using non linear optimization methods derived from the ones described in the following articles;

“

A Flexible New Technique for Camera Calibration”

by Zhengyou Zhang;

“

An Efficient and Accurate Camera Calibration Techniques for 3D Machine Vision”

by Roger Y. Tsai;

ii) 3D pose estimation from lines correspondence in which selected features are edges as described in “

Determination of Camera Location from 2D to 3D Line and Point Correspondences”

by Yucai Liu, Thomas S. Huang, Olivier D. Faugeras;

iii) pose estimation using “

orthogonal iteration”

described in “

Fast and Globally Convergent Pose Estimation from Video Images”

by Chien Ping Lu, Gregory D. Hager, Eric Mjolsness;

iv) approximate object location under weak perspective conditions as demonstrated in “

Uniqueness of 3D Pose Under Weak Perspective;

A Geometric Proof”

by Thomas Huang, Alfred Bruckenstein, Robert Holt, Arun Netravali;

or

v) approximate object location using Direct Linear Transformation (DLT) as described in “

An investigation on the accuracy of three-dimensional space reconstruction using Direct Linear Transformation techniques”

by Chen, Armstrong, Raftopoulos.

2. A method of three-dimensional handling of an object by a robot using a tool and one camera mounted on the robot, comprising:

i) calibrating the camera by finding

a) the camera intrinsic parameters;

b) the position of the camera relative to the tool of the robot (“

hand-eye”

calibration);

ii) teaching the object features by

a) putting the object in the field of view of the camera and capturing an image of the object;

b) selecting at least 6 visible features from the image;

c) calculating the 3D position in real world co-ordinates of said selected features inside a space connected to the object (“

Object Space”

);

d) computing the “

Object Space→

Camera”

transformation using the 3D position of the features inside this space and the position in the image;

e) defining an “

Object Frame”

inside “

Object Space”

to be used for teaching the handling path;

f) computing the “

Object Frame”

position and orientation in “

Tool Frame”

using the transformation from “

Object Frame→

Camera” and

“

Camera→

Tool”

;

g) sending the computed “

Object Frame”

to the robot; and

h) training the intended operation path inside the “

Object Frame”

;

iii) carrying out object finding and positioning by

a) positioning the robot in a predefined position above the bin containing the target object;

b) if an insufficient number of selected features are in the field of view, moving the robot until at least 6 features can be located;

c) with the positions of features from the image and their corresponding position in “

Object Space”

as calculated in the training session, computing the object location as the transformation between the “

Object Space” and

“

Camera Space”

;

d) using the said transformation to calculate the movement of the robot to position the camera so that it appears orthogonal to the object;

e) finding the “

Object Space→

Camera Space”

transformation in the same way as in step d);

f) computing the object frame memorized at training using the found transformation and “

Camera→

Tool”

transformation;

g) sending the computed “

Object Frame”

to the robot;

h) using the “

Tool”

position to define the frame in “

Robot Space” and

performing the intended operation path on the object inside the “

Robot Space”

.

5. The method of claim 2 wherein said 6 visible features comprise at least one anchor feature and at least 5 other visible features and the position and orientation of said anchor feature is used to compute the position of the remaining set of selected features.

6. The method of claim 5 wherein the position and orientation of the anchor features is first used to estimate the position of the remaining selected features, and the actual position of the remaining selected features is then searched for and determined.

7. The method of claim 2 wherein, in step ii) c) said 3D positions of features are provided using a CAD model of the object;

8. The method of claim 2 wherein, in step ii) c) said 3D positions of features is calculated automatically.

10. The method of claim 2 wherein step ii) g) and h) are carried out by allowing the user to select the desired operation path points interactively and sending to the robot the coordinates of the selected points which are computed using the object'"'"'s current position and orientation.

12. The method of claim 2 wherein in step ii) h) the intended operation path points are sourced from an offline robot programming software application.

14. The method of claim 2 wherein the coordinates of the “

Object Frame”

points, operation path points or other points of interest are transformed using the transformation from tool to robot base and all the coordinates are sent to robot in the robot base coordinate frame.

15. The method of claim 2 wherein steps i) a) and b) and ii) c) are eliminated by using a self calibration method of robotic eye and hand-eye relationship with model identification.

17. The method of claim 2 wherein the determination of object location in step iii d) is carried out using any of the following algorithms:

i) 3D pose estimation using non linear optimization methods derived from the ones described in the following articles;

“

A Flexible New Technique for Camera Calibration”

by Zhengyou Zhang;

“

An Efficient and Accurate Camera Calibration Techniques for 3D Machine Vision”

by Roger Y. Tsai;

ii) 3D pose estimation from lines correspondence in which selected features are edges as described in “

Determination of Camera Location from 2D to 3D Line and Point Correspondences”

by Yucai Liu, Thomas S. Huang, Olivier D. Faugeras;

iii) pose estimation using “

orthogonal iteration”

described in. “

Fast and Globally Convergent Pose Estimation from Video Images”

by Chien Ping Lu, Gregory D. Hager, Eric Mjolsness;

iv) approximate object location under weak perspective conditions as demonstrated in “

Uniqueness of 3D Pose Under Weak Perspective;

A Geometric Proof”

by Thomas Huang, Alfred Bruckenstein, Robert Holt, Arun Netravali;

or

v) approximate object location using Direct Linear Transformation (DLT) as described in “

An investigation on the accuracy of three-dimensional space reconstruction using Direct Linear Transformation techniques”

by Chen, Armstrong, Raftopoulos.