Method and apparatus for single image 3D vision guided robotics

• US 8,095,237 B2
• Filed: 08/06/2003
• Issued: 01/10/2012
• Est. Priority Date: 01/31/2002
• Status: Expired due to Term

• Current Assignee: Roboticvisiontech LLC
• Original Assignee: Roboticvisiontech LLC
• Inventors: Habibi, Babak, Pescaru, Simona, Sameti, Mohammad, Boca, Remus Florinel
• Primary Examiner(s): Trammell, James
• Assistant Examiner(s): Marc, McDieunel
• Application Number:
  US10/634,874
• Publication Number:
  US 20040172164A1
• Time in Patent Office: 3,079 Days
• Field of Search: 348/287, 348/291, 348/154, 348/190, 348/42, 348/94, 348/552, 348/191, 901/14, 901/17, 901 46- 47, 901/6, 414/730, 414/737, 382/154, 382/103, 382/106, 382/276, 382/293, 382/295, 382/298, 700245-246, 700257-259, 700/253, 700/279, 294/070.4, 297/14, 294/071, 297/02, 297/20, 292/18, 318/568.15, 318/568.13, 318/568.16, 318/640
• US Class Current: 700/245
• CPC Class Codes: B25J 9/1692 Calibration of manipulator
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/39057 Hand eye calibration, eye, camera on hand, end effector
• CPC Class Codes: G05B 2219/39393 Camera detects projected image, compare with reference image, position end effector
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: Y10T 29/53009 with comparator
• CPC Class Codes: Y10T 29/53061 Responsive to work or work-related machine element
• CPC Class Codes: Y10T 29/53087 with signal, scale, illuminator, or optical viewer

Abstract

A method of three-dimensional object location and guidance to allow robotic manipulation of an object with variable position and orientation using a sensor array which is a collection of one or more sensors capable of forming a single image.

Claims

1. A useful in three-dimensional pose estimation for use with a single camera mounted to a movable portion of a robot, the method comprising:

capturing a two-dimensional image of a volume containing a target object;

locating a number of features in the captured image of the target object; and

determining by a processor an object space-to-camera space transformation for the target object based at least in part on a position of at least some of the located features using only the single captured image and an algorithm that employs a known or determinable physical relationship between at least some of the located features.

2. The method of claim 1, further comprising:

capturing a number of images of a calibration object by the camera;

determining a set of intrinsic parameters of the camera from at least one of the number of images of the calibration object captured by the camera; and

determining a set of extrinsic parameters of the camera from at least one of the number of images of the calibration object captured by the camera, the set of extrinsic parameters comprising a camera space-to-training space transformation defining a transformation between a camera space reference frame and a training space reference frame.

3. The method of claim 2, further comprising:

positioning the camera with respect to the calibration object.

4. The method of claim 3 wherein positioning the camera with respect to the calibration object comprises positioning the camera orthogonally with respect to a ruled template with a number of features, where a known or determinable physical relationship exists between at least some of the features.

5. The method of claim 3 wherein positioning the camera with respect to the calibration object comprises positioning the camera with respect to a sample of a type of object the robot will manipulate, the sample having a number of features, where a known or determinable physical relationship exists between at least some of the features.

6. The method of claim 3 wherein capturing a number of images of a calibration object by the camera comprises capturing at least one image at each of a plurality of positions spaced perpendicularly from the calibration object.

7. The method of claim 3 wherein capturing a number of images of a calibration object by the camera comprises capturing at least one image at each of a plurality of different orientations with respect to the calibration object.

8. The method of claim 2 wherein determining a set of intrinsic parameters of the camera from the at least one of the number of images of the calibration object captured by the camera comprises determining at least one of a focal length, a first order radial lens distortion coefficient, a set of coordinates of a center of a radial lens distortion, or a scale factor indicative of a framegrabber scanline resampling uncertainty.

9. The method of claim 2 wherein determining a set of extrinsic parameters of the camera from at least one of the number of images of the calibration object captured by the camera, the set of extrinsic parameters comprising a camera space-to-training space transformation defining a transformation between a camera space reference frame and a training space reference frame comprises determining a respective translation component along three orthogonal axes, and a respective rotation component about the three orthogonal axes.

10. The method of claim 2, further comprising:

determining a camera space-to-tool space transformation based at least in part on at least two of the number of images captured by the camera of the calibration object.

11. The method of claim 2, further comprising:

determining a camera space-to-tool space transformation based on single one of the number of images captured by the camera of the calibration object and on a number of physical coordinates of at least one feature of the calibration object.

12. The method of claim 1, further comprising:

capturing an image of a teaching object of a type of object that will be manipulated by the robot;

selecting a number of features from the captured image of the teaching object; and

determining a set of object space coordinates for each of the selected features from the captured image of the teaching object.

13. The method of claim 12 wherein selecting a number of features from the captured image of the teaching object comprises selecting six features from the captured image of the teaching object.

14. The method of claim 12, further comprising:

determining an object space-to-camera space transformation defining a transformation between an object space reference frame and the camera space reference frame.

15. The method of claim 14, further comprising:

determining a position and an orientation of an object frame in the tool frame reference frame based at least in part on the object frame-to-camera space and camera space-to-tool space transformations.

16. The method of claim 15, further comprising:

providing the position and orientation of the object frame to the robot; and

training an intended operation path inside the object frame.

17. The method of claim 1, further comprising:

adjusting a position of the movable portion of the robot if the number of features located in the captured image of the target object is determined to be an insufficient number of features; and

capturing another two-dimensional image of the volume containing the target object before determining the object space-to-camera space transformation for the target object.

18. The method of claim 1, further comprising:

determining at least one movement of the robot that orients the camera orthogonally with respect to the target object based at least on part on the object space-to-camera space transformation.

19. The method of claim 18, further comprising:

determining a position of the object frame in the tool space reference frame; and

providing an object frame to the robot.

20. An apparatus useful in robotics, the apparatus comprising:

a single camera operable to capture a number of images of a calibration object;

means for calibrating the camera, by;

determining a set of intrinsic parameters of the camera from at least one of the number of images of the calibration object captured by the camera; and

determining a set of extrinsic parameters of the camera from at least one of the number of images of the calibration object captured by the camera, the set of extrinsic parameters comprising a camera space-to-training space transformation defining a transformation between a camera space reference frame and a training space reference frame; and

means for estimating a pose of a target object, by;

capturing a two-dimensional image of a volume containing a target object; and

locating at least six features in the captured image of the target object; and

determining an object space-to-camera space transformation based at least in part on a position of at least some of the located features in solely the captured image using an algorithm that employs a known or determinable physical relationship between at least some of the located features.

21. The apparatus of claim 20, further comprising:

means for training, comprising;

capturing an image of a teaching object of a type of object that will be manipulated by the robot;

selecting a number of features from the captured image of the teaching object;

determining a set of object space coordinates for each of the selected features from the captured image of the teaching object; and

determining an object space-to-camera space transformation defining a transformation between an object space reference frame and the camera space reference frame.

22. The apparatus of claim 21 wherein the means for calibrating, the means for estimating a pose, and the means for training comprises at least one programmed computer.

23. The apparatus of claim 21 wherein the means for calibrating, the means for estimating a pose, and the means for training comprises at least one computer-readable medium storing instructions operating at least one computer.

24. The apparatus of claim 20 wherein the pose estimating means estimates the pose of the target object further by:

adjusting a position of the movable portion of the robot if the number of features located in the captured image of the target object is determined to be an insufficient number of features.

25. An apparatus useful in robotics, the apparatus comprising:

a single camera operable to capture a number of images of a calibration object;

means for calibrating the camera, by;

determining a set of intrinsic parameters of the camera from at least one of the number of images of the calibration object captured by the camera; and

determining a set of extrinsic parameters of the camera from at least one of the number of images of the calibration object captured by the camera, the set of extrinsic parameters comprising a camera space-to-training space transformation defining a transformation between a camera space reference frame and a training space reference frame; and

means for estimating a pose of a target object, by;

capturing a two-dimensional image of a volume containing a target object;

locating at least five features in the captured image of the target object; and

determining an object space-to-camera space transformation based at least in part on a position of at least some of the located features using the captured image without any additional captured images and an algorithm that employs a known or determinable physical relationship between at least some of the located features.

26. The apparatus of claim 25 wherein the means for calibrating and the means for estimating a pose comprises at least one programmed computer.

27. The apparatus of claim 25 wherein the means for calibrating and the means for estimating a pose comprises at least one computer-readable medium storing instructions operating at least one computer.

28. The apparatus of claim 25 wherein the pose estimating means estimates the pose of the target object further by:

adjusting a position of the movable portion of the robot if the number of features located in the captured image of the target object is determined to be an insufficient number of features.