Method and system for automatically determining the position and orientation of an object in 3-D space

• US 4,942,539 A
• Filed: 12/21/1988
• Issued: 07/17/1990
• Est. Priority Date: 12/21/1988
• Status: Expired due to Term

• Current Assignee: Fanuc Robotics America Incorporated (Fanuc Corporation)
• Original Assignee: GMF Robotics Corporation
• Inventors: Coldren, Bruce E., Krause, Kenneth W., McGee, H. Dean
• Primary Examiner(s): MacDonald, Allen
• Application Number:
  US07/287,885
• Time in Patent Office: 573 Days
• Field of Search: 364/513, 364/516, 364/559, 901/47, 358/101, 358/107, 356/2, 356/375
• US Class Current: 700/253
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G01S 5/163 Determination of attitude using inertial means G01C9/00; control of attitude G05D1/49
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

A method and system for automatically determining the position and orientation of an object by utilizing as few as a single digital image generated by as few as a single camera without the use of a structured light. The digital image contains at least three non-colinear geometric features of the object. The three features may be either coplanar or non-coplanar. The features or targets are viewed such that perspective information is present in the digital image. In a single camera system the geometric features are points, and in a multi-camera system, the features are typically combinations of points and lines. The location of the features are determined and processed within a programmed computer together with reference data and camera calibration data to provide at least three non-parallel 3-D lines. The 3-D lines are utilized by an iterative algorithm to obtain data relating to the position and orientation of the object in 3-D space. The resultant data is subsequently utilized to calculate an offset of the object from the camera. The offset is then transformed into the coordinate system or frame of a peripheral device such as a robot, programmable controller, numerically controlled machine, etc. Finally, the programmed computer transfers the transformed offset to the peripheral device which utilizes the transformed offset to modify its preprogrammed path.

Claims

1. Method for automatically determining the position and orientation of an object in 3-D space from as few as one digital image generated by as few as one 2-D sensor, the method comprising the steps of:

generating sensor calibration data relating the position and orientation of the 2-D sensor to the 3-D space;

generating reference data relating to at least three non-collinear geometric features of an ideal object;

generating the digital image containing the at least three-non-collinear geometric features of the object wherein at least two of the features reside in the image generated by the one 2-D sensor;

locating each of the features in the digital image;

computing at least three non-parallel 3-D lines as a function of the feature locations and the sensor calibration data, each of the 3-D lines passing through its respective feature of the object at a reference point;

determining a starting point on each of the 3-D lines in the vicinity of its associated reference point;

determining the actual 3-D location of each feature on its associated 3-D line utilizing the reference data and its respective starting point on the 3-D line; and

utilizing the reference data and the actuaI 3-D location of each of the features to determine the position and orientation of the object.

2. The method of claim 1 wherein the step of generating the digital image includes the step of:

generating first and second digital sub-images, the first and second sub-images defining the digital image.

3. The method of claim 2 wherein the method utilizes first and second 2-D sensors to generate the first and second digital sub-images, respectively, wherein the method further comprises the step of generating sensor calibration data relating the position and orientation of the second 2-D sensor to the 3-D space and wherein the at least three 3-D lines are also a function of the calibration data of the second 2-D sensor.

4. The method of claim 1 wherein each of the at least three features of the ideal object is a point and wherein the 3-D lines intersect at a common point.

5. The method of claim 1 wherein the image is a gray scale digital image.

6. The method of claim 4 wherein the 2-D sensor is a camera having a lens system with an optical center and wherein the common point is the optical center of the lens system.

7. The method of claim 1 wherein the object is a rigid body.

8. The method of claim 1 wherein the 2-D sensor is a camera.

9. The method of claim 1 wherein said step of determining a starting point on each of the 3-D lines includes the step of generating at least three initial position estimates of the starting points as a function of the locations of the features of the ideal object and each of said 3-D lines.

10. The method of claim 9 wherein said step of determining the actual 3-D location of each feature includes the step of computing the distances between pairs of the initial position estimates and between pairs of the features of the object.

11. The method of claim 10 wherein said step of determining the actual 3-D location of each feature includes the step of minimizing the differences between the distances between the pairs of the initial position estimates and the distances between the pairs of the features of the objects.

12. Method for automatically generating offset data for use by a programmable peripheral device having a coordinate frame such as a robot to move to an object in 3-D space, the method utilizing as few as one digital image generated by as few as one 2-D sensor, the method comprising the steps of:

generating sensor calibration data relating the position and orientation of the 2-D sensor to the 3-D space;

generating reference data relating to at least three non-collinear geometric features of an ideal object;

generating the digital image containing the at least three non-collinear geometric features of the object wherein at least two of the features reside in the image generated by the one 2-D sensor;

locating each of the features in the digital image;

computing at least three non-parallel 3-D lines as a function of the feature locations and the sensor calibration data, each of the 3-D lines passing through its respective feature of the object at a reference point;

determining a starting point on each of the 3-D lines in the vicinity of its assoicated reference point;

determining the actual 3-D location of each feature on its associated 3-D line utilizing the reference data and its respective starting point on the 3-D line;

utilizing the reference data and the actual 3-D location of each of the features to determine the position and orientation of the object;

calculating an offset of the object from the 2-D sensor as a function of the position and orientation of the object; and

transforming the offset of the object to the coordinate frame of the peripheral device.

13. The method of claim 12 wherein the step of generating the digital image includes the step of:

generating first and second digital sub-images, the first and second sub-images defining the digital image.

14. The method of claim 13 wherein the method utilizes first and second 2-D sensors to generate the fist and second digital sub-images, respectively, wherein the method further comprises the step of generating sensor calibration data relating the position and orientation of the second 2-D sensor to the 3-D space and wherein the at least three 3-D lines are also a function of the calibration data of the second 2-D sensor.

15. The method of claim 12 wherein each of the at least three features of the ideal object is a point and wherein the 3-D lines intersect at a common point.

16. The method of claim 15 wherein the 2-D sensor is a camera and wherein the common point is the optical center of the lens system.

17. The method of claim 12 wherein the object is a rigid body.

18. The method of claim 12 wherein the 2-D sensor is a camera.

19. The method of claim 12 wherein said step of determining a starting point on each of the 3-D lines includes the step of generating at least three initial position estimates of the starting points as a function of the locations of the features of the ideal object and each of said 3-D lines.

20. The method of claim 19 wherein said step of determining the actual 3-D location of each feature includes the step of generating at least three initial position estimates as a function of the locations of the features of the ideal object and each of said 3-D lines.

21. The method of claim 20 wherein said step of determining the actual 3-D location of each feature includes the step of computing the distances between pairs of the initial position estimates and between pairs of the features of the object.

22. A vision system for automatically determining the position and orientation of an object in 3-D D space from as few as one digital image generated by as few as one 2-D sensor, the system comprising:

first means for storing reference data relating to at least three non-collinear geometric features of an ideal object;

2-D sensor means for generating the digital image containing the at least three non-collinear geometric features of the object, the 2-D sensor means including the one 2-D sensor wherein at least two of the features reside in the image generated by the one 2-D sensor;

second means for storing sensor calibration data relating the position and orientation of the 2-D sensor means to the 3-D space;

means for locating each of the features in the digital image;

means for computing at least three non-parallel 3-D lines as a function of the feature locations and the sensor calibration data, each of the 3-D lines passing through its respective feature of the object at a reference point;

determining a starting point on each of the 3-D lines in the vicintiy of its associated reference point;

determining the actual 3-D location of each feature on its associated 3-D line utilizing the reference data and its respective starting point on the 3-D line; and

means for utilzing the reference data and the actual 3-D locations of each of the features to determine the position and orientation of the object.

23. The system of claim 22 wherein said 2-D sensor means includes first and second 2-D sensors for generating first and second digital sub-images, the first and second sub-images defining the digital image.

24. The system of claim 23 wherein said second means for storing stores sensor calibration data relating the position and orientation of the first and second 2-D sensors to the 3-D space and wherein the at least three 3-D lines are a function of the calibration data of the first and second 2-D sensors.

25. The system of claim 22 wherein each of the at least three features of the ideal object is a point and wherein the 3-D lines intersect at a common point.

26. The system of claim 25 wherein the 2-D sensor is a camera having a lens system with an optical center and wherein the common point is the optical center of the lens system.

27. The system of claim 22 wherein the object is a rigid body.

28. The system of claim 22 wherein the 2-D sensor is a camera.

29. The system of claim 22 wherein said means for determining a starting point includes position estimating means for generating at least three initial position estimates of the starting points as a function of the locations of the features of the ideal object and each of said 3-D lines.

30. The system of claim 29 wherein said means for determining the actual 3-D location of each feature includes means for computing the distances between piars of the initial position estimates and between pairs of the features of the object.

31. The system as claimed in claim 30 wherein said means for determining the actual 3-D location of each feature includes means for minimizng the differences between the distances between the pairs of the initial position estimates and the distances between the pairs of the features of the object.

32. A vision system for automatically generating offset data for use by a programmble peripheral device having a coordinate frame such as a robot to enable the peripheral device to move to an object in 3-D space from as few as one digital image generated by as few as one 2-D sensor, the system comprising:

first means for storing sensor calibration data relating to at least three non-collinear geometric features of an ideal object;

2-D sensor means for generating the digital image containing the at least three non-collinear geometric features of the object, the 2-D sensor means including the one 2-D sensor wherein at least two of the features reside in the image generated by the one 2-D sensor;

second means for storing sensor calibration data relating the position and orientation of the 2-D sensor means to the 3-D space;

means for locating each of the features in the digital image;

means for computing at least three non-parallel 3-D lines as a function of the feature locations and the sensor calibration data, each of the 3-D lines passing through its respective features of the object at a reference point;

determining a starting point on each of the 3-D lines in the vicinity of its associated reference point;

determining the actual 3-D location of each feature on its associated 3-D line utilizing the reference data and its respective starting point on the 3-D line;

means for utilizing the reference data and the actual 3-D locations of each of the features to determine the position and orientation of the object;

means for calculating an offset of the object from the 2-D sensor means as a function of the position and orientation of the object; and

means for transforming the offset of the object to the coordinate frame of the peripheral device.

33. The system of claim 32 wherein said 2-D sensor means includes first and second 2-D sensors for generating first and second digital sub-images, the first and second sub-images defining the digital image.

34. The system of claim 33 wherein said second means for storing stores sensor calibration data relating the position and orientation of the first and second 2-D sensors to the 3-D space and wherein the at least three 3-D lines are a function of the calibration data of the first and second 2-D sensors.

35. The system of claim 32 wherein each of the at least three features of the ideal object is a point and wherein the 3-D lines intersect at a common point.

36. The system of claim 35 wherein the 2-D sensor is a camera having a lens system with an optical canter and wherein the common point is the optical canter of the lens system.

37. The system of claim 32 wherein the object is a rigid body.

38. The system of claim 32 wherein the 2-D sensor is a camera.

39. The system of claim 32 wherein said means for determining a starting point includes position estimating means for generating at least three initial position estimates of the starting points as a function of the locations of the features of the ideal object and each said 3-D lines.

40. The system of claim 39 wherein said means for determining the actual 3-D location of each feature includes means for computing the distances between pairs of the initial position estimates and between pairs of the features of the object.

41. The system of claim 40 wherein said means for determining the actual 3-D location of each feature includes means for minimizing the differences between the distances between the pairs of the initial position estimates and the distances between the pairs of the features of the object.