Method and apparatus for machine-vision

• US 7,336,814 B2
• Filed: 07/14/2005
• Issued: 02/26/2008
• Est. Priority Date: 07/14/2004
• Status: Active Grant

• Current Assignee: Roboticvisiontech LLC
• Original Assignee: Braintech Canada, Inc.
• Inventors: Sameti, Mohammad, Pescaru, Simona, Habibi, Babak, Boca, Remus F
• Primary Examiner(s): LE, BRIAN Q
• Application Number:
  US11/183,228
• Publication Number:
  US 20060088203A1
• Time in Patent Office: 957 Days
• Field of Search: 382141-153, 382155-160
• US Class Current: 382/141
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image

Abstract

A system and method facilitate machine-vision, for example three-dimensional pose estimation for target objects, using one or more images sensors to acquire images of the target object at one or more positions, and to identify features of the target object in the resulting images. A set of equations is set up exploiting invariant physical relationships between features such as constancy of distances, angles, and areas or volumes enclosed by or between features. The set of equations may be solved to estimate a 3D pose. The number of positions may be determined based on the number of image sensors, number of features identified, and/or number of known physical relationships between less than all features. Knowledge of physical relationships between image sensors and/or between features and image sensors may be employed. A robot path may be transformed based on the pose, to align the path with the target object.

Claims

1. A method useful in machine-vision of objects, the method comprising:

acquiring a number of images of a first view of a training object from a number of image sensors;

identifying a number of features of the training object in the acquired at least one image of the first view;

determining a number of additional views to be obtained based at least in part on the number of image sensors, the number of features identified, the number of features having an invariant physical relationship associated thereto, and a type of the invariant physical relationship associated with the features, sufficient to provide a system of equations and unknowns where the number of unknowns is not greater than the number of equations;

acquiring at least one image of each of the number of additional views of the training object by the at least one camera; and

identifying at least some of the number of features of the training object in the acquired at least one image of the number of additional views of the training object

employing at least one of a consistency of physical relationships between some of the identified features to set up the system of equations; and

automatically computationally solving the system of equations.

2. The method of claim 1 wherein employing at least one of a consistency of physical relationships between some of the identified features to set up the system of equations comprises employing at least one invariant distance between at least two of the features.

3. The method of claim 1 wherein employing at least one of a consistency of physical relationships between some of the identified features to set up the system of equations comprises employing at least one invariant area enclosed between or by at least one of the features.

4. The method of claim 1 wherein employing at least one of a consistency of physical relationships between some of the identified features to set up the system of equations comprises employing at least one invariant volume enclosed between or by at least one of the features.

5. The method of claim 1, wherein

determining a number of additional views to be obtained includes computationally solving the equation m≧

(f²−

f−

2k−

2r +6(c−

ck))/(f²−

3f)−

1, where m is the number of views, f the number of features, k the number of known distances between pairs of the features, r is the number of rays with a known distance between a feature and an image sensor, c is the number of image sensors and ck is the number of known transformation between an imager sensor reference frame and a common reference frame.

6. The method of claim 1 wherein automatically computationally solving the system of equations includes automatically computationally solving the system of equations to determine a three-dimensional pose estimation.

7. A machine-vision system, comprising:

at least one image sensor operable to acquire images of a training object and of target objects;

processor-readable medium storing instructions for facilitating machine-vision for objects having invariant physical relationships between a number of features on the objects, by;

acquiring a number of images of a first view of a training object from a number of image sensors;

identifying a number of features of the training object in the acquired at least one image of the first view;

determining a number of additional views to be obtained based at least in part on the number of image sensors, the number of features identified, the number of features having an invariant physical relationship associated thereto, and a type of the invariant physical relationship associated with the features, sufficient to provide a system of equations and unknowns where the number of unknowns is not greater than the number of equations;

employing at least one of a consistency of physical relationships between some of the identified features to set up the system of equations; and

automatically computationally solving the system of equations; and

a processor coupled to receive acquired images from the at least one image sensor and operable to execute the instructions stored in the processor-readable medium.

8. The machine-vision system of claim 7 wherein the instructions cause the processor to employ at least one of a consistency of physical relationships between some of the identified features to set up the system of equations by employing at least one invariant distance between at least two of the features.

9. The machine-vision system of claim 7 wherein the instructions cause the processor to employ at least one of a consistency of physical relationships between some of the identified features to set up the system of equations by employing at least one invariant area enclosed between or by at least one of the features.

10. The machine-vision system of claim 7 wherein the instructions cause the processor to employ at least one of a consistency of physical relationships between some of the identified features to set up the system of equations by employing at least one invariant volume enclosed between or by at least one of the features.

11. The machine-vision system of claim 7 wherein determining a number of additional views to be obtained includes computationally solving the equation m≧

(f²−

f−

2k−

2r+6(c−

ck))/(f²−

3f)−

1, where m is the number of views, f the number of features, k the number of known distances between pairs of the features, r is the number of rays with a known distance between a feature and an image sensor, c is the number of image sensors and ck is the number of known transformation between an imager sensor reference frame and a common reference frame.

12. The machine-vision system of claim 7 wherein automatically computationally solving the system of equations includes automatically computationally solving the system of equations to determine a three-dimensional pose estimation.

13. A processor readable medium storing instructions for causing a processor to facilitate machine-vision for objects having invariant physical relationships between a number of features on the objects, by:

acquiring a number of images of a first view of a training object from a number of cameras;

identifying a number of features of the training object in the acquired at least one image of the first view;

determining a number of additional views to be obtained based at least in part on the number of image sensors, the number of features identified, the number of features having an invariant physical relationship associated thereto, and a type of the invariant physical relationship associated with the features, sufficient to provide a system of equations and unknowns where the number of unknowns is not greater than the number of equations;

employing at least one of a consistency of physical relationships between some of the identified features to set up the system of equations; and

automatically computationally solving the system of equations.

14. The processor readable medium of claim 13 wherein the instructions cause the processor to employ at least one of a consistency of physical relationships between some of the identified features to set up the system of equations by employing at least one invariant distance between at least two of the features.

15. The processor readable medium of claim 13 wherein the instructions cause the processor to employ at least one of a consistency of physical relationships between some of the identified features to set up the system of equations by employing at least one invariant area enclosed between or by at least one of the features.

16. The processor readable medium of claim 13 wherein the instructions cause the processor to employ at least one of a consistency of physical relationships between some of the identified features to set up the system of equations by employing at least one invariant volume enclosed between or by at least one of the features.

17. The processor readable medium of claim 13 wherein determining a number of additional views to be obtained includes computationally solving the equation m≧

(f²−

f−

2k−

2r+6(c−

ck))/(f²−

3f)−

1where m is the number of views, f the number of features, k the number of known distances between pairs of the features, r is the number of rays with a known distance between a feature and an image sensor, c is the number of image sensors and ck is the number of known transformation between an imager sensor reference frame and a common reference frame.

18. The processor readable medium of claim 13 wherein automatically computationally solving the system of equations includes automatically computationally solving the system of equations to determine a three-dimensional pose estimation.

19. A method useful in machine-vision of objects, the method comprising:

acquiring a number of images of a first view of a training object from a number of cameras;

identifying a number of features of the training object in the acquired at least one image of the first view;

associating parameters to less than all of the identified features which parameters define an invariant physical relationship between either the feature and at least one other feature, the feature and the at least one camera, or between the at least one camera and at least another camera where an invariant physical relationship between each one of the features and at least one other feature is not known when associating the parameters before a runtime;

determining a number of additional views to be obtained based at least in part on the number of cameras, the number of features identified, and the number of features having parameters associated thereto, sufficient to provide a system of equations and unknowns where the number of unknowns is not greater than the number of equations; and

acquiring at least one image of each of the number of additional views of the training object by the at least one camera;

identifying at least some of the number of features of the training object in the acquired at least one image of the number of additional views of the training object.

20. The method of claim 19 wherein determining a number of additional views to be obtained comprises computationally solving the equation m≧

(f²−

f−

2k−

2r+6(c−

ck))/(f²−

3f)−

1, where m is the number of views, f the number of features, k the number of known distances between pairs of the features, r is the number of rays with a known distance between a feature and an image sensor, c is the number of image sensors and ck is the number of known transformation between an imager sensor reference frame and a common reference frame.

21. The method of claim 19, further comprising:

simultaneously solving the system of equations to determine three-dimensional poses for at least some of the features in respective camera coordinate reference frames.

22. The method of claim 21, further comprising:

acquiring at least one image of a view of a target object by the at least one image sensor during the runtime;

identifying at least some of the features in the acquired at least one image of the view of the target object that were previously identified in the at least one image of the view of the training object; and

determining a three-dimensional object pose based at least in part on the determined three-dimensional poses for at least some of the features in respective camera coordinate reference frames.

23. A method useful in machine-vision for objects having invariant physical relationships between a number of features on the objects, the method comprising:

in a pre-runtime environment;

acquiring at least one image of a first view of a training object by at least one image sensor;

identifying a number of features of the training object in the acquired at least one image of the first view; and

associating a number of parameters to less than all of the identified features which define an invariant physical relationship between the either the feature and at least one other feature or between the feature and the at least one image sensor;

determining a number of additional views to be obtained based at least in part on the number of image sensors acquiring at least one image, the number of features of the training object identified, the number of features having parameters associated therewith, and a type of invariant physical relationship associated with each of the parameter;

acquiring at least one image of a second view of the training object by the at least one image sensor; and

identifying at least some of the number of features of the training object in the acquired at least one image of the second view; and

in at least one of a pre-run time environment or a runtime environment, computationally determining a local model using the identified features in each of a number of respective image sensor coordinate frames.

24. The method of claim 23 wherein determining a number of additional views to be obtained comprises computationally solving the equation m≧

(f²−

f−

2k−

2r+6(c−

ck))/(f²−

3f)−

1, where m is the number of views, f the number of features, k the number of known distances between pairs of the features, r is the number of rays with a known distance between a feature and an image sensor, c is the number of image sensors and ck is the number of known transformation between an imager sensor reference frame and a common reference frame.

25. The method of claim 23 wherein acquiring at least one image of a first view of a training object by at least one image sensor comprises acquiring the image of the object with a first position and a first orientation with respect to the first image sensor, and wherein acquiring at least one image of a second view of the training object by the at least one image sensor comprises acquiring the image of the object with at least one of a second position or a second orientation with respect to the first image sensor.

26. The method of claim 23 wherein identifying a number of features of the training object in the acquired at least one image of the first view comprises identifying at least one of a number of inherent structural features of the training object or a number of markers added to the training object.

27. The method of claim 23 wherein computationally determining a local model using the identified features in each of a number of respective image sensor coordinate frames comprises determining a pose for each of the identified features in a respective one of the acquired images the respective image sensor coordinate frame.

28. The method of claim 23, further comprising:

pose estimating in a runtime environment, by;

acquiring at least one image of a view of a target object by the at least one image sensor;

identifying at least some of the features in the acquired at least one image of the view of the target object that were previously identified in the at least one image of the view of the training object;

computationally determining a reference pose of each of the local models; and

computationally determining a pose of each of the local models in a common reference coordinate system.

29. The method of claim 28 wherein the pose estimating in the runtime environment is further by computationally determining a transformation between the pose of the target object and the pose of the training object by simultaneously solving a system of a number of equations and a number of unknowns, where the number of equations is greater than or equal to the number of unknowns.

30. The method of claim 23, further comprising:

calibrating the at least one image sensor in the pre-runtime environment.

31. The method of claim 30 wherein calibrating the at least one image sensor comprises obtaining a set of intrinsic parameters for the at least one image sensor.

32. The method of claim 30 wherein calibrating the at least one image sensor comprises determining a pose of the at least one image sensor with reference to a robot coordinate system.

33. A machine-vision system, comprising:

at least one image sensor operable to acquire images of a training object and of target objects;

a processor-readable medium storing instructions for facilitating pose estimation for objects having invariant physical relationships between a number of features on the objects, by;

in a pre-runtime environment;

acquiring at least one image of a first view of a training object by at least one image sensor;

identifying a number of features of the training object in the acquired at least one image of the first view; and

associating a number of parameters to less than all of the identified features which define an invariant physical relationship between the either the feature and at least one other feature or between the feature and the at least one image sensor;

determining a number of additional views to be obtained based at least in part on the number of image sensors acquiring at least one image, the number of features of the training object identified, the number of features having parameters associated therewith, and a type of invariant physical relationship associated with each of the parameter;

acquiring at least one image of a second view of the training object by the at least one image sensor; and

identifying at least some of the number of features of the training object in the acquired at least one image of the second view; and

in at least one of a pre-run time environment or a runtime environment, computationally determining a local model using the identified features in each of a number of respective image sensor coordinate frames; and

a processor coupled to receive acquired images from the at least one image sensor and operable to execute the instructions stored in the processor-readable medium.

34. The machine-vision system of claim 33 wherein the at least one image sensor comprises one of a video camera or a digital still camera.

35. A processor readable medium storing instructions for causing a processor to facilitate machine-vision for objects having invariant physical relationships between a number of features on the objects, by:

in a pre-runtime environment;

acquiring at least one image of a first view of a training object by at least one image sensor;

identifying a number of features of the training object in the acquired at least one image of the first view; and

associating parameters to less than all of the identified features which define a physical relationship between the either the feature and at least one other feature or between the feature and the at least one image sensor; and

determining a number of additional views to be obtained based at least in part on the number of image sensors acquiring at least one image and the number of features of the training object identified;

acquiring at least one image of a second view of the training object by the at least one image sensor; and

identifying at least some of the number of features of the training object in the acquired at least one image of the second view; and

in at least one of a pre-run time environment or a runtime environment, computationally determining a local model using the identified features in each of a number of respective image sensor coordinate frames.

36. The processor-readable medium of claim 35 wherein the instructions cause the processor to determine the number of additional views to be obtained by computationally solving the equation m≧

(f²−

f−

2k−

2r+6(c−

ck))/(f²−

3f)−

1, where m is the number of views, f the number of features, k the number of known distances between pairs of the features, r is the number of rays with a known distance between a feature and an image sensor, c is the number of image sensors and ck is the number of known transformation between an imager sensor reference frame and a common reference frame.

37. The processor-readable medium of claim 35 wherein the instructions cause the processor to computationally determine the local model using the identified features in each of the number of respective image sensor coordinate frames by determining a pose for each of the identified features in a respective one of the acquired images the respective image sensor coordinate frame.

38. The processor-readable medium of claim 35 wherein the instructions cause the processor to facilitate machine-vision, further by:

pose estimating in a runtime environment, by;

acquiring at least one image of a view of a target object by the at least one image sensor;

identifying at least some of the features in the acquired at least one image of the view of the target object that were previously identified in the at least one image of the view of the training object;

computationally determining a reference pose of each of the local models;

computationally determining a pose of each of the local models in a common reference coordinate system; and

computationally determining a transformation between the pose of the target object and the pose of the training object.

39. The processor-readable medium of claim 38 wherein the instructions cause the processor to facilitate pose estimation, further by:

calibrating the at least one image sensor in the pre-runtime environment.