Method and apparatus for multi-nodal, three-dimensional imaging

• US 6,915,008 B2
• Filed: 03/08/2002
• Issued: 07/05/2005
• Est. Priority Date: 03/08/2001
• Status: Active Grant

• Current Assignee: FLIR Commercial Systems Incorporated (Teledyne Technologies Incorporated)
• Original Assignee: Point Grey Research Inc (Teledyne Technologies Incorporated)
• Inventors: Murray, Don, Barman, Rod, Steenburgh, Malcolm, Ku, Shyan
• Primary Examiner(s): Ahmed, Samir
• Assistant Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US10/092,879
• Publication Number:
  US 20020167726A1
• Time in Patent Office: 1,215 Days
• Field of Search: 382/100, 382/151, 382/153, 382/154, 348/42, 348/47, 348/48, 345419-427, 356/12, 356/15, 356/611, 396/324, 396/325, 396331-333
• US Class Current: 382/154
• CPC Class Codes: G01B 11/00 Measuring arrangements characterised by the use of optical techniques
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A system for localizing objects has two or more measurement units. Calibration methods provide transformations to a common reference frame by locating calibration features in local reference frames of the measurement units. A calibration target spanning measurement fields of the measurement units may be used. The measurement units may have overlapping measurement fields. The measurement units may comprise stereo vision units or other units capable of locating objects or object features.

Claims

1. A method for calibrating a three-dimensional imaging system the method comprising:

providing first and second measurement units, the first measurement unit configured to acquire information from which locations of object features within a first measurement field can be determined in a first local coordinate system, the second measurement unit configured to acquire information from which locations of object features within a second measurement field can be determined in a second local coordinate system;

determining first locations in the first local coordinate system of a first plurality of calibration features located in the first measurement field, and, based on the first locations, determining locations in the first local coordinate system of a second plurality of calibration features located in the second measurement field;

determining locations of the second plurality of features in the second local coordinate system; and

,

deriving a transformation between the local coordinate systems and a common coordinate system based upon the locations of the second plurality of calibration features in the first and second local coordinate systems.

2. The method of claim 1 wherein the first and second measurement fields overlap within an overlap region and the first and second pluralities of calibration features are located within the overlap region.

3. The method of claim 2 wherein the first and second pluralities of calibration features each comprise the same calibration features.

4. The method of claim 1 wherein the first and second pluralities of calibration features are rigidly connected to one another.

5. The method of claim 1 wherein determining the locations in the first local coordinate system of the second plurality of calibration features comprises determining locations of each of the first and second pluralities of calibration features relative to one another using one or more additional measurement units having measurement fields overlapping with the first and second measurement fields.

6. A method for calibrating a three-dimensional imaging system the method comprising:

providing first and second measurement units, the first measurement unit configured to determine locations of object features within a first measurement field in a first local coordinate system, the second measurement unit configured to determine locations of object features within a second measurement field in a second local coordinate system, the first and second measurement fields overlapping within an overlap region;

determining locations of a plurality of calibration features located in the overlap region in each of the first and second local coordinate systems;

deriving a transformation between the local coordinate systems and a common coordinate system based upon the locations of the plurality of calibration features.

7. The method of claim 6 wherein the common coordinate system is the same as one of the first and second local coordinate systems.

8. The method of claim 6 wherein deriving a transformation between the local coordinate systems and a common coordinate system to comprises performing an iterative process.

9. The method of claim 8 wherein the iterative process comprises:

obtaining an estimated transformation,

applying the estimated transformation to the locations of the plurality of calibration features,

determining an error in the estimated transformation,

determining a correction based on the error, and

applying the correction to the estimated transformation.

10. The method of claim 9 wherein the iterative process comprises performing steps in a least-squares algorithm.

11. The method of claim 9 wherein the iterative process comprises applying different weightings to different calibration features.

12. The method of claim 8 wherein the iterative process comprises:

obtaining an estimated transformation;

correcting the estimated transformation;

identifying one or more redundant calibration features for which the estimated trasformations produces statistically outlying results; and

,

correcting the estimated transformation again without reference to the one or more redundant calibration features.

13. The method of claim 6 comprising:

providing an additional measurement unit having a measurement field overlapping with one of the first and second measurement fields in an additional overlap region, the additional measurement unit configured to determine locations of object features within its measurement field in an additional local coordinate system;

determining locations of a plurality of calibration features located in the additional overlap region in the additional local coordinate system;

deriving a transformation between the additional local coordinate system and a common coordinate system based upon the locations of the plurality of calibration features located in the additional overlap region.

14. The method of claim 6 wherein determining locations of a plurality of calibration features comprises determining locations of a plurality of features of a known calibration object.

15. The method of claim 6 wherein determining locations of a plurality of calibration features comprises:

determining a location of a calibration feature on a calibration object;

moving the calibration object;

determining a new location of the calibration feature; and

,

repeating moving the calibration object and determining a new location of the calibration feature until a desired number of new locations has been determined.

16. The method of claim 6 comprising determining in the common coordinate system a location of a first object feature located in the first measurement field and not located in the second measurement field and a location of a second object feature located in the second measurement field and not located in the first measurement field.

17. The method of claim 6 wherein the calibration features comprise a feature of a calibration target comprising a spherical object.

18. The method of claim 17 wherein determining locations of a plurality of calibration features comprises performing a circular Hough transform on image data from a plurality of imaging devices.

19. The method of claim 6 wherein the calibration features comprise a feature of a planar calibration target.

20. The method of claim 19 wherein the planar calibration target comprises a circular figure on a contrasting background.

21. The method of claim 20 wherein determining locations of a plurality of calibration features comprises performing a circular Hough transform on image data from a plurality of imaging devices.

22. A method for calibrating a three-dimensional imaging system the method comprising:

obtaining image data from a plurality of imaging devices of a first stereo measurement unit, the imaging devices having fields of view that overlap in a first stereo measurement field;

obtaining image data from a plurality of imaging devices of a second stereo measurement unit, the imaging devices having fields of view that overlap in a second stereo measurement field, the first and second stereo measurement fields overlapping one another in an overlap region containing at least one calibration target;

determining three-dimensional characteristics of at least one calibration target relative to the first stereo measurement unit using image data obtained from the imaging devices of the first stereo measurement unit;

determining three-dimensional characteristics of at least one calibration target relative to the second stereo measurement unit using image data obtained from the imaging devices of the second stereo measurement unit; and

,

deriving calibration parameters from the three-dimensional characteristics of the calibration targets relative to the first and second stereo measurement units, the calibration parameters comprising information about a position and orientation of the first stereo measurement units relative to the second stereo measurement unit.

23. A method according to claim 22 comprising deriving transformation functions based on the calibration parameters, the transformation functions providing a relationship between position data obtained from a pair of the imaging devices and a common coordinate system.

24. A method according to claim 23, wherein determining the three-dimensional characteristics of at least one calibration target comprises determining three-dimensional characteristics of a plurality of calibration features.

25. A method according to claim 23, wherein determining the three-dimensional characteristics of at least one calibration target comprises determining three-dimensional characteristics of at least one of:

circles, spheres, lines, and corners of a calibration object.

26. A method according to claim 23, comprising measuring three-dimensional characteristics of an object located in any of the first stereo measurement field and the second stereo measurement field using the calibration parameters and using image data obtained from at least two imaging devices selected from:

the imaging devices of the first stereo measurement unit and the imaging devices of the second stereo measurement unit.

27. A method according to claim 23, comprising:

obtaining image data from a plurality of imaging devices of each of one or more additional stereo measurement units, the imaging devices of each additional stereo measurement unit having fields of view that overlap one another in additional stereo measurement fields, at least one of the additional stereo measurement fields connected to at least one of the first and second stereo measurement fields by overlapping regions; and

,

deriving, from the image data obtained from the plurality of imaging devices of the additional stereo measurement units, calibration parameters comprising information about a position and orientation of each additional stereo measurement unit relative to a common coordinate system.

28. A method according to claim 24, wherein deriving each transformation function comprises starting with an estimated transformation function and then iteratively optimizing the transformation function to reduce differences between the three-dimensional characteristics of calibration targets in transformed image data and three-dimensional characteristics of the calibration targets in the common coordinate system.

29. A method according to claim 28, wherein optimizing the transformation function comprises selectively weighing three-dimensional characteristics of some of the one or more calibration targets.

30. Apparatus for determining the locations of object features in three dimensions, the apparatus comprising:

a first measurement unit configured to generate first data from which locations of object features in a first measurement field can be determined in a first local coordinate system;

a second measurement unit configured to generate second data from which locations of object features in a second measurement field can be determined in a second local coordinate system; and

,

a processor configured to;

receive the first and second data;

extract from the first data information about a first plurality of calibration features in the first measurement field; and

,

based on the calibration information, determine locations in the first local coordinate system of a second plurality of calibration features in the second measurement field;

determine from the second data locations of the second plurality of calibration features in the second local coordinate system; and

, based on the locations of the second plurality of calibration features in the first and second local coordinate systems, generate a transformation which, when applied to the second data, expresses locations of object features in a common coordinate system.

31. The apparatus of claim 30 wherein the first and second measurement fields overlap in an overlap region and the second plurality of calibration features is in the overlap region.

32. The apparatus of claim 30 wherein the first and second pluralities of calibration features each comprise the same calibration features.

33. Apparatus for determining the locations of object features in three dimensions, the apparatus comprising:

a first measurement unit configured to generate first data from which locations of object features in a first measurement field can be determined in a first local coordinate system;

a second measurement unit configured to generate second data from which locations of object features in a second measurement field can be determined in a second local coordinate system, the second measurement unit located so that the first and second measurement fields overlap in an overlap region; and

,

a processor configured to;

receive the first and second data;

extract from each of the first and second data calibration information about a plurality of calibration features of one or more calibration objects located in the overlap region; and

,

based on the calibration information, generate a transformation which, when applied to the second data expresses locations of object features in a common coordinate system.

34. The apparatus of claim 33 wherein the first and second measurement fields have substantially parallel axes of symmetry.

35. The apparatus of claim 33 wherein the first and second measurement fields cross one another at an angle.

36. The apparatus of claim 33 wherein each of the measurement units comprises two or more spaced-apart imaging devices and the imaging devices of each of the measurement units have fields of view which overlap with one another in the corresponding measurement field.

37. The apparatus of claim 36 wherein each of the imaging devices has an optical axis and the optical axes of the plurality of imaging devices are substantially parallel to one another.

38. The apparatus of claim 33 wherein the processor is configured to perform a model-based recognition algorithm on the first and second data.

39. The apparatus of claim 33 wherein the processor is configured to perform a circular Hough transform on the first and second data to locate features of circular calibration targets.

40. The apparatus of claim 33 comprising a plurality of additional measurement units having corresponding additional measurement fields each connected to at least one of:

the first measurement field, the second measurement field and another additional measurement field, by way of one or more overlapping regions.

41. Apparatus for determining the locations of object features in three dimensions, the apparatus comprising:

a first stereo measurement unit comprising a plurality of imaging devices, the imaging devices having fields of view that overlap in a first stereo measurement field;

a second stereo measurement unit comprising a plurality of imaging devices, the imaging devices having fields of view that overlap in a second stereo measurement field, the first and second stereo measurement fields overlapping one another in an overlap region; and

,

a processor, configured to;

receive image data from individual imaging devices in the fist and second stereo measurement units;

derive calibration parameters related to relative positions and orientations of the first and second stereo measurement units, such that image data received from the first and second stereo vision units may be transformed to a common reference coordinate system, using the calibration parameters, and,

locate features of objects in the first or second stereo measurement field in a common coordinate system using the image data and the calibration parameters.

42. An apparatus according to claim 41, wherein the imaging devices comprise at least one of:

digital cameras, analog cameras, acoustic sensors and laser range finders.

43. An apparatus according to claim 41, wherein the processor is one of:

a computer executing a program;

an embedded processor that is part of at least one of the first and second stereo measurement units and that is configured to execute instructions contained in memory accessible to the embedded processor; and

a detachable processor that is attachable to, and detachable from, the first and second stereo measurement units and that is configured to execute instructions contained in memory accessible to the detachable processor.

44. An apparatus for determining locations of object features, the apparatus comprising:

means for obtaining first data representing locations of object features in a first measurement field in a first local coordinate system;

means for obtaining second data representing locations of object features in a second measurement field in a second local coordinate system;

means for locating in the first data a first plurality of calibration features in the first measurement field and determining based on the first plurality of calibration features, locations of a second plurality of calibration features in the second measurement field;

means for locating in the second data the plurality of calibration features;

means for determining calibration parameters based on the calibration features located in the first and second data; and

,

means for locating object features relative to a common coordinate system using the calibration parameters.

45. The apparatus of claim 44 wherein the first and second measurement fields overlap in an overlap region.

46. The apparatus of claim 44 wherein the means for determining calibration parameters comprises a means for deriving one or more transformation functions from the first and second data.

48. An apparatus according to claim 46 wherein the first data comprises image data from two or more imaging devices and the apparatus comprises means for locating object features in the image data from each of two or more imaging devices and determining a position of the object features by triangulation.

49. An apparatus according to claim 44 wherein the first and second data respectively comprise coordinates in the first and second local coordinate systems of points in a point cloud.

47. The apparatus of clam 45 wherein the transformation functions are adapted to provide as outputs the locations of object features in a common coordinate system when the transformation functions are used in processing the first data or the second data or both the first and second data.

50. Processing apparatus for use in determining locations of object features detected by one of a plurality of measurement in units, the processing apparatus comprising:

an input for receiving first data from which locations of object features in a first measurement field can be determined in a first local coordinate system and second data from which locations of object features in a second measurement field can be determined in a second local coordinate system, the first and second measurement fields overlapping in an overlap region; and

,

a processor configured to;

extract from each of the first and second data calibration information about a plurality of calibration features of one or more calibration objects located in the overlap region; and

,

based on the calibration information, generate a transformation which, when applied to the second data expresses locations of object features in a common coordinate system.

51. A machine readable medium carrying a set of instructions which, when executed by a data processor cause the data processor to perform a method for calibrating a three-dimensional imaging system, comprising;

receiving first image data from a plurality of first imaging devices of a first measurement unit, the first imaging devices having fields of view that overlap in a first measurement field and second image data from a plurality of second imaging devices of a second measurement unit, the second imaging devices having fields of view that overlap in a second measurement field, the first and second stereo measurement fields overlapping one another in an overlap region;

locating a plurality of calibration features in each of the first and second image data;

determining, from the locations in the first and second data of the calibration features, information characterizing a position and orientation of the second stereo measurement unit relative to the first stereo measurement unit.

52. A method of calibrating a three-dimensional imaging system comprising a plurality of measurement units each having a local coordinate system, the method comprising:

a step for acquiring locations of a plurality of calibration features in a first local coordinate system of a first measurement unit;

a step for acquiring locations of the plurality of calibration features in each of one or more additional local coordinate systems; and

,

a step for creating a transformation from one or more of the local coordinate systems to a common coordinate system based on the locations of the plurality of calibration features in the first local coordinate system and the locations of the plurality of calibration features in at least one of the one or more additional local coordinate systems.

53. A method for calibrating a three-dimensional imaging system comprising:

providing first and second measurement units, the first measurement unit configured to determine locations in a first local coordinate system of object features within a first measurement field, the second measurement unit configured to determine in a second local coordinate system locations of object features within a second measurement field;

determining locations of a first plurality of calibration features in the first local coordinate system, the first plurality of calibration features located in the first measurement field;

determining locations of a second plurality of calibration features in the second local coordinate system the second plurality of calibration features located in the second measurement field, first plurality of features having a fixed location relative to the second plurality of features; and

,

deriving a transformation between the local coordinate systems and a common coordinate system based upon the locations of the first and second pluralities of calibration features.

54. A method according to claim 53, wherein said first and second stereo measurement fields overlap within an overlap region and the first and second pluralities of features are located in the overlap region.

55. A method according to claim 54, wherein determining locations of a first plurality of calibration features in the first local coordinate system also determines the locations of the second plurality of calibration features in the first local coordinate system.