3D imaging system

• US 20060221072A1
• Filed: 02/13/2006
• Published: 10/05/2006
• Est. Priority Date: 02/11/2005
• Status: Active Grant

• Current Assignee: Macdonald Dettwiler and Associates Limited
• Original Assignee: Macdonald Dettwiler and Associates Limited
• Inventors: Jakola, Roy Henry, Jasiobedzki, Piotr, Parry, David Owen, SE, Shuen Yan Stephen
• Granted Patent: US 7,860,301 B2
• Time in Patent Office: Days
• US Class Current: 345/420
• CPC Class Codes: A61B 5/113 occurring during breathing
• CPC Class Codes: G01C 11/06 by comparison of two or more pictures of the same area
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/593 from stereo images

Abstract

The present invention provides a system (method and apparatus) for creating photorealistic 3D models of environments and/or objects from a plurality of stereo images obtained from a mobile stereo camera and optional monocular cameras. The cameras may be handheld, mounted on a mobile platform, manipulator or a positioning device. The system automatically detects and tracks features in image sequences and self-references the stereo camera in 6 degrees of freedom by matching the features to a database to track the camera motion, while building the database simultaneously. A motion estimate may be also provided from external sensors and fused with the motion computed from the images. Individual stereo pairs are processed to compute dense 3D data representing the scene and are transformed, using the estimated camera motion, into a common reference and fused together. The resulting 3D data is represented as point clouds, surfaces, or volumes. The present invention also provides a system (method and apparatus) for enhancing 3D models of environments or objects by registering information from additional sensors to improve model fidelity or to augment it with supplementary information by using a light pattern projector. The present invention also provides a system (method and apparatus) for generating photo-realistic 3D models of underground environments such as tunnels, mines, voids and caves, including automatic registration of the 3D models with pre-existing underground maps.

Claims

1. A method of producing a three-dimensional (3D) model of an environment or an object, comprising the steps of:

a) acquiring a plurality of stereo images of an environment or an object from at least one stereo camera, moving with respect to said environment or object, the stereo camera having at least two individual image capture means where there is an overlap between images captured by said at least two individual image capture means;

b) detecting 3D features in the stereo images and computing a 3D position and descriptor for each detected 3D feature, and storing said 3D position and said descriptor of each 3D feature in a database;

c) computing relative motion of the at least one stereo camera with respect to the environment or object by matching the detected 3D features in the stereo images with 3D features stored in the database using descriptors of said 3D features;

d) computing dense 3D data sets, representative of the environment or object, from at least one range sensing device;

e) transforming the computed dense 3D data from step d) into a selected coordinate frame of reference using the computed relative motion from step c) to give transformed dense 3D data in the selected coordinate frame of reference; and

f) storing the transformed dense 3D data, and producing a 3D model of the environment or object from the stored transformed dense 3D data suitable for visualization, analysis or post-processing.

2. The method according to claim 1 wherein said at least one range sensing device is said at least one stereo camera, and the dense 3D data is computed from said stereo images captured by said at least two individual image capture means, by matching image intensity or color distributions between the stereo images.

3. The method according to claim 1 wherein said at least one range sensing device is a rangefinder, and wherein said dense 3D data is computed from range data acquired from said rangefinder.

4. The method according to claim 1 including estimating motion of said at least one stereo camera from at least one external sensor which gives position, velocity or acceleration information, and wherein the step c) of computing relative motion of the at least one stereo camera includes combining said estimated motion of said at least one stereo camera from said external sensor information with the computed relative motion to give an improved estimated motion of said at least one stereo camera.

5. The method according to claim 1 wherein said at least one stereo camera is two or more stereo cameras, wherein said two or more stereo cameras are in a fixed spatial relationship with respect to each other.

6. The method according to claim 1 wherein said representation of the environment or object suitable for visualization, analysis or post-processing is selected from the group consisting of point clouds, surface meshes and 3D shapes.

7. The method according to claim 1 wherein said 3D features are 3D tie-points, and wherein said step b) of detecting 3D features in the stereo images includes detecting 2D tie-points in each stereo image independently, and matching them between two or more stereo images using similarities in 2D tie-point descriptors and computing a 3D position of each 2D tie-point to give a 3D tie-point.

8. The method according to claim 1 wherein said 3D features are 3D tie-points, and wherein said step b) of detecting 3D features in the stereo images includes detecting 3D tie-points in the computed dense 3D data from step d).

9. The method according to claim 7 wherein said step c) of computing relative motion of the at least one stereo camera includes matching, validating and computing a transformation between the detected 3D tie-points and 3D tie-points stored in the database, and wherein if new tie-points are detected they are added to the database.

10. The method according to claim 8 wherein said step c) of computing relative motion of the at least one stereo camera includes matching, validating and computing a transformation between the detected 3D tie-points and 3D tie-points stored in the database, and wherein if new tie-points are detected they are added to the database.

11. The method according to claim 1 wherein step c) gives a first computed relative motion, including aligning successive dense 3D data sets from step d) to give a second computed relative motion, and combining the first and second relative motions to give an improved computed relative motion.

12. The method according to claim 1 wherein step c) gives a first computed relative motion, an wherein step f) includes accumulating the transformed dense 3D data in a data base, and including aligning new dense 3D data sets from step d) with dense 3D data stored in the data base to give a second computed relative motion, and combining the first and second relative motions to give an improved computed relative motion.

13. The method according to claim 1 including removing redundant or erroneous data from the transformed dense 3D data.

14. The method according to claim 1 including a step of filling in missing data in the produced 3D model of the environment or object by interpolation.

15. The method according to claim 1 including a step of selecting and applying texture to the 3D model.

16. The method according to claim 4 wherein said at least one external sensor includes any one or combination of inertial measurement devices, Global Positioning Systems (GPS), wheel odometry or pedometers, and telemetry from a camera manipulator, camera positioning device and camera pose tracking devices.

17. The method according to claim 16 wherein said inertial measurement device is selected from the group consisting of gyroscopes, inclinometers, compasses, accelerometers.

18. The method according to claim 1 wherein step c) includes using predicted positions of the 3D features to select a subset of 3D features stored in the data base for more efficient matching.

19. The method according to claim 1 wherein step c) includes matching newly acquired 3D features with all previously acquired 3D features stored in the data base.

20. The method according to claim 1 wherein said at least one stereo camera is a first stereo camera, and wherein said data base is a first data base, and including at least a second stereo camera for detecting second 3D features and at least a second data base for storing 3D positions and descriptors of each second 3D feature, and wherein step c) includes matching 3D features stored in the first data base with second 3D features stored in the second data base.

21. The method according to claim 1 wherein said data base is a first data base for storing 3D features acquired in a first scan of the at least one stereo camera, and including at least a second data base for storing subsequent 3D features obtained in a subsequent scan by said at least one stereo camera, and wherein step c) includes matching 3D features stored in the first data base with 3D features stored in the second data base.

22. The method according to claim 18 wherein said 3D features are 3D tie-points, and wherein said step b) of detecting 3D features in the stereo images includes detecting 2D tie-points in each stereo image independently, and matching them between two or more stereo images using similarities in 2D tie-point descriptors and computing a 3D position of each 2D tie-point to give a 3D tie-point.

23. The method according to claim 18 wherein said 3D features are 3D tie-points, and wherein said step b) of detecting 3D features in the stereo images includes detecting 3D tie-points in the computed dense 3D data from step d).

24. The method according to claim 19 wherein said 3D features are 3D tie-points, and wherein said step b) of detecting 3D features in the stereo images includes detecting 2D tie-points in each stereo image independently, and matching them between two or more stereo images using similarities in 2D tie-point descriptors and computing a 3D position of each 2D tie-point to give a 3D tie-point.

25. The method according to claim 19 wherein said 3D features are 3D tie-points, and wherein said step b) of detecting 3D features in the stereo images includes detecting 3D tie-points in the computed dense 3D data from step d).

26. The method according to claim 20 wherein said 3D features are 3D tie-points, and wherein said step b) of detecting 3D features in the stereo images includes detecting 2D tie-points in each stereo image independently, and matching them between two or more stereo images using similarities in 2D tie-point descriptors and computing a 3D position of each 2D tie-point to give a 3D tie-point.

27. The method according to claim 20 wherein said 3D features are 3D tie-points, and wherein said step b) of detecting 3D features in the stereo images includes detecting 3D tie-points in the computed dense 3D data from step d).

28. The method according to claim 21 wherein said 3D features are 3D tie-points, and wherein said step b) of detecting 3D features in the stereo images includes detecting 2D tie-points in each stereo image independently, and matching them between two or more stereo images using similarities in 2D tie-point descriptors and computing a 3D position of each 2D tie-point to give a 3D tie-point.

29. The method according to claim 21 wherein said 3D features are 3D tie-points, and wherein said step b) of detecting 3D features in the stereo images includes detecting 3D tie-points in the computed dense 3D data from step d).

30. An apparatus for producing a three-dimensional (3D) model of an environment or an object, comprising the steps of:

a) camera means for acquiring a plurality of stereo images of an environment or an object from at least one stereo camera having at least two individual image capture means where there is an overlap between images captured by said at least two individual image capture means;

b) processing means for detecting 3D features in the stereo images and computing a 3D position and descriptor for each detected 3D feature, and storing said 3D position and said descriptor of each 3D feature in a database, said processing means including means for computing relative motion of the at least one stereo camera with respect to the environment or object by matching the detected 3D features in the stereo images with 3D features stored in the database using descriptors of said 3D features;

c) at least one range sensing device and processing means for computing dense 3D data sets, representative of the environment or object, from said at least one range sensing device;

e) processing means for transforming the computed dense 3D data sets into a selected coordinate frame of reference using the computed relative motion of the at least one stereo camera to give transformed dense 3D data in the selected coordinate frame of reference; and

f) storage means for storing the transformed dense 3D data sets, and processing means for producing a 3D model of the environment or object from the stored transformed dense 3D data suitable for visualization, analysis or post-processing.

31. The apparatus according to claim 30 wherein said at least one range sensing device is said at least one stereo camera, and wherein the processing means for computing dense 3D data sets computes the dense 3D data sets from said stereo images captured by said at least two individual image capture means, by matching image intensity or color distributions between the stereo images.

32. The apparatus according to claim 30 wherein said at least one range sensing device is a rangefinder.

33. The apparatus according to claim 30 including at least one external sensor which gives position, velocity or acceleration information for estimating motion of said at least one stereo camera, and wherein said processing means for computing relative motion of the at least one stereo camera includes combining said estimated motion of said at least one stereo camera from said external sensor information with the computed relative motion.

34. The method according to claim 33 wherein said at least one external sensor includes any one or combination of inertial measurement devices, Global Positioning Systems (GPS), wheel odometry or pedometers, and telemetry from a camera manipulator, camera positioning device and camera pose tracking devices.

35. The method according to claim 34 wherein said inertial measurement device is selected from the group consisting of gyroscopes, inclinometers, compasses and accelerometers.

36. The apparatus according to claim 30 wherein said at least one stereo camera is two or more stereo cameras, wherein said two or more stereo cameras are in a fixed spatial relationship with respect to each other.

37. A method of producing an enhanced three-dimensional (3D) model of an environment or an object, comprising the steps of:

a) acquiring 3D information of an environment or an object from at least one range sensing device;

b) producing a 3D model of the environment or object from the acquired 3D information suitable for visualization, analysis or post-processing;

c) projecting a pattern onto a region of interest of the environment or object;

d) acquiring data from a sensing device trained on the region of interest of the environment or object onto which the light pattern is projected;

registering the acquired data from the sensing device of region of interest with the 3D information from the range sensing device; and

e) combining the data from the sensing device with the 3D model to give an enhanced 3D model.

38. The method according to claim 37 wherein said least one range sensing device is a first stereo camera integrated with said light pattern projection means.

39. The method according to claim 38 wherein said first stereo camera includes at least two individual image capture means where there is an overlap between stereo images captured by said at least two individual image capture means, and wherein step b) of producing a 3D model of the environment or object from the acquired 3D information includes computing dense 3D data sets, representative of the environment, from stereo images captured by said at least two individual image capture means by matching image intensity or color distributions between the stereo images.

40. The method according to claim 38 wherein said sensing device trained on the region of interest is a second camera.

41. The method according to claim 40 wherein the projected light pattern comprises identification features allowing establishing correspondence between images captured by the first stereo camera and the second camera.

42. The method according to claim 37 including sampling areas of the region of interest using a sensor.

43. The method according to claim 42 wherein said sensor is a chemical, biological or radiological sensor for determining chemical or biological compositions of the region of interest.

44. The method according to claim 37 wherein said at least one range sensing device is a at least one stereo camera mounted for movement with respect to the environment or an object, and wherein the steps of acquiring 3D information of the environment or object from the at least one range sensing device and producing a 3D model of the environment from the acquired 3D information suitable for visualization, analysis or post-processing includes:

a) acquiring a plurality of stereo images of the environment or an object from said at least one stereo camera, moving with respect to said environment or object, the stereo camera having at least two individual image capture means where there is an overlap between images captured by said at least two individual image capture means;

b) detecting 3D features in the stereo images and computing a 3D position and descriptor for each detected 3D feature, and storing said 3D position and said descriptor of each 3D feature in a database;

c) computing relative motion of the at least one stereo camera with respect to the environment or object by matching the detected 3D features in the stereo images with 3D features stored in the database using descriptors of said 3D features;

d) computing dense 3D data sets, representative of the environment or object, from at least one range sensing device;

e) transforming the computed dense 3D data from step d) into a selected coordinate frame of reference using the computed relative motion from step c) to give transformed dense 3D data in the selected coordinate frame of reference; and

f) storing the transformed dense 3D data, and producing a 3D model of the environment or object from the stored transformed dense 3D data suitable for visualization, analysis or post-processing.

45. An apparatus for producing an enhanced three-dimensional (3D) model of an environment or an object, comprising the steps of:

a) at least one range sensing device for acquiring 3D information of an environment or an object;

b) processing means for producing a 3D model of the environment or object from the acquired 3D information suitable for visualization, analysis or post-processing;

c) light pattern projection means for projecting a pattern of light onto a region of interest of the environment or object;

d) a sensing device trained on the region of interest of the environment or object acquiring onto which the light pattern is projected; and

e) processing means for registering the acquired data from the sensing device containing the projected light pattern with the 3D information from the range sensing device and combining the data from the sensing device with the 3D model to give an enhanced 3D model.

46. The apparatus according to claim 45 wherein said at least one range sensing device is a stereo camera integrated with said light pattern projection means.

47. The apparatus according to claim 46 wherein said stereo camera integrated with said light pattern projection means are mounted on a pan and tilt unit.

48. The apparatus according to claim 47 wherein pan and tilt unit is mounted on any one of a stationary tripod and a mobile platform.

49. The apparatus according to claim 46 wherein said stereo camera integrated with said light pattern projection means are a hand-held device

50. The apparatus according to claim 45 including a sensor mounted to sample the region of interest of the environment or object.

51. The method according to claim 50 wherein said sensor is a chemical, biological or radiological sensor for determining chemical or biological compositions of the region of interest.

52. A method of producing a three-dimensional (3D) model of an underground environment, comprising the steps of:

a) acquiring 3D information of an underground environment from at least one range sensing device;

b) producing a 3D model of the underground environment from the acquired 3D information suitable for visualization, analysis or post-processing;

c) locating the range sensing device by back-sighting to at least two existing survey stations located in the underground environment; and

d) transforming the 3D model of the underground mine to a map of the underground environment.

53. The method according to claim 52 wherein said at least one range sensing device is a at least one stereo camera having at least two individual image capture means where there is an overlap between stereo images captured by said at least two individual image capture means, and wherein step b) of producing a 3D model of the underground environment from the acquired 3D information includes computing dense 3D data sets, representative of the underground environment, from stereo images captured by said at least two individual image capture means by matching image intensity or color distributions between the stereo images.

54. The method according to claim 52 wherein said at least one range sensing device is a at least one stereo camera mounted for movement through the underground environment, and wherein the steps of acquiring 3D information of an underground environment from at least one range sensing device and producing a 3D model of the underground environment from the acquired 3D information suitable for visualization, analysis or post-processing includes:

a) acquiring a plurality of stereo images of an underground environment from said at least one stereo camera, moving with respect to said underground environment, the stereo camera having at least two individual image capture means where there is an overlap between images captured by said at least two individual image capture means;

b) detecting 3D features in the stereo images and computing a 3D position and descriptor for each detected 3D feature, and storing said 3D position and said descriptor of each 3D feature in a database;

c) computing relative motion of the at least one stereo camera with respect to the environment or object by matching the detected 3D features in the stereo images with 3D features stored in the database using descriptors of said 3D features;

d) computing dense 3D data sets, representative of the underground environment, from at least one range sensing device;

e) transforming the computed dense 3D data from step d) into a selected coordinate frame of reference using the computed relative motion from step c) to give transformed dense 3D data in the selected coordinate frame of reference; and

f) storing the transformed dense 3D data, and producing a 3D model of the environment or object from the stored transformed dense 3D data suitable for visualization, analysis or post-processing.

55. The method according to claim 52 wherein said range sensing device is mounted on a pan-tilt unit and said 3D model is produced by accumulating dense 3D data from different positions using pan-tilt unit telemetry information.

56. The method according to claim 52 wherein said range sensing device is mounted on a total station and the said 3D model is produced by accumulating dense 3D data from different positions using total station telemetry information.

57. The method according to claim 52 including processing means for assessing ore grade and geological interpretation of the 3D model.

58. The method according to claim 52 wherein step c) of back-sighting is accomplished using a rangefinder mounted on a pan-tilt unit to locate the said range sensing device.

59. The method according to claim 52 wherein step c) of back-sighting is accomplished using a total station to locate the said range sensing device.

60. The method according to claim 52 including loading the transformed 3D model into a mine management processing means at an appropriate location of the overall mine map.

61. The method according to claim 60 including importing geological assessment data into the mine management processing means at the appropriate location of the overall mine map.

62. The method according to claim 52 including repeatedly registering multiple 3D models in the overall mine map as the mine advances.

63. An apparatus for producing a three-dimensional (3D) model of An underground environment, comprising:

a) a first range sensing device for acquiring 3D information of an underground environment;

b) processing means for producing a 3D model of the underground mine from the acquired 3D information suitable for visualization, analysis or post-processing; and

c) a second range sensing device for back-sighting to at least two existing survey stations located in the underground mine for locating the first and second range sensing devices in the underground mine, wherein said processing means includes means for transforming the 3D model of the underground mine to a map of the underground mine using the locations of the first and second range sensing devices.

64. The apparatus according to claim 63 wherein said first range sensing device is at least one stereo camera having at least two individual image capture means where there is an overlap between stereo images captured by said at least two individual image capture means, and wherein said processing means for producing a 3D model of the underground mine includes processing means for computing dense 3D data sets, representative of the underground environment, from stereo images captured by said at least two individual image capture means by matching image intensity or color distributions between the stereo images.

65. The apparatus according to claim 63 wherein the said first range sensing device is mounted on a pan-tilt unit.

66. The apparatus according to claim 63 wherein said second range sensing device is a rangefinder.

67. The apparatus according to claim 63 wherein said second range sensing device is a total station, and wherein said first range sensing device is mounted on said total station and is panned and tilted by said total station.

68. The apparatus according to claim 63 mounted on a vehicle.