Method and apparatus for producing 3D model of an underground environment

• US 8,031,909 B2
• Filed: 12/18/2009
• Issued: 10/04/2011
• Est. Priority Date: 02/11/2005
• Status: Active Grant

• Current Assignee: Macdonald Dettwiler and Associates Limited
• Original Assignee: Macdonald Dettwiler and Associates Limited
• Inventors: Jakola, Roy Henry, Se, Shuen Yan Stephen, Jasiobedzki, Piotr, Parry, David Owen
• Primary Examiner(s): Tucker; Wesley
• Assistant Examiner(s): WOLDEMARIAM, AKILILU K
• Application Number:
  US12/654,451
• Publication Number:
  US 20100098327A1
• Time in Patent Office: 655 Days
• Field of Search: 382/106, 382/120, 382/121, 382/122
• US Class Current: 382/106
• CPC Class Codes: A61B 5/113 occurring during breathing
• CPC Class Codes: G01C 11/06 by comparison of two or more pictures of the same area
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/593 from stereo images

Abstract

The present invention provides a system (method and apparatus) for creating photorealistic 3D models of environments and/or objects from a plurality of stereo images obtained from a mobile stereo camera and optional monocular cameras. The cameras may be handheld, mounted on a mobile platform, manipulator or a positioning device. The system automatically detects and tracks features in image sequences and self-references the stereo camera in 6 degrees of freedom by matching the features to a database to track the camera motion, while building the database simultaneously. A motion estimate may be also provided from external sensors and fused with the motion computed from the images. Individual stereo pairs are processed to compute dense 3D data representing the scene and are transformed, using the estimated camera motion, into a common reference and fused together. The resulting 3D data is represented as point clouds, surfaces, or volumes. The present invention also provides a system (method and apparatus) for enhancing 3D models of environments or objects by registering information from additional sensors to improve model fidelity or to augment it with supplementary information by using a light pattern projector. The present invention also provides a system (method and apparatus) for generating photo-realistic 3D models of underground environments such as tunnels, mines, voids and caves, including automatic registration of the 3D models with pre-existing underground maps.

Claims

1. A method of producing a three-dimensional (3D) model of an underground environment, comprising the steps of:

a) acquiring 3D information of an underground environment from at least one range sensing device;

b) producing a 3D model of the underground environment from the acquired 3D information suitable for visualization, analysis or post-processing;

c) locating the range sensing device by back-sighting to at least two existing survey stations located in the underground environment using a second range sensing device in a fixed relationship to said first range sensing device; and

d) based on the position information from step c, transforming the 3D model of the underground mine to a map of the underground environment in a coordinate frame of reference of the underground environment.

2. The method according to claim 1 wherein said at least one range sensing device is a at least one stereo camera having at least two individual image capture means where there is an overlap between stereo images captured by said at least two individual image capture means, and wherein step b) of producing a 3D model of the underground environment from the acquired 3D information includes computing dense 3D data sets, representative of the underground environment, from stereo images captured by said at least two individual image capture means by matching image intensity or color distributions between the stereo images.

3. The method according to claim 1 wherein said at least one range sensing device is a at least one stereo camera mounted for movement through the underground environment, and wherein the steps of acquiring 3D information of an underground environment from at least one range sensing device and producing a 3D model of the underground environment from the acquired 3D information suitable for visualization, analysis or post-processing includes:

a) acquiring a plurality of overlapping successive stereo images of the underground environment from said at least one stereo camera, moving with respect to said underground environment, the at least one stereo camera having at least two individual image capture means where there is an overlap between images captured by said at least two individual image capture means;

b) for each successive stereo image detecting features in the stereo images and computing a 3D position and descriptor for each detected feature, and storing said 3D position and said descriptor as a 3D feature in a database;

c) for each successive stereo image computing relative motion of the at least one stereo camera with respect to the environment or object by matching the detected features in the stereo images with said 3D features stored in the database using descriptors of said 3D features;

d) for each successive stereo image computing dense 3D data sets, representative of the environment or object, from said at least one range sensing device;

e) for each successive stereo image transforming the computed dense 3D data from step d) into a selected coordinate frame of reference using the computed relative motion from step c) to give transformed dense 3D data in the selected coordinate frame of reference;

f) storing the transformed dense 3D data for each successive stereo image and producing a 3D model of the environment or object from the stored transformed dense 3D data.

4. The method according to claim 1 wherein said first range sensing device is mounted on a pan-tilt unit and said 3D model is produced by accumulating dense 3D data from different positions using pan-tilt unit telemetry information.

5. The method according to claim 1 wherein said first range sensing device is mounted on a total station and the said 3D model is produced by accumulating dense 3D data from different positions using total station telemetry information.

6. The method according to claim 1 including processing means for assessing ore grade and geological interpretation of the 3D model.

7. The method according to claim 1 wherein step c) of back-sighting is accomplished using a rangefinder mounted on a pan-tilt unit to locate said first range sensing device.

8. The method according to claim 1 wherein step c) of back-sighting is accomplished using a total station to locate said first range sensing device.

9. The method according to claim 1 including loading the transformed 3D model into a mine management processing means at an appropriate location of the overall mine map.

10. The method according to claim 9 including importing geological assessment data into the mine management processing means at the appropriate location of the overall mine map.

11. The method according to claim 1 including repeatedly registering multiple 3D models in the overall mine map as the mine advances.

18. The method according to claim 1 wherein the underground environment is a mine undergoing excavation, and wherein the method includes repeating steps a) to d) inclusive after a successive amounts of excavation and producing successive maps of the underground environment.

19. The apparatus according to claim 1 wherein said first range sensing device is at least one stereo camera mounted for movement through the underground environment, and wherein the steps of acquiring 3D information of an underground environment from at least one range sensing device and producing a 3D model of the underground environment, and wherein the processing means is configured for:

a) acquiring a plurality of overlapping successive stereo images of the underground environment from said at least one stereo camera, moving with respect to said underground environment, the at least one stereo camera having at least two individual image capture means where there is an overlap between images captured by said at least two individual image capture means;

b) for each successive stereo image detecting features in the stereo images and computing a 3D position and descriptor for each detected feature, and storing said 3D position and said descriptor as a 3D feature in a database;

c) for each successive stereo image computing relative motion of the at least one stereo camera with respect to the environment or object by matching the detected features in the stereo images with said 3D features stored in the database using descriptors of said 3D features;

d) for each successive stereo image computing dense 3D data sets, representative of the environment or object, from said at least one range sensing device;

e) for each successive stereo image transforming the computed dense 3D data from step d) into a selected coordinate frame of reference using the computed relative motion from step c) to give transformed dense 3D data in the selected coordinate frame of reference; and

f) storing the transformed dense 3D data for each successive stereo image and producing a 3D model of the environment or object from the stored transformed dense 3D data.

12. An apparatus for producing a three-dimensional (3D) model of an underground environment, comprising:

a) a first range sensing device for acquiring 3D information of an underground environment;

b) processing means for producing a 3D model of the underground environment from the acquired 3D information suitable for visualization, analysis or post-processing; and

c) a second range sensing device for back-sighting to at least two existing survey stations located in the underground environment for locating the first and second range sensing devices in the underground environment, wherein said second range sensing device is in a fixed relationship with said first range sensing device and wherein said processing means includes means for transforming the 3D model of the underground environment to a map of the underground environment using the locations of the first and second range sensing devices.

13. The apparatus according to claim 12 wherein said first range sensing device is at least one stereo camera having at least two individual image capture means where there is an overlap between stereo images captured by said at least two individual image capture means, and wherein said processing means for producing a 3D model of the underground environment includes processing means for computing dense 3D data sets, representative of the underground environment, from stereo images captured by said at least two individual image capture means by matching image intensity or color distributions between the stereo images.

14. The apparatus according to claim 12 wherein the said first range sensing device is mounted on a pan-tilt unit.

15. The apparatus according to claim 12 wherein said second range sensing device is a rangefinder.

16. The apparatus according to claim 12 wherein said second range sensing device is a total station, and wherein said first range sensing device is mounted on said total station and is panned and tilted by said total station.

17. The apparatus according to claim 12 mounted on a vehicle.