3D imaging system

• US 7,860,301 B2
• Filed: 02/13/2006
• Issued: 12/28/2010
• Est. Priority Date: 02/11/2005
• Status: Active Grant

• Current Assignee: Macdonald Dettwiler and Associates Limited
• Original Assignee: Macdonald Dettwiler and Associates Limited
• Inventors: Jakola, Roy Henry, Se, Shuen Yan Stephen, Jasiobedzki, Piotr, Parry, David Owen
• Primary Examiner(s): Mariam; Daniel G
• Assistant Examiner(s): Woldemariam; Aklilu K
• Application Number:
  US11/352,399
• Publication Number:
  US 20060221072A1
• Time in Patent Office: 1,779 Days
• Field of Search: 382/154, 382/219, 702/16, 701/28, 348/43, 345/420
• US Class Current: 382/154
• CPC Class Codes: A61B 5/113 occurring during breathing
• CPC Class Codes: G01C 11/06 by comparison of two or more pictures of the same area
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/593 from stereo images

Abstract

A system for creating photorealistic 3D models of environments and/or objects from a plurality of stereo images obtained from a mobile stereo camera and optional monocular cameras, for enhancing 3D models of environments or objects by registering information from additional sensors to improve model fidelity or to augment it with supplementary information using a light pattern projector, and for generating photo-realistic 3D models of underground environments such as tunnels, mines, voids and caves, including automatic registration of the 3D models with pre-existing underground maps. The cameras may be handheld, mounted on a mobile platform, manipulator or a positioning device. The system automatically detects and tracks features in image sequences and self-references the stereo camera in 6 degrees of freedom by matching the features to a database to track the camera motion, while building the database simultaneously. A motion estimate may be provided from external sensors and fused with the motion computed from the images.

Claims

1. A method of producing a three-dimensional (3D) model of an environment, comprising the steps of:

a) acquiring a plurality of overlapping successive stereo images of an environment from one or more stereo cameras which are moving with respect to the environment, each of the one or more stereo cameras having at least two individual image capture means where there is an overlap between images captured by said at least two individual image capture means;

b) for each successive stereo image, detecting features in the stereo images computing a 3D position and descriptor for each detected feature, and storing said 3D position and said descriptor as a 3D feature in a database;

c) for each successive stereo image, computing relative motion of the one or more stereo cameras with respect to the environment by matching the detected features in the stereo images with said 3D features stored in the database using descriptors of said 3D features;

d) for each successive stereo image, computing dense 3D data sets, representative of the environment from one or more range sensing devices;

e) for each successive stereo image, transforming the computed dense 3D data from step d) into a selected coordinate frame of reference using the computed relative motion from step c) to give transformed dense 3D data in the selected coordinate frame of reference; and

f) storing the transformed dense 3D data for each successive stereo image and producing a 3D model of the environment from the stored transformed dense 3D data.

2. The method according to claim 1 wherein said one or more range sensing devices is said one or more stereo cameras, and the dense 3D data is computed from said stereo images captured by said one or more stereo cameras, by matching image intensity or color distributions between the stereo images.

3. The method according to claim 1 wherein said one or more range sensing devices is a rangefinder, and wherein said dense 3D data is computed from range data acquired from said rangefinder.

4. The method according to claim 1 including estimating motion of said one or more stereo cameras from at least one external sensor which gives position, velocity or acceleration information, and wherein the step c) of computing relative motion of the one or more stereo cameras includes combining said estimated motion of said one or more stereo cameras from said external sensor information with the computed relative motion to give an improved estimated motion of said one or more stereo cameras.

5. The method according to claim 1 wherein said one or more stereo cameras is two or more stereo cameras, wherein said two or more stereo cameras are in a fixed spatial relationship with respect to each other.

6. The method according to claim 1 wherein said representation of the environment suitable for visualization, analysis or post-processing is selected from the group consisting of point clouds, surface meshes and 3D shapes.

7. The method according to claim 1 wherein said 3D features are 3D tie-points, and wherein said step b) of detecting 3D features in the successive stereo images includes detecting 2D tie-points in each stereo image independently, and matching them between two or more stereo images using similarities in 2D tie-point descriptors and computing a 3D position of each 2D tie-point to give a 3D tie-point.

8. The method according to claim 1 wherein said 3D features are 3D tie-points, and wherein said step b) of detecting 3D features in the successive stereo images includes detecting 3D tie-points in the computed dense 3D data from step d).

9. The method according to claim 7 wherein said step c) of computing relative motion of the one or more stereo cameras includes matching, validating and computing a transformation between the detected 3D tie-points and 3D tie-points stored in the database, and wherein if new tie-points are detected they are added to the database.

10. The method according to claim 8 wherein said step c) of computing relative motion of the one or more stereo cameras includes matching, validating and computing a transformation between the detected 3D tie-points and 3D tie-points stored in the database, and wherein if new tie-points are detected they are added to the database.

11. The method according to claim 1 wherein step c) gives a first computed relative motion, including aligning successive dense 3D data sets from step d) to give a second computed relative motion, and combining the first and second relative motions to give an improved computed relative motion.

12. The method according to claim 1 wherein step c) gives a first computed relative motion, an wherein step f) includes accumulating the transformed dense 3D data in a data base, and including aligning new dense 3D data sets from step d) with dense 3D data stored in the data base to give a second computed relative motion, and combining the first and second relative motions to give an improved computed relative motion.

13. The method according to claim 1 including removing redundant or erroneous data from the transformed dense 3D data.

14. The method according to claim 1 including a step of filling in missing data in the produced 3D model of the environment by interpolation.

15. The method according to claim 1 including a step of selecting and applying texture to the 3D model.

16. The method according to claim 4 wherein said at least one external sensor includes any one or combination of inertial measurement devices, Global Positioning Systems (GPS), wheel odometry or pedometers, and telemetry from a camera manipulator, camera positioning device and camera pose tracking devices.

17. The method according to claim 16 wherein said inertial measurement device is selected from the group consisting of gyroscopes, inclinometers, compasses, accelerometers.

18. The method according to claim 1 wherein step c) includes using predicted positions of the 3D features to select a subset of 3D features stored in the data base for more efficient matching.

19. The method according to claim 1 wherein step c) includes matching newly acquired 3D features with all previously acquired 3D features stored in the data base.

20. The method according to claim 1 wherein said one or more stereo cameras is a first stereo camera, and wherein said data base is a first data base, and including at least a second stereo camera for detecting second 3D features and at least a second data base for storing 3D positions and descriptors of each second 3D feature, and wherein step c) includes matching 3D features stored in the first data base with second 3D features stored in the second data base.

21. The method according to claim 1 wherein said data base is a first data base for storing 3D features acquired in a first scan of the one or more stereo cameras, and including at least a second data base for storing subsequent 3D features obtained in a subsequent scan by said one or more stereo cameras, and wherein step c) includes matching 3D features stored in the first data base with 3D features stored in the second data base.

22. The method according to claim 18 wherein said 3D features are 3D tie-points, and wherein said step b) of detecting 3D features in the successive stereo images includes detecting 2D tie-points in each stereo image independently, and matching them between two or more stereo images using similarities in 2D tie-point descriptors and computing a 3D position of each 2D tie-point to give a 3D tie-point.

23. The method according to claim 18 wherein said 3D features are 3D tie-points, and wherein said step b) of detecting 3D features in the successive stereo images includes detecting 3D tie-points in the computed dense 3D data from step d).

24. The method according to claim 19 wherein said 3D features are 3D tie-points, and wherein said step b) of detecting 3D features in the stereo images includes detecting 2D tie-points in each stereo image independently, and matching them between two or more stereo images using similarities in 2D tie-point descriptors and computing a 3D position of each 2D tie-point to give a 3D tie-point.

25. The method according to claim 19 wherein said 3D features are 3D tie-points, and wherein said step b) of detecting 3D features in the successive stereo images includes detecting 3D tie-points in the computed dense 3D data from step d).

26. The method according to claim 20 wherein said 3D features are 3D tie-points, and wherein said step b) of detecting 3D features in the successive stereo images includes detecting 2D tie-points in each stereo image independently, and matching them between two or more stereo images using similarities in 2D tie-point descriptors and computing a 3D position of each 2D tie-point to give a 3D tie-point.

27. The method according to claim 20 wherein said 3D features are 3D tie-points, and wherein said step b) of detecting 3D features in the successive stereo images includes detecting 3D tie-points in the computed dense 3D data from step d).

28. The method according to claim 21 wherein said 3D features are 3D tie-points, and wherein said step b) of detecting 3D features in the successive stereo images includes detecting 2D tie-points in each stereo image independently, and matching them between two or more stereo images using similarities in 2D tie-point descriptors and computing a 3D position of each 2D tie-point to give a 3D tie-point.

29. The method according to claim 21 wherein said 3D features are 3D tie-points, and wherein said step b) of detecting 3D features in the successive stereo images includes detecting 3D tie-points in the computed dense 3D data from step d).

37. The method according to claim 1 wherein said environment is an object.

39. The method according to claim 1 wherein the movement of said one or more stereo cameras is movement of the one or more stereo cameras throughout the environment mounted on a mobile platform.

40. The method according to claim 1 wherein the movement of said one or more stereo cameras is scanning movement of the one or more stereo cameras throughout the environment mounted on a stationary platform and being scanned by a pan and tilt unit connected to said one or more stereo cameras.

41. The method according to claim 1 wherein the movement of said one or more stereo cameras is by an operator holding said one or more stereo cameras moving through said environment.

30. An apparatus for producing a three-dimensional (3D) model of an environment, comprising:

a) one or more stereo cameras for acquiring a plurality of overlapping successive stereo images of an environment from each of said one or more stereo cameras having at least two individual image capture means where there is an overlap between images captured by said at least two individual image capture means;

b) processing means for detecting features in the successive stereo images and computing a 3D position and descriptor for each detected feature, and storing said 3D position and said descriptor of each 3D feature in a database, said processing means including means for computing relative motion of the one or more stereo cameras with respect to the environment by matching the detected features in the stereo images with 3D features stored in the database using descriptors of said features;

c) one or more range sensing devices and processing means for computing dense 3D data sets, representative of the environment, from said one or more range sensing devices;

d) processing means for transforming the computed dense 3D data sets into a selected coordinate frame of reference using the computed relative motion of the at least one stereo camera to give transformed dense 3D data in the selected coordinate frame of reference; and

e) storage means for storing the transformed dense 3D data sets, and processing means for producing a 3D model of the environment from the stored transformed dense 3D data.

31. The apparatus according to claim 30 wherein said one or more range sensing devices is said one or more stereo cameras, and wherein the processing means for computing dense 3D data sets computes the dense 3D data sets from said images captured by said one or more stereo cameras, by matching image intensity or color distributions between the stereo images.

32. The apparatus according to claim 30 wherein said one or more range sensing devices is a rangefinder.

33. The apparatus according to claim 30 including at least one external sensor which gives position, velocity or acceleration information for estimating motion of said one or more stereo cameras, and wherein said processing means for computing relative motion of the one or more stereo cameras includes combining said estimated motion of said one or more stereo cameras from said external sensor information with the computed relative motion.

34. The method according to claim 33 wherein said at least one external sensor includes any one or combination of inertial measurement devices, Global Positioning Systems (GPS), wheel odometry or pedometers, and telemetry from a camera manipulator, camera positioning device and camera pose tracking devices.

35. The method according to claim 34 wherein said inertial measurement device is selected from the group consisting of gyroscopes, inclinometers, compasses and accelerometers.

36. The apparatus according to claim 30 wherein said one or more stereo cameras is two or more stereo cameras, wherein said two or more stereo cameras are in a fixed spatial relationship with respect to each other.

38. The system according to claim 30 wherein said environment is an object.