Method and apparatus for producing 3D model of an environment
First Claim
1. A method of producing a three-dimensional (3D) model of an environment, comprising the steps of:
- a) acquiring a first and second plurality of successive stereo images of an environment from at least one of a plurality of stereo cameras which are moving with respect to the environment, the at least one stereo camera having at least two individual image capture means where there is an overlap between images captured by said at least two individual image capture means;
b) for each image of said first plurality of successive stereo images, detecting features in the stereo images and computing a 3D position and descriptor for each detected feature, and storing said 3D position and said descriptor as a 3D feature in a database;
c) for each image of said first plurality of successive stereo images, computing relative motion of the at least one stereo camera with respect to the environment by matching the detected features in the stereo images with said 3D features stored in the database using descriptors of said 3D features;
d) for each image of said second plurality of successive stereo images, computing dense 3D data sets, representative of the environment from the at least one or another of said plurality of stereo cameras;
e) for each image of said second plurality of successive stereo images, transforming the computed dense 3D data from step d) into a selected coordinate frame of reference using the computed relative motion from step c) to give transformed dense 3D data in the selected coordinate frame of reference; and
f) storing the transformed dense 3D data for each image of said second plurality of successive stereo images and producing a 3D model of the environment from the stored transformed dense 3D data.
9 Assignments
0 Petitions
Accused Products
Abstract
The present invention provides a system (method and apparatus) for creating photorealistic 3D models of environments and/or objects from a plurality of stereo images obtained from a mobile stereo camera and optional monocular cameras. The cameras may be handheld, mounted on a mobile platform, manipulator or a positioning device. The system automatically detects and tracks features in image sequences and self-references the stereo camera in 6 degrees of freedom by matching the features to a database to track the camera motion, while building the database simultaneously. A motion estimate may be also provided from external sensors and fused with the motion computed from the images. Individual stereo pairs are processed to compute dense 3D data representing the scene and are transformed, using the estimated camera motion, into a common reference and fused together. The resulting 3D data is represented as point clouds, surfaces, or volumes. The present invention also provides a system (method and apparatus) for enhancing 3D models of environments or objects by registering information from additional sensors to improve model fidelity or to augment it with supplementary information by using a light pattern projector. The present invention also provides a system (method and apparatus) for generating photo-realistic 3D models of underground environments such as tunnels, mines, voids and caves, including automatic registration of the 3D models with pre-existing underground maps.
-
Citations
33 Claims
-
1. A method of producing a three-dimensional (3D) model of an environment, comprising the steps of:
-
a) acquiring a first and second plurality of successive stereo images of an environment from at least one of a plurality of stereo cameras which are moving with respect to the environment, the at least one stereo camera having at least two individual image capture means where there is an overlap between images captured by said at least two individual image capture means; b) for each image of said first plurality of successive stereo images, detecting features in the stereo images and computing a 3D position and descriptor for each detected feature, and storing said 3D position and said descriptor as a 3D feature in a database; c) for each image of said first plurality of successive stereo images, computing relative motion of the at least one stereo camera with respect to the environment by matching the detected features in the stereo images with said 3D features stored in the database using descriptors of said 3D features; d) for each image of said second plurality of successive stereo images, computing dense 3D data sets, representative of the environment from the at least one or another of said plurality of stereo cameras; e) for each image of said second plurality of successive stereo images, transforming the computed dense 3D data from step d) into a selected coordinate frame of reference using the computed relative motion from step c) to give transformed dense 3D data in the selected coordinate frame of reference; and f) storing the transformed dense 3D data for each image of said second plurality of successive stereo images and producing a 3D model of the environment from the stored transformed dense 3D data. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
-
-
13. An apparatus for producing a three-dimensional (3D) model of an environment, comprising:
-
a) at least one stereo camera of a plurality of stereo cameras for acquiring a first and second plurality of successive stereo images of an environment which are moving with respect to the environment, the at least one stereo camera having at least two individual image capture means where there is an overlap between images captured by said at least two individual image capture means; b) processing means programmed with instructions for i) detecting, for each image of said first plurality of successive stereo images, features in the stereo images and computing a 3D position and descriptor for each detected feature, and storing said 3D position and said descriptor as a 3D feature in a database; ii) for each image of said first plurality of successive stereo images, computing relative motion of the at least one stereo camera with respect to the environment by matching the detected features in the stereo images with said 3D features stored in the database using descriptors of said 3D features; iii) for each image of said second plurality of successive stereo images, computing dense 3D data sets, representative of the environment from the at least one or another of said plurality of stereo cameras; iv) for each image of said second plurality of successive stereo images, transforming the computed dense 3D data from step b iii) into a selected coordinate frame of reference using the computed relative motion from step b ii) to give transformed dense 3D data in the selected coordinate frame of reference; and v) storing the transformed dense 3D data for each image of said second plurality of successive stereo images and producing a 3D model of the environment from the stored transformed dense 3D data. - View Dependent Claims (14, 15, 16, 17, 18, 19, 20, 21, 22, 23)
-
-
24. A method of producing a three-dimensional (3D) model of an underground environment, comprising the steps of:
-
a) acquiring 3D information of an underground environment from at least one range sensing device; b) producing a 3D surface model of the underground environment from the acquired 3D information suitable for visualization, analysis or post-processing; c) acquiring an image of said underground environment from a camera; d) texturing the 3D surface model with said image; e) locating the range sensing device by back-sighting to at least two existing survey stations located in the underground environment using a second range sensing device in a fixed relationship to said first range sensing device; and f) based on the position information from step e, transforming the 3D model of the underground mine to a map of the underground environment in a coordinate frame of reference of the underground environment. - View Dependent Claims (25, 26, 27, 28)
-
-
29. An apparatus for producing a three-dimensional (3D) model of an underground environment, comprising:
-
a) at least a first range sensing device for acquiring 3D information of an underground environment; and b) processing means programmed with instructions for i) producing a 3D surface model of the underground environment from the acquired 3D information suitable for visualization, analysis or post-processing; ii) acquiring an image of said underground environment from a camera; iii) texturing the 3D surface model with said image; iv) locating the range sensing device by back-sighting to at least two existing survey stations located in the underground environment using a second range sensing device in a fixed relationship to said first range sensing device; and v) based on the position information from iv), transforming the 3D model of the underground mine to a map of the underground environment in a coordinate frame of reference of the underground environment. - View Dependent Claims (30, 31, 32, 33)
-
Specification