Geoaccurate three-dimensional reconstruction via image-based geometry

• US 9,466,143 B1
• Filed: 05/03/2013
• Issued: 10/11/2016
• Est. Priority Date: 05/03/2013
• Status: Active Grant

• Current Assignee: Harris Corporation (L3Harris Technologies, Inc.)
• Original Assignee: Exelis Incorporated (L3Harris Technologies, Inc.)
• Inventors: Walvoord, Derek J., Rossi, Adam J., Paul, Bradley D., Brower, Bernard V.
• Primary Examiner(s): Zimmerman, Mark
• Assistant Examiner(s): Le, Sarah
• Application Number:
  US13/887,096
• Time in Patent Office: 1,257 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A63F 13/5252 using two or more virtual cameras concurrently or sequentially, e.g. automatically switching between fixed virtual cameras when a character changes room or displaying a rear-mirror view in a car-driving game
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/10116 X-ray image
• CPC Class Codes: G06T 2207/30181 Earth observation
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems

Abstract

A technique for generating a three-dimensional reconstruction of a scene involves generating a high-fidelity point cloud representing a three-dimensional reconstruction of a scene from two-dimensional images generated by at least one sensor whose position and orientation are known relative to a fixed coordinate system for each of the images. The high-fidelity point cloud is generated in a relative coordinate system without regard to the position and orientation of the sensor(s). A low-fidelity point cloud is generated in the fixed coordinate system from the two-dimensional images using the position and orientation of the sensor(s) relative to the fixed coordinate system. A transformation between the relative and fixed coordinate systems is determined based on a relationship between points in the high-fidelity and low-fidelity point clouds, and the high-fidelity point cloud is converted from the relative coordinate system to the fixed coordinate system by applying the transformation to the high-fidelity point cloud.

Claims

1. A method of generating a three-dimensional reconstruction of a scene, the method comprising:

generating a high-fidelity point cloud representing a three-dimensional reconstruction of a scene from a plurality of two-dimensional images generated by at least one sensor whose position and orientation are known relative to a fixed coordinate system for each of the two-dimensional images, the high-fidelity point cloud being generated in a relative coordinate system without regard to the position and orientation of the at least one sensor relative to the fixed coordinate system;

generating in the fixed coordinate system a low-fidelity point cloud representing a three-dimensional reconstruction of the scene from the plurality of two-dimensional images using the position and orientation of the at least one sensor relative to the fixed coordinate system such that the high-fidelity point cloud and the low-fidelity point cloud are generated from the same plurality of two-dimensional images, the low-fidelity point cloud having a lower fidelity than the high-fidelity point cloud due to physical sensor model errors in the position and orientation of the at least one sensor;

determining a transformation between the relative coordinate system and the fixed coordinate system based on a relationship between points in the high-fidelity point cloud and corresponding points in the low-fidelity point cloud; and

converting the high-fidelity point cloud from the relative coordinate system to the fixed coordinate system by applying the transformation to points in the high-fidelity point cloud, wherein an impact of the physical sensor model errors on the high-fidelity point cloud in the fixed coordinate system is limited to errors introduced in the transformation.

2. The method of claim 1, wherein the high-fidelity point cloud is generated using structure from motion (SfM) processing.

3. The method of claim 1, wherein the relative coordinate system is an arbitrary world coordinate system (WCS) and the fixed coordinate system is an earth-based coordinate system, and wherein the low-fidelity point cloud is generated based on a ground-to-image function of the at least one sensor.

4. The method of claim 1, wherein the transformation is determined using a sparse high-fidelity point cloud containing only a subset of the points in the high-fidelity point cloud.

5. The method of claim 1, wherein the transformation is applied to the high-fidelity point cloud via a transformation matrix that performs translation, scaling and rotation to each point in the high-fidelity point cloud such that relative distances between the points in the high-fidelity point cloud are maintained in the fixed coordinate system.

6. The method of claim 1, wherein the high-fidelity point cloud is converted from the relative coordinate system to the fixed coordinate system without use of a digital elevation map (DEM) or ground control points (GCPs).

7. The method of claim 1, wherein the position and orientation of the at least one sensor are received as metadata associated with each of the plurality of two-dimensional images, the metadata including global positioning system (GPS) data and inertial navigation system (INS) data.

8. An image reconstruction system for generating a three-dimensional reconstruction of a scene, comprising:

an interface configured to receive a plurality of two-dimensional images generated by at least one sensor whose position and orientation are known relative to a fixed coordinate system for each of the two-dimensional images; and

a processor configured to;

generate a high-fidelity point cloud representing a three-dimensional reconstruction of a scene from the plurality of two-dimensional images, the high-fidelity point cloud being generated in a relative coordinate system without regard to the position and orientation of the at least one sensor relative to the fixed coordinate system;

generate in the fixed coordinate system a low-fidelity point cloud representing a three-dimensional reconstruction of the scene from the plurality of two-dimensional images using the position and orientation of the at least one sensor relative to the fixed coordinate system such that the high-fidelity point cloud and the low-fidelity point cloud are generated from the same plurality of two-dimensional images, the low-fidelity point cloud having a lower fidelity than the high-fidelity point cloud due to physical sensor model errors in the position and orientation of the at least one sensor;

determine a transformation between the relative coordinate system and the fixed coordinate system based on a relationship between points in the high-fidelity point cloud and corresponding points in the low-fidelity point cloud; and

convert the high-fidelity point cloud from the relative coordinate system to the fixed coordinate system by applying the transformation to points in the high-fidelity point cloud, wherein an impact of the physical sensor model errors on the high-fidelity point cloud in the fixed coordinate system is limited to errors introduced in the transformation.

9. The system of claim 8, wherein the processor is further configured to generate the high-fidelity point cloud using structure from motion (SfM) processing.

10. The system of claim 8, wherein the relative coordinate system is an arbitrary world coordinate system (WCS) and the fixed coordinate system is an earth-based coordinate system, and wherein the processor is further configured to generate the low-fidelity point cloud based on a ground-to-image function of the at least one sensor.

11. The system of claim 8, wherein the processor is further configured to determine the transformation using a sparse high-fidelity point cloud containing only a subset of the points in the high-fidelity point cloud.

12. The system of claim 8, wherein processor is further configured to apply the transformation to the high-fidelity point cloud via a transformation matrix that performs translation, scaling and rotation to each point in the high-fidelity point cloud such that relative distances between the points in the high-fidelity point cloud are maintained in the fixed coordinate system.

13. The system of claim 8, wherein the processor is further configured to convert the high-fidelity point cloud from the relative coordinate system to the fixed coordinate system without use of a digital elevation map (DEM) or ground control points (GCPs).

14. The system of claim 8, wherein the interface is further configured to receive the position and orientation of the at least one sensor as metadata associated with each of the plurality of two-dimensional images, the metadata including global positioning system (GPS) data and inertial navigation system (INS) data.

15. A non-transitory computer readable medium encoded with software comprising processor-executable instructions that, when executed by one or more processors, cause the one or more processors to perform functions of:

generate a high-fidelity point cloud representing a three-dimensional reconstruction of a scene from a plurality of two-dimensional images generated by at least one sensor whose position and orientation are known relative to a fixed coordinate system for each of the two-dimensional images, the high-fidelity point cloud being generated in a relative coordinate system without regard to the position and orientation of the at least one sensor relative to the fixed coordinate system;

generate in the fixed coordinate system a low-fidelity point cloud representing a three-dimensional reconstruction of the scene from the plurality of two-dimensional images using the position and orientation of the at least one sensor relative to the fixed coordinate system such that the high-fidelity point cloud and the low-fidelity point cloud are generated from the same plurality of two-dimensional images, the low-fidelity point cloud having a lower fidelity than the high-fidelity point cloud due to physical sensor model errors in the position and orientation of the at least one sensor;

determine a transformation between the relative coordinate system and the fixed coordinate system based on a relationship between points in the high-fidelity point cloud and corresponding points in the low-fidelity point cloud; and

convert the high-fidelity point cloud from the relative coordinate system to the fixed coordinate system by applying the transformation to points in the high-fidelity point cloud, wherein an impact of the physical sensor model errors on the high-fidelity point cloud in the fixed coordinate system is limited to errors introduced in the transformation.

16. The non-transitory computer readable medium of claim 15, wherein the instructions to generate the high-fidelity point cloud comprise instructions that cause the one or processors to generate the high-fidelity point cloud using structure from motion (SfM) processing.

17. The non-transitory computer readable medium of claim 15, wherein the relative coordinate system is an arbitrary world coordinate system (WCS) and the fixed coordinate system is an earth-based coordinate system, and wherein the instructions to generate the low-fidelity point cloud comprise instructions that cause the one or more processors to generate the low-fidelity point cloud based on a ground-to-image function of the at least one sensor.

18. The non-transitory computer readable medium of claim 15, wherein the instructions to determine the transformation comprise instructions that cause the one or processors to determine the transformation using a sparse high-fidelity point cloud containing only a subset of the points in the high-fidelity point cloud.

19. The non-transitory computer readable medium of claim 15, wherein the instructions to convert the high-fidelity point cloud comprise instructions that cause the one or more processors to apply the transformation to the high-fidelity point cloud via a transformation matrix that performs translation, scaling and rotation to each point in the high-fidelity point cloud such that relative distances between the points in the high-fidelity point cloud are maintained in the fixed coordinate system.

20. The non-transitory computer readable medium of claim 15, wherein the instructions to convert the high-fidelity point cloud comprise instructions that cause the one or more processors to convert the high-fidelity point cloud from the relative coordinate system to the fixed coordinate system without use of a digital elevation map (DEM) or ground control points (GCPs).