Real-time camera position estimation with drift mitigation in incremental structure from motion

• US 10,269,147 B2
• Filed: 05/01/2017
• Issued: 04/23/2019
• Est. Priority Date: 05/01/2017
• Status: Active Grant

• Current Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Original Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Inventors: Jones, Michael, Dickin, Adam James
• Primary Examiner(s): Tran, Phuoc
• Application Number:
  US15/582,987
• Publication Number:
  US 20180315221A1
• Time in Patent Office: 722 Days
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5862 using texture
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 11/005 Specific pre-processing for tomographic reconstruction, e.g. calibration, source positioning, rebinning, scatter correction, retrospective gating
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2210/52 Parallel processing
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/17 taken from planes or by drones
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 30/242 : Division of the character sequences into groups prior to recognition; Selection of dictionaries

Abstract

A system provides camera position and point cloud estimation 3D reconstruction. The system receives images and attempts existing structure integration to integrate the images into an existing reconstruction under a sequential image reception assumption. If existing structure integration fails, the system attempts dictionary overlap detection by accessing a dictionary database and searching to find a closest matching frame in the existing reconstruction. If overlaps are found, the system matches the images with the overlaps to determine a highest probability frame from the overlaps, and attempts existing structure integration again. If overlaps are not found or existing structure integration fails again, the system attempts bootstrapping based on the images. If any of existing structure integration, dictionary overlap detection, or bootstrapping succeeds, and if multiple disparate tracks have come to exist, the system attempts reconstructed track merging.

Claims

1. A method for camera position and point cloud estimation for three-dimensional (3D) reconstruction, comprising:

receiving images;

attempting a first existing structure integration to integrate the images into an existing reconstruction under a sequential image reception assumption;

if the first existing structure integration fails, attempting a dictionary overlap detection by accessing a dictionary database and searching to find a closest matching frame in the existing reconstruction;

if overlaps are found, matching the images with the overlaps to determine a highest probability frame from the overlaps and attempting a second existing structure integration under the sequential image reception assumption;

if overlaps are not found or the second existing structure integration fails, attempting bootstrapping based on the images; and

if one of the first existing structure integration, the dictionary overlap detection, the second existing structure integration, or the bootstrapping succeeds, and if multiple disparate tracks have come to exist, attempting reconstructed track merging.

2. The method of claim 1, wherein the images comprise a new image and corresponding matched images, wherein the first existing structure integration comprises:

using feature matches to find existing world points in the new image under the sequential image reception assumption;

if sufficient existing world points are found, estimating an initial relative pose;

refining the initial relative pose to obtain a refined pose;

nominating additional existing overlap frames through spatial partitioning;

matching the new image with the additional existing overlap frames;

making a copy of the refined pose for each overlap frame and refining each copy;

discarding all pose copies and combining all inliers from each overlap frame;

refining the refined pose using combined inliers from the additional existing overlap frames to obtain a further refined pose and add it to the spatial partitioning;

triangulating all new inlier points and adding them to a structure database;

performing bundle adjustment over overlap poses and points with most views;

triangulating all affected world points; and

updating active point status across all affected points and views.

3. The method of claim 2, wherein the copies of the refined pose are refined in parallel threads.

4. The method of claim 2, wherein the refining of each copy of the refined pose is performed using only matches for a corresponding overlap frame.

5. The method of claim 2, wherein the bundle adjustment is run on a network of triangulated points and poses for each added frame by observing poses not included in overlapping frames and locking world points in the overlapping frames.

6. The method of claim 2, wherein the refining of the initial relative pose comprises applying iterative pose improvement on the initial relative pose.

7. The method of claim 2, wherein the refining of the initial relative pose comprises inlier tightening, a sequence of parallelized triangulations, and non-linear refinement with variable error margins, to obtain the refined pose.

8. The method of claim 2, wherein each point is flagged as active if it has sufficient observers to be considered stable and it does not have a degenerate angle of incidence.

9. The method of claim 2, wherein all structure changes are enqueued for visualization.

10. The method of claim 1, wherein estimated structures are integrated into a structure database.

11. The method of claim 10, wherein the structure database comprises a global relational data structure including all poses, world points, images, and their relationships, and including a caching mechanism and data structures for relating data.

12. A non-transitory computer readable medium having instructions stored thereon that, when executed by a processor, cause the processor to provide camera position and point cloud estimation for three-dimensional (3D) reconstruction, the processor:

receiving images;

attempting a first existing structure integration to integrate the images into an existing reconstruction under a sequential image reception assumption;

if the first existing structure integration fails, attempting a dictionary overlap detection by accessing a dictionary database and searching to find a closest matching frame in the existing reconstruction;

if overlaps are found, matching the images with the overlaps to determine a highest probability frame from the overlaps and attempting a second existing structure integration under the sequential image reception assumption;

if overlaps are not found or the second existing structure integration fails, attempting bootstrapping based on the images; and

if one of the first existing structure integration, the dictionary overlap detection, the second existing structure integration, or the bootstrapping succeeds, and if multiple disparate tracks have come to exist, attempting reconstructed track merging.

13. The computer readable medium of claim 12, wherein the images comprise a new image and corresponding matched images, wherein the first existing structure integration comprises:

using feature matches to find existing world points in the new image under the sequential image reception assumption;

if sufficient existing world points are found, estimating an initial relative pose;

refining the initial relative pose to obtain a refined pose;

nominating additional existing overlap frames through spatial partitioning;

matching the new image with the additional existing overlap frames;

making a copy of the refined pose for each overlap frame and refining each copy;

discarding all pose copies and combining all inliers from each overlap frame;

refining the refined pose using combined inliers from the additional existing overlap frames to obtain a further refined pose and add it to the spatial partitioning;

triangulating all new inlier points and adding them to a structure database;

performing bundle adjustment over overlap poses and points with most views;

triangulating all affected world points; and

updating active point status across all affected points and views.

14. The computer readable medium of claim 13, wherein the copies of the refined pose are refined in parallel threads.

15. The computer readable medium of claim 13, wherein the refining of each copy of the refined pose is performed using only matches for a corresponding overlap frame.

16. The computer readable medium of claim 13, wherein the bundle adjustment is run on a network of triangulated points and poses for each added frame by observing poses not included in overlapping frames and locking world points in the overlapping frames.

17. The computer readable medium of claim 13, wherein the refining of the initial relative pose comprises applying iterative pose improvement on the initial relative pose.

18. The computer readable medium of claim 13, wherein the refining of the initial relative pose comprises inlier tightening, a sequence of parallelized triangulations, and non-linear refinement with variable error margins, to obtain the refined pose.

19. The computer readable medium of claim 13, wherein each point is flagged as active if it has sufficient observers to be considered stable and it does not have a degenerate angle of incidence.

20. A system for camera position and point cloud estimation for three-dimensional (3D) reconstruction, the system comprising:

a processor; and

a memory coupled with the processor and storing instructions that, when executed by the processor, cause the processor to;

receive images;

attempt a first existing structure integration to integrate the images into an existing reconstruction under a sequential image reception assumption;

if the first existing structure integration fails, attempt a dictionary overlap detection by accessing a dictionary database and searching to find a closest matching frame in the existing reconstruction;

if overlaps are found, match the images with the overlaps to determine a highest probability frame from the overlaps and attempt a second existing structure integration under the sequential image reception assumption;

if overlaps are not found or the second existing structure integration fails, attempt bootstrapping based on the images; and

if one of the first existing structure integration, the dictionary overlap detection, the second existing structure integration, or the bootstrapping succeeds, and if multiple disparate tracks have come to exist, attempt reconstructed track merging.