Method and apparatus for recovering a three-dimensional scene from two-dimensional images
First Claim
1. A method of recovering a three-dimensional scene from two-dimensional images, the method comprising:
- providing a sequence of frames;
dividing the sequence of frames into frame segments comprising at least one base frame and at least one next frame wherein the frames in the sequence comprise feature points and wherein the sequence of frames is divided into frame segments based upon the at least one next frame in each frame segment having at least a minimum number of feature points being tracked to the at least one base frame in the frame segment;
performing three-dimensional reconstruction individually for each frame segment derived by dividing the sequence of frames; and
combining the three-dimensional reconstructed segments together to recover a three-dimensional scene for the sequence of images.
3 Assignments
0 Petitions
Accused Products
Abstract
A method and apparatus for recovering a three-dimensional (3D) scene from two-dimensional (2D) images. A sequence of images is divided into a number of smaller segments and a 3D reconstruction is performed on each segment individually. All the reconstructed segments are then combined together through an efficient bundle adjustment to complete the 3D reconstruction. Segmenting may be achieved by dividing the segments based on the number of feature points that are in each frame. The number of frames per segment is reduced by creating virtual key frames. The virtual key frames encode the 3D structure for each segment, but are only a small subset of the original frames in the segment. A final bundle adjustment is performed on the virtual key frames, rather than all of the original frames. Thus, the final bundle adjustment is two orders of magnitude faster than a conventional bundle adjustment.
-
Citations
34 Claims
-
1. A method of recovering a three-dimensional scene from two-dimensional images, the method comprising:
-
providing a sequence of frames; dividing the sequence of frames into frame segments comprising at least one base frame and at least one next frame wherein the frames in the sequence comprise feature points and wherein the sequence of frames is divided into frame segments based upon the at least one next frame in each frame segment having at least a minimum number of feature points being tracked to the at least one base frame in the frame segment; performing three-dimensional reconstruction individually for each frame segment derived by dividing the sequence of frames; and combining the three-dimensional reconstructed segments together to recover a three-dimensional scene for the sequence of images. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A method of recovering a three-dimensional scene from two-dimensional images, the method comprising:
-
identifying a sequence of two-dimensional frames that include two-dimensional images; dividing the sequence of frames into segments, wherein a segment includes a plurality of frames and wherein dividing includes;
identifying a base frame, identifying feature points in the base frame; and
determining the segments such that every frame in a segment has at least a predetermined percentage of feature points identified in the base frame;for each segment, encoding the frames in the segment into at least two virtual frames that include a three-dimensional structure for the segment and an uncertainty associated with the segment and wherein encoding includes choosing at least two frames in the segment that are at least a threshold number of frames apart; for each of the at least two chosen frames, projecting a plurality of three-dimensional points into a corresponding virtual frame; and for each of the at least two chosen frames, projecting an uncertainty into the corresponding virtual frame. - View Dependent Claims (9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19)
-
-
20. A method of recovering a three-dimensional scene from a sequence of two-dimensional frames, comprising:
-
(a) identifying at least a first base frame in a sequence of two-dimensional frames; (b) adding the at least first base frame to create a first segment of frames of the sequence; (c) selecting feature points in at least the first base frame in the first segment of frames in the sequence; (d) analyzing a next frame in the sequence to identify the selected feature points in the next frame; (e) determining a number of the selected feature points from the base frame that are also identified in the next frame; and (f) if the number of the selected feature points from the base frame that are also identified in the next frame is greater than or equal to a threshold number, adding the next frame to the first segment of frames of the sequence. - View Dependent Claims (21, 22, 23, 24, 25, 26, 27)
-
- 28. In a method of recovering a three-dimensional scene from a sequence of two-dimensional frames, an improvement comprising dividing a long sequence of frames into segments and reducing the number of frames in each segment by representing the segments using between two and five representative frames per segment, wherein the representative frames are used to recover the three-dimensional scene and remaining frames are discarded so that the three-dimensional scene is effectively compressed, wherein dividing the long sequence into segments includes identifying a base frame and tracking feature points between frames in the sequence and the base frame and ending a segment whenever a frame does not contain a predetermined threshold of feature points that are contained in the base frame.
-
33. A computer-readable medium having computer-executable instructions for performing a method comprising:
-
providing a sequence of two-dimensional frames; dividing the sequence into segments, each segment having at least two frames; calculating a partial model for each segment, wherein the partial model includes the same number of frames as the segment said partial model represents and wherein the partial model includes three-dimensional coordinates and camera pose, the camera pose comprising rotation and translation, for features within the frames; extracting virtual key frames from each partial model, the virtual key frames having three-dimensional coordinates for the frames and an uncertainty associated with the frames; and bundle adjusting the virtual key frames to obtain a complete three-dimensional reconstruction of the two-dimensional frames.
-
-
34. An apparatus for recovering a three-dimensional scene from a sequence of two-dimensional frames by segmenting the frames, comprising:
-
means for capturing two-dimensional images; means for dividing the sequence into segments comprising at least two frames; means for calculating a partial model for each segment that includes three-dimensional coordinates and camera pose for features within the frames of the segment, the three-dimensional coordinates and camera pose being derived from the frames of the segment; means for extracting virtual key frames from each partial model; and means for bundle adjusting the virtual key frames to obtain a complete three-dimensional reconstruction of the two-dimensional frames.
-
Specification