Plane-based self-calibration for structure from motion

• US 9,552,639 B2
• Filed: 10/15/2015
• Issued: 01/24/2017
• Est. Priority Date: 08/19/2011
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Jin, Hailin, Zhou, Zihan
• Primary Examiner(s): Bayat, Ali
• Application Number:
  US14/884,338
• Publication Number:
  US 20160104286A1
• Time in Patent Office: 467 Days
• Field of Search: 382/103
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 13/20 Image signal generators

Abstract

Robust techniques for self-calibration of a moving camera observing a planar scene. Plane-based self-calibration techniques may take as input the homographies between images estimated from point correspondences and provide an estimate of the focal lengths of all the cameras. A plane-based self-calibration technique may be based on the enumeration of the inherently bounded space of the focal lengths. Each sample of the search space defines a plane in the 3D space and in turn produces a tentative Euclidean reconstruction of all the cameras that is then scored. The sample with the best score is chosen and the final focal lengths and camera motions are computed. Variations on this technique handle both constant focal length cases and varying focal length cases.

Claims

1. A method, comprising:

generating, by one or more hardware computing devices having at least one processor executing instructions, a reconstruction for an image sequence comprising a plurality of frames according to a set of inter-image homographies estimated from point correspondences of structures in an entire scene for the image sequence, wherein each inter-image homography represents a projective transformation from one image in the sequence to another image in the sequence, wherein said generating comprises;

estimating at least one reconstruction at each of two or more focal lengths according to the set of inter-image homographies, wherein each reconstruction comprises camera motion and camera intrinsic parameters at each frame of the image sequence and a plane normal for the image sequence; and

selecting one of the reconstructions that best fits the set of inter-image homographies as the reconstruction for the image sequence.

2. The method as recited in claim 1, wherein the camera motion includes rotation and translation, and wherein the camera intrinsic parameters include focal length.

3. The method as recited in claim 1, wherein said estimating at least one reconstruction at each of the two or more focal lengths according to the set of inter-image homographies comprises, for each of the two or more focal lengths:

solving for a plane normal at the focal length according to the set of inter-image homographies, wherein said solving generates two solutions for the plane normal; and

estimating a reconstruction for each solution to the plane normal at the focal length.

4. The method as recited in claim 3, wherein said estimating a reconstruction for each solution to the plane normal at the focal length comprises, for each solution for the plane normal:

estimating camera motion for the frames according to the solution for the plane normal and the focal length;

refining the estimated camera motion, the focal length, and the plane normal across all the frames to generate a reconstruction for the solution for the plane normal at the focal length; and

calculating a score for the reconstruction according to how well the reconstruction fits the set of inter-image homographies.

5. The method as recited in claim 4, wherein said refining is performed according to a non-linear least-squares technique.

6. The method as recited in claim 1, wherein said estimating at least one reconstruction at each of the two or more focal lengths according to the set of inter-image homographies comprises:

partitioning the image sequence into a plurality of segments each comprising at least two consecutive frames; and

for each of the two or more focal lengths;

solving for a plane normal at the focal length for a first segment of the image sequence according to the respective inter-image homographies, wherein said solving generates two solutions for the plane normal at the first segment; and

estimating a reconstruction for each solution for the plane normal at the first segment, wherein said estimating comprises;

estimating camera motion for the first segment according to the solution for the plane normal and the focal length and refining the estimated camera motions, the focal length, and the plane normal for the first segment; and

for each subsequent segment, estimating camera motion for the segment according to the refined focal length at the previous segment and refining the estimated camera motion, focal length, and plane normal for the segment.

7. The method as recited in claim 1, wherein said estimating at least one reconstruction at each of the two or more focal lengths according to the set of inter-image homographies comprises, for at least two combinations of a focal length at a first frame and a focal length at a last frame in the video sequence:

solving for a plane normal according to the set of inter-image homographies and the combination of focal lengths at the first frame and the last frame, wherein said solving generates two solutions for the plane normal; and

for each solution for the plane normal;

estimating focal length at each of the frames between the first frame and the last frame according to the set of inter-image homographies;

estimating camera motion for the frames according to the estimated focal lengths;

refining the estimated camera motion, the focal length, and the plane normal across all the frames to generate a reconstruction for the solution to the plane normal; and

calculating a score for the reconstruction according to how well the reconstruction fits the set of inter-image homographies.

8. A system, comprising:

at least one processor; and

a non-transitory memory comprising program instructions, wherein the program instructions are executable by at least one of the one or more processors to generate a reconstruction for an image sequence comprising a plurality of frames according to a set of inter-image homographies estimated from point correspondences of structures in an entire scene for the image sequence, wherein each inter-image homography represents a projective transformation from one image in the sequence to another image in the sequence, wherein to generate the reconstruction for the image sequence, the program instructions are executable by the at least one processor to;

estimate at least one reconstruction at each of two or more focal lengths according to the set of inter-image homographies, wherein each reconstruction comprises camera motion and camera intrinsic parameters at each frame of the image sequence and a plane normal for the image sequence; and

select one of the reconstructions that best fits the set of inter-image homographies as the reconstruction for the image sequence.

9. The system as recited in claim 8, wherein the camera motion includes rotation and translation, and wherein the camera intrinsic parameters include focal length.

10. The system as recited in claim 8, wherein, to estimate at least one reconstruction at each of the two or more focal lengths according to the set of inter-image homographies, the program instructions are executable by the at least one processor to, for each of the two or more focal lengths:

solve for a plane normal at the focal length according to the set of inter-image homographies, wherein said solving generates two solutions for the plane normal; and

estimate a reconstruction for each solution to the plane normal at the focal length.

11. The system as recited in claim 10, wherein, to estimate a reconstruction for each solution to the plane normal at the focal length, the program instructions are executable by the at least one processor to, for each solution for the plane normal:

estimate camera motion for the frames according to the solution for the plane normal and the focal length;

refine the estimated camera motion, the focal length, and the plane normal across all the frames to generate a reconstruction for the solution for the plane normal at the focal length; and

calculate a score for the reconstruction according to how well the reconstruction fits the set of inter-image homographies.

12. The system as recited in claim 11, wherein said refining is performed according to a non-linear least-squares technique.

13. The system as recited in claim 8, wherein, to estimate at least one reconstruction at each of the two or more focal lengths according to the set of inter-image homographies, the program instructions are executable by the at least one processor to:

partition the image sequence into a plurality of segments each comprising at least two consecutive frames; and

for each of the two or more focal lengths;

solve for a plane normal at the focal length for a first segment of the image sequence according to the respective inter-image homographies, wherein said solving generates two solutions for the plane normal at the first segment; and

estimate a reconstruction for each solution for the plane normal at the first segment, wherein, to estimate the reconstruction for the solution, the program instructions are executable by the at least one processor to;

estimate camera motion for the first segment according to the solution for the plane normal and the focal length and refining the estimated camera motions, the focal length, and the plane normal for the first segment; and

for each subsequent segment, estimate camera motion for the segment according to the refined focal length at the previous segment and refine the estimated camera motion, focal length, and plane normal for the segment.

14. The system as recited in claim 8, wherein, to estimate at least one reconstruction at each of the two or more focal lengths according to the set of inter-image homographies, the program instructions are executable by the at least one processor to, for at least two combinations of a focal length at a first frame and a focal length at a last frame in the video sequence:

solve for a plane normal according to the set of inter-image homographies and the combination of focal lengths at the first frame and the last frame, wherein said solving generates two solutions for the plane normal;

for each solution for the plane normal;

estimate focal length at each of the frames between the first frame and the last frame according to the set of inter-image homographies;

estimate camera motion for the frames according to the estimated focal lengths;

refining the estimated camera motion, the focal length, and the plane normal across all the frames to generate a reconstruction for the solution to the plane normal; and

calculate a score for the reconstruction according to how well the reconstruction fits the set of inter-image homographies.

15. A non-transitory computer-readable storage medium storing program instructions, wherein the program instructions are computer-executable to implement:

generating a reconstruction for an image sequence comprising a plurality of frames according to a set of inter-image homographies estimated from point correspondences of structures in an entire scene for the image sequence, wherein each inter-image homography represents a projective transformation from one image in the sequence to another image in the sequence, wherein said generating comprises;

estimating at least one reconstruction at each of two or more focal lengths according to the set of inter-image homographies, wherein each reconstruction comprises camera motion and camera intrinsic parameters at each frame of the image sequence and a plane normal for the image sequence; and

selecting one of the reconstructions that best fits the set of inter-image homographies as the reconstruction for the image sequence.

16. The non-transitory computer-readable storage medium as recited in claim 15, wherein the camera motion includes rotation and translation, and wherein the camera intrinsic parameters include focal length.

17. The non-transitory computer-readable storage medium as recited in claim 15, wherein, in said estimating at least one reconstruction at each of the two or more focal lengths according to the set of inter-image homographies, the program instructions are computer-executable to implement, for each of the two or more focal lengths:

solving for a plane normal at the focal length according to the set of inter-image homographies, wherein said solving generates two solutions for the plane normal; and

estimating a reconstruction for each solution to the plane normal at the focal length.

18. The non-transitory computer-readable storage medium as recited in claim 17, wherein, in said estimating a reconstruction for each solution to the plane normal at the focal length, the program instructions are computer-executable to implement, for each solution for the plane normal:

estimating camera motion for the frames according to the solution for the plane normal and the focal length;

refining the estimated camera motion, the focal length, and the plane normal across all the frames to generate a reconstruction for the solution for the plane normal at the focal length; and

calculating a score for the reconstruction according to how well the reconstruction fits the set of inter-image homographies.

19. The non-transitory computer-readable storage medium as recited in claim 15, wherein, in said estimating at least one reconstruction at each of the two or more focal lengths according to the set of inter-image homographies, the program instructions are computer-executable to implement:

partitioning the image sequence into a plurality of segments each comprising at least two consecutive frames; and

for each of the two or more focal lengths;

solving for a plane normal at the focal length for a first segment of the image sequence according to the respective inter-image homographies, wherein said solving generates two solutions for the plane normal at the first segment; and

estimating a reconstruction for each solution for the plane normal at the first segment, wherein said estimating comprises;

estimating camera motion for the first segment according to the solution for the plane normal and the focal length and refining the estimated camera motions, the focal length, and the plane normal for the first segment; and

for each subsequent segment, estimating camera motion for the segment according to the refined focal length at the previous segment and refining the estimated camera motion, focal length, and plane normal for the segment.

20. The non-transitory computer-readable storage medium as recited in claim 15, wherein, in said estimating at least one reconstruction at each of the two or more focal lengths according to the set of inter-image homographies, the program instructions are computer-executable to implement, for at least two combinations of a focal length at a first frame and a focal length at a last frame in the video sequence:

solving for a plane normal according to the set of inter-image homographies and the combination of focal lengths at the first frame and the last frame, wherein said solving generates two solutions for the plane normal; and

for each solution for the plane normal;

estimating focal length at each of the frames between the first frame and the last frame according to the set of inter-image homographies;

estimating camera motion for the frames according to the estimated focal lengths;

refining the estimated camera motion, the focal length, and the plane normal across all the frames to generate a reconstruction for the solution to the plane normal; and

calculating a score for the reconstruction according to how well the reconstruction fits the set of inter-image homographies.