Plane detection and tracking for structure from motion

• US 9,747,699 B2
• Filed: 08/07/2015
• Issued: 08/29/2017
• Est. Priority Date: 08/19/2011
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Jin, Hailin, Zhou, Zihan
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): BEG, SAMAH A
• Application Number:
  US14/821,441
• Publication Number:
  US 20150348265A1
• Time in Patent Office: 753 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 13/20 Image signal generators

Abstract

Plane detection and tracking algorithms are described that may take point trajectories as input and provide as output a set of inter-image homographies. The inter-image homographies may, for example, be used to generate estimates for 3D camera motion, camera intrinsic parameters, and plane normals using a plane-based self-calibration algorithm. A plane detection and tracking algorithm may obtain a set of point trajectories for a set of images (e.g., a video sequence, or a set of still photographs). A 2D plane may be detected from the trajectories, and trajectories that follow the 2D plane through the images may be identified. The identified trajectories may be used to compute a set of inter-image homographies for the images as output.

Claims

1. A method of camera self-calibration for an image sequence implemented by a computing device, comprising:

receiving, by the computing device, the image sequence comprising a plurality of frames;

determining, by the computing device, a set of inter-image homographies for the image sequence based on inlier point trajectories that follow a plane through at least one frame pair of the image sequence;

estimating, by the computing device, structure from motion and camera parameters for a plurality of focal lengths of a camera, the plurality of focal lengths generated based on the plurality of frames;

calculating, by the computing device, scores for the plurality of focal lengths based on how well the structure from motion and camera parameters for the respective focal lengths fit the set of inter-image homographies;

selecting, by the computing device, a particular focal length from the plurality of focal lengths based at least in part on the calculated scores for the respective focal lengths; and

outputting, by the computing device, a particular structure from motion and camera parameters corresponding to the particular focal length.

2. The method of claim 1, further comprising computing, by the computing device, at least one plane normal;

and

wherein the estimating the structure from motion and camera parameters is performed for the at least one plane normal.

3. The method of claim 2, wherein the at least one plane normal comprises a first plane normal calculated by the computing device based on a first homography of the inter-image homographies and a second plane normal calculated by the computing device based on a second homography of the inter-image homographies.

4. The method of claim 2, further comprising:

refining, by the computing device, the estimated structure from motion and camera parameters for the plurality of focal lengths via a non-linear least squares technique.

5. The method of claim 1, wherein the determining the set of inter-image homographies for the image sequence comprises tracking, by the computing device, a single plane through the image sequence.

6. The method of claim 1, wherein the determining the set of inter-image homographies comprises tracking, by the computing device, a first plane through a first section of the image sequence and tracking, by the computing device, a second plane through a second section of the image sequence.

7. The method of claim 1, wherein the camera parameters comprise intrinsic parameters, rotation, and translation for each frame.

8. The method of claim 1, wherein the structure from motion comprises a Euclidean transformation between each frame.

9. The method of claim 1, wherein the structure from motion comprises a pinhole model of a projective camera.

10. The method of claim 1, wherein the calculating scores for the plurality of focal lengths utilizes a Levenberg-Marquardt (LM) non-linear optimization method.

11. A system, comprising:

at least one processor; and

a memory comprising program instructions, wherein the program instructions are executable by the at least one processor to perform camera self-calibration for an image sequence by;

receiving the image sequence comprising a plurality of frames;

determining a set of inter-image homographies for the image sequence based on inlier point trajectories that follow a plane through at least one frame pair of the image sequence;

estimating a plurality of focal lengths of a camera;

estimating structure from motion and camera parameters for the plurality of focal lengths;

calculating scores for the plurality of focal lengths based on how well the structure from motion and camera parameters for the respective focal lengths fit the set of inter-image homographies;

selecting a particular focal length from the plurality of focal lengths based at least in part on the calculated scores for the respective focal lengths; and

outputting a particular structure from motion and camera parameters corresponding to the particular focal length.

12. The system of claim 11, wherein the operations further comprise computing at least one plane normal;

and

wherein the estimating the structure from motion and camera parameters is performed for the at least one plane normal.

13. The system of claim 12, wherein the at least one plane normal comprises a first plane normal calculated based on a first homography of the inter-image homographies and a second plane normal calculated based on a second homography of the inter-image homographies.

14. The system of claim 12, wherein the operations further comprise

refining the estimated structure from motion and camera parameters for the plurality of focal lengths via a non-linear least squares technique.

15. The system of claim 11, wherein the determining the set of inter-image homographies for the image sequence comprises tracking a single plane through the image sequence.

16. The system of claim 11, wherein the determining the set of inter-image homographies for the image sequence comprises tracking a first plane through a first section of the image sequence and tracking a second plane through a second section of the image sequence.

17. The system of claim 11, wherein the structure from motion comprises a Euclidean transformation between each frame of the image sequence relative to a projective camera.

18. A method of plane-based camera self-calibration with varying focal length implemented by a computing device, comprising:

receiving, by the computing device, an image sequence comprising a plurality of frames;

generating, by the computing device, a set of inter-image homographies for the image sequence using a plane detection and tracking algorithm;

partitioning, by the computing device, the image sequence into a plurality of segments;

generating, by the computing device, a plurality of reconstructions by;

generating, by the computing device, a plurality of focal lengths of a camera based on the plurality of frames;

for each of the plurality of focal lengths;

setting, by the computing device, the focal length for a first segment of the plurality of segments and computing, by the computing device, at least one plane normal according to a homography for the first segment; and

for the at least one plane normal;

estimating, by the computing device, a camera motion for the first segment according to the plane normal;

refining, by the computing device, the camera motion for the first segment, camera parameters for the first segment, and the plane normal for the first segment according to the estimation for the first segment; and

for each subsequent segment of the plurality of segments;

 

setting, by the computing device, the focal length for the subsequent segment to a refined focal length corresponding to a previous segment;

 

estimating, by the computing device, a camera motion for the subsequent segment according to a refined plane normal corresponding to a previous segment; and

 

refining, by the computing device, the camera motion for the subsequent segment, camera parameters for the subsequent segment, and the plane normal for the subsequent segment according to the estimation for the subsequent segment; and

fitting, by the computing device, each of the reconstructions to the set of inter-image homographies for the image sequence;

calculating, by the computing device, a score for each of the reconstructions based on the fittings; and

outputting, by the computing device, a chosen reconstruction of the plurality of reconstructions based at least in part on the calculated scores.

19. The method of claim 18, wherein the refinement of the camera motions for the first and subsequent segments comprises a non-linear least squares technique.

20. The method of claim 18, wherein the reconstruction comprises a Euclidean transformation between each frame of the image sequence relative to a projective camera.