Method for determining camera position from two-dimensional images that form a panorama

• US 7,565,029 B2
• Filed: 07/08/2005
• Issued: 07/21/2009
• Est. Priority Date: 07/08/2005
• Status: Active Grant

• Current Assignee: Seiko Epson Corporation (Seiko Group)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Kwok, Derek Hang Chun, Clarke, Ian, Zhou, Hui
• Primary Examiner(s): Strege; John B
• Application Number:
  US11/177,811
• Publication Number:
  US 20070008312A1
• Time in Patent Office: 1,474 Days
• Field of Search: 382/284, 382/285, 382/294, 382/154
• US Class Current: 382/284
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Abstract

A method of estimating three-dimensional camera position information from a series of two-dimensional images that form a panorama employs common features in adjoining image pairs in the series to estimate a transform between the images in the pairs. The common features are subsequently employed to adjust an estimated rotational component of each transform by reducing error between coordinates corresponding to the common features in three-dimensional space in image pairs, on a pair-by-pair basis. A global optimization of the position estimation, used for long sequences of images such as 360 degree panoramas, refines the estimates of the rotational and focal length components of the transforms by concurrently reducing error between all 3D common feature coordinates for all adjoining pairs.

Claims

1. A method of estimating camera position information from a series of source images that form a panorama, the method comprising:

using a processor to carry out the steps of;

estimating a transform between adjoining image pairs in said series based on common features within said adjoining image pairs;

determining a rotational component for each transform and three-dimensional coordinates for said common features;

adjusting each of the rotational components to reduce errors between corresponding ones of said coordinates in adjoining image pairs; and

calculating focal lengths based on the adjusted rotational components;

wherein said determining is done on the basis of an initial focal length;

in the event that predetermined conditions are satisfied, concurrently refining the focal lengths and rotational components to globally reduce errors between corresponding ones of said coordinates in all image pairs;

prior to said refining, adjusting a position of each of said images to an average latitude of all images in said series;

setting the y-component of the x-axis of each image to zero; and

setting the y-axis of each image to the cross-product of said x-axis and the respective image normal.

2. The method of claim 1, wherein said determining is done on the basis that said source images are evenly spaced across said panorama.

3. The method of claim 1, wherein said common features are matching corners.

4. The method of claim 3, wherein said transform is a projective transform.

5. The method of claim 1, wherein said initial estimate of focal length for each image is a respective image diagonal length calculated according to:

$\mathrm{Diagonal\_Length}=\sqrt{{\left(\mathrm{IMAGE\_Width}\right)}^2+{\left(\mathrm{IMAGE\_Height}\right)}^2}$

6. The method of claim 1, wherein three-dimensional image parameters for use during said determining are calculated based on said initial estimate of focal length.

7. The method of claim 6, wherein said three-dimensional image parameters comprise image center point, image vertices, image normal, x-axis and y-axis parameters.

12. The method of claim 1, wherein said predetermined conditions comprise at least one of:

said series of source images form a 360 degree panorama;

said series of source images form greater than one row with an extent greater than one; and

said series of source images form greater than one column with an extent greater than one.

8. A method of estimating camera position information from a series of source images that form a panorama, the method comprising:

using a processor to carry out the steps of;

estimating a transform between adjoining image pairs in said series based on common features within said adjoining image pairs;

determining a rotational component for each transform and three-dimensional coordinates for said common features;

adjusting each of the rotational components to reduce errors between corresponding ones of said coordinates in adjoining image pairs; and

calculating focal lengths based on the adjusted rotational components;

wherein in the event that predetermined conditions are satisfied, concurrently refining the focal lengths and rotational components to globally reduce errors between corresponding ones of said coordinates in all image pairs;

prior to said refining, adjusting a position of each of said images to an average latitude of all images in said series;

setting the v-component of the x-axis of each image to zero; and

setting the v-axis of each image to the cross-product of said x-axis and the respective image normal;

wherein said determining is done on the basis of an initial focal length;

prior to said adjusting, placing a reference image onto an equator of a three-dimensional model, at a distance equal to a respective initial estimate of focal length from an origin of said model; and

sequentially placing adjoining images into said model based on respective initial estimates of focal length and rotational component and identifying rays from said origin to corresponding ones of said coordinates in adjoining images;

wherein said adjusting comprises minimizing error in direction between corresponding ones of said rays.

9. The method of claim 8, wherein said error in direction is the Euclidean distance between corresponding ones of said coordinates, according to:

$q<br><br>\left(\begin{array}{c}{x}_r\\ y_r\\ z_r\end{array}\right)=\left(\begin{array}{c}{x}_c\\ y_c\\ z_c\end{array}\right)+s<br><br>\left(\begin{array}{c}{x}_A\\ y_A\\ z_A\end{array}\right)+t<br><br>\left(\begin{array}{c}{x}_B\\ y_B\\ z_B\end{array}\right)$

where (X_r y_r Z_r)^T is the normalized vector of a ray to a coordinate in a first adjoining image in one of said pairs;

(x_c y_c z_c)^T is the three-dimensional center of a second adjoining image in said one pair;

(X_A Y_A Z_A)^T is the x-axis of said second adjoining image; and

(XB y z8 )T is the y-axis of said second adjoining image.

10. The method of claim 9, wherein center points of all images in a common row are assumed to have a common latitude.

11. The method of claim 10, wherein said placing and adjusting steps are performed iteratively a predetermined number of times.

13. The method of 1, wherein said refining comprises correcting focal lengths, and rotation angles for each of three-dimensions.

14. An apparatus for estimating three dimensional camera position information from a set of two dimensional images forming a panorama, the device comprising:

a transform estimator estimating a transform between adjoining image pairs in said series based on common features within said adjoining image pairs;

a rotational component determinor determining a rotational component for each transform and three-dimensional coordinates for said common features;

an adjustor adjusting each of the rotational components to reduce errors between corresponding ones of said coordinates in adjoining image pairs; and

a focal length calculator calculating focal lengths based on the adjusted rotational components

wherein said determining is done on the basis of an initial focal length;

in the event that predetermined conditions are satisfied, concurrently refining the focal lengths and rotational components to globally reduce errors between corresponding ones of said coordinates in all image pairs;

prior to said refining, adjusting a position of each of said images to an average latitude of all images in said series;

setting the y-component of the x-axis of each image to zero; and

setting the v-axis of each image to the cross-product of said x-axis and the respective image normal.