Apparatus and methods for image alignment

• US 9,973,696 B1
• Filed: 09/12/2016
• Issued: 05/15/2018
• Est. Priority Date: 11/23/2015
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Meler, Antoine, Derbanne, Thomas
• Primary Examiner(s): Brier, Jeffery A
• Application Number:
  US15/263,332
• Time in Patent Office: 610 Days
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: G06T 3/14 Transformations for image registration, e.g. adjusting or mapping for alignment of images
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2624 for obtaining an image which is composed of whole input images, e.g. splitscreen
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

Multiple images may be combined to obtain a composite image. Individual images may be obtained with different camera sensors and/or at different time instances. In order to obtain the composite image source images may be aligned in order to produce a seamless stitch. Source images may be characterized by a region of overlap. A disparity measure may be determined for pixels along a border region between the source images. A warp transformation may be determined using a refinement process configured to determine displacement of pixels of the border region based on the disparity. Pixel displacement at a given location may be constrained to direction configured tangential to an epipolar line corresponding to the location. The warp transformation may be propagated to pixels of the image. Spatial and/or temporal smoothing may be applied. In order to obtain refined solution, the warp transformation may be determined at multiple spatial scales.

Claims

1. A computerized system configured to obtain a composite image, the system comprising:

a capture device configured to obtain source images, the source images including a first source image and a second source image; and

one or more physical processors configured by machine-readable instructions to;

identify overlapping portions of the source images, such overlapping portions including a first portion comprised of a first plurality of pixels within the first source image and a second portion comprised of a second plurality of pixels within the second source image;

obtain a disparity measure for corresponding locations in the first source image and the second source image based on an evaluation of pixels of the first plurality of pixels and pixels of the second plurality of pixels at the corresponding locations;

apply a transformation operation to at least pixels within the overlapping portions of the source images to generate transformed source images, the transformation operation configured to displace pixels of one or both of the first plurality of pixels and/or the second plurality of pixels to reduce the disparity measure and determined based on a discrete refinement; and

obtain the composite image by combining the transformed source images.

2. The system of claim 1, wherein the first plurality of pixels and the second plurality of pixels form a boundary between the first source image and the second source image.

3. The system of claim 2, wherein the transformation operation is configured to displace one or more of the pixels of the first plurality of pixels disposed at the boundary by a first displacement along a first direction and one or more of the pixels of the second plurality of pixels disposed at the boundary by the first displacement along a second direction.

4. The system of claim 3, wherein for a given pixel of the one or more of the pixels of the first plurality of pixels and the pixels of the second plurality of pixels disposed at the boundary, the first direction and the second direction are being configured tangential to an epipolar line associated with the given pixel location, the epipolar line configured based on a geometry of a camera used to obtain the first source image.

5. The system of claim 4, wherein the direction is characterized by a first displacement component configured based on a trigonometric cosine operation of the epipolar line and a second displacement component configured based on a trigonometric sine operation of the epipolar line.

6. The system of claim 4, wherein:

the transformation operation comprises an extrapolation operation configured to displace at least one pixel disposed outside the boundary by a second displacement, the second displacement based on the first displacement and a distance between the given pixel and the at least one pixel disposed outside the boundary.

7. The system of claim 6, wherein the second displacement is reduced inversely with the distance.

8. The system of claim 6, wherein the extrapolation operation comprises a spatial smoothing operation.

9. The system of claim 2, wherein:

individual pixels of the first source image and the second source image are characterized by a brightness component and a chromaticity component; and

the one or more processors are further configured to;

apply a color correction operation configured to reduce difference between chromaticity components of pixels of the first source image at the boundary relative to chromaticity components of pixels of the second source image at the boundary; and

apply an exposure correction operation configured to reduce difference between brightness components of the pixels of the first source image at the boundary relative to brightness components of the pixels of the second source image at the boundary.

10. The system of claim 2, wherein:

the capture device comprises;

a first lens optically coupled to a first image sensor configured to obtain the first source image; and

a second lens optically coupled to a second image sensor configured to obtain the second source image; and

the boundary comprises an iso-line encompassing 180 degree field of view boundary of the first lens, and the first plurality of pixels and the second plurality of pixels form a boundary between the first source image and the second source image.

11. The system of claim 10, wherein:

a first field of view of the first lens is configured to point in a first direction;

a second field of view of the second lens is configured to point in a second direction opposing the first direction.

12. The system of claim 11, wherein:

the first lens and the second lens are characterized by a fish-eye lens pattern with a field of view of at least 180 degrees;

the first source image and the second source image cover at least 180×

180 degree field of view; and

the composite image is configured to cover 360×

180 degree field of view.

13. The system of claim 12, wherein:

the composite image is configured in an equirectangular plane characterized by latitudinal and longitudinal dimensions;

the boundary is configured at a half point along the latitudinal dimension.

14. The system of claim 1, wherein the transformation operation is determined further based on a continuous refinement.

15. The system of claim 14, wherein a result of the discrete refinement is used to initialize the continuous refinement.

16. The system of claim 15, wherein the discrete refinement and the continuous refinement is performed on a highest resolution.

17. The system of claim 2, wherein the transformation operation is configured to reduce a contrast measure within the composite image, the contrast measure being determined based on difference of chromaticity components of pixels of the first source image and pixels of the second source image at the boundary.

18. The system of claim 2, wherein the transformation operation is configured to reduce a contrast measure within the composite image, the contrast measure being determined based on difference of brightness components of pixels of the first source image and pixels of the second source image at the boundary.

19. A capture apparatus, comprising:

a first lens optically coupled to a first image sensor configured to obtain a first source image;

a second lens optically coupled to a second image sensor configured to obtain a second source image;

one or more physical processors in operable communication with the first image sensor and the second image sensor;

a non-transitory storage medium configured to store a plurality of computer instructions which when executed by the one or more physical processors;

identify a boundary region between the first source image and the second source image, the boundary region comprised of first pixels of the first source image and second pixels of the second source image;

obtain a disparity measure based on an evaluation of one or more of the first pixels and one or more of the second pixels disposed at a given location within the boundary region, the given location having an epipolar line associated therewith; and

apply a transformation operation configured to displace pixels within the boundary region along a direction configured tangential to the epipolar line for the given location, the transformation operation determined based on a discrete refinement;

wherein

the first lens is characterized by a first optical center location and the second lens is characterized by a second optical center location;

the epipolar line comprises a segment configured to connect the first optical center location, the second optical center location, and the given location; and

the transformation operation is configured to reduce the disparity measure.

20. The apparatus of claim 19, wherein

the first lens and the second lens are characterized by a fish eye field of view;

and

the plurality of computer instructions when executed by the one or more physical processors are further configured to convert the first source image and the second source image into an equirectangular plane so that the boundary region is transformed into a straight segment; and

the transformation operation is configured to be applied to an output of the conversion of the first source image and the second source image into the equirectangular plane.