Apparatus and methods for image alignment

• US 10,498,958 B2
• Filed: 05/14/2018
• Issued: 12/03/2019
• Est. Priority Date: 11/23/2015
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Meler, Antoine, Derbanne, Thomas
• Primary Examiner(s): Brier, Jeffery A
• Application Number:
  US15/978,714
• Publication Number:
  US 20180262685A1
• Time in Patent Office: 568 Days
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: G06T 3/14 Transformations for image registration, e.g. adjusting or mapping for alignment of images
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2624 for obtaining an image which is composed of whole input images, e.g. splitscreen
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

Multiple images may be combined to obtain a composite image. Individual images may be obtained with different camera sensors and/or at different time instances. In order to obtain the composite image source images may be aligned in order to produce a seamless stitch. Source images may be characterized by a region of overlap. A disparity measure may be determined for pixels along a border region between the source images. A warp transformation may be determined using a refinement process configured to determine displacement of pixels of the border region based on the disparity. Pixel displacement at a given location may be constrained to direction configured tangential to an epipolar line corresponding to the location. The warp transformation may be propagated to pixels of the image. Spatial and/or temporal smoothing may be applied. In order to obtain refined solution, the warp transformation may be determined at multiple spatial scales.

Claims

1. A system that obtains a composite image, the system comprising:

one or more physical processors configured by machine-readable instructions to;

obtain source images, the source images including a first source image and a second source image;

identify overlapping portions of the source images, such overlapping portions including a first portion of the first source image and a second portion of the second source image;

obtain a disparity measure based on an evaluation of one or more pixels within the first portion of the first source image and one or more pixels within the second portion of the second source image;

apply a transformation operation to one or more pixels within the overlapping portions of the source images to generate transformed source images, the transformation operation configured to displace the one or more pixels to reduce the disparity measure and determined based on a discrete refinement; and

obtain the composite image based on the transformed source images.

2. The system of claim 1, wherein the one or more pixels within the first portion and the one or more pixels within the second portion form a boundary between the first source image and the second source image.

3. The system of claim 2, wherein the transformation operation is configured to displace one or more pixels of the first source image disposed at the boundary by a first displacement along a first direction and one or more pixels of the second source image disposed at the boundary by the first displacement along a second direction.

4. The system of claim 3, wherein for a given pixel that is displaced, the first direction and the second direction are tangential to an epipolar line associated with a location of the given pixel, the epipolar line configured based on a geometry of a capture device used to obtain the first source image.

5. The system of claim 4, wherein the direction is characterized by a first displacement component configured based on a trigonometric cosine operation of the epipolar line and a second displacement component configured based on a trigonometric sine operation of the epipolar line.

6. The system of claim 4, wherein:

the transformation operation comprises an extrapolation operation configured to displace at least one pixel disposed outside the boundary by a second displacement, the second displacement determined based on the first displacement and a distance between the given pixel and the at least one pixel disposed outside the boundary.

7. The system of claim 6, wherein the second displacement is reduced inversely with the distance.

8. The system of claim 6, wherein the extrapolation operation comprises a spatial smoothing operation.

9. The system of claim 2, wherein:

individual pixels of the first source image and the second source image are characterized by a brightness component and a chromaticity component; and

the one or more processors are further configured to;

apply a color correction operation configured to reduce difference between chromaticity components of pixels of the first source image at the boundary relative to chromaticity components of pixels of the second source image at the boundary; and

apply an exposure correction operation configured to reduce difference between brightness components of the pixels of the first source image at the boundary relative to brightness components of the pixels of the second source image at the boundary.

10. The system of claim 2, wherein:

the source images are captured by a capture device, the capture device comprising;

a first lens optically coupled to a first image sensor configured to obtain the first source image; and

a second lens optically coupled to a second image sensor configured to obtain the second source image; and

the boundary comprises an iso-line encompassing 180 degree field of view boundary of the first lens.

11. The system of claim 10, wherein:

a first field of view of the first lens is configured to point in a first direction;

a second field of view of the second lens is configured to point in a second direction opposing the first direction.

12. The system of claim 11, wherein:

the first lens and the second lens are characterized by a fish-eye lens pattern with a field of view of at least 180 degrees;

the first source image and the second source image cover at least 180×

180 degree field of view; and

the composite image is configured to cover 360×

180 degree field of view.

13. The system of claim 12, wherein:

the composite image is configured in an equirectangular plane characterized by latitudinal and longitudinal dimensions;

the boundary is configured at a half point along the latitudinal dimension.

14. The system of claim 1, wherein the transformation operation is determined further based on a continuous refinement.

15. The system of claim 14, wherein a result of the discrete refinement is used to initialize the continuous refinement.

16. The system of claim 15, wherein the discrete refinement and the continuous refinement is performed on a highest resolution.

17. The system of claim 2, wherein the transformation operation is configured to reduce a contrast measure within the composite image, the contrast measure being determined based on difference of chromaticity components of pixels of the first source image and pixels of the second source image at the boundary.

18. The system of claim 2, wherein the transformation operation is configured to reduce a contrast measure within the composite image, the contrast measure being determined based on difference of brightness components of pixels of the first source image and pixels of the second source image at the boundary.

19. A capture apparatus, comprising:

a first lens optically coupled to a first image sensor configured to obtain a first source image;

a second lens optically coupled to a second image sensor configured to obtain a second source image;

one or more physical processors in operable communication with the first image sensor and the second image sensor;

a non-transitory storage medium configured to store computer instructions which, when executed, cause the one or more physical processors to perform;

identifying a boundary region between the first source image and the second source image, the boundary region comprised of a first portion of the first source image and a second portion of the second source image;

obtaining a disparity measure based on an evaluation of one or more pixels within the first portion and one or more pixels within the second portion disposed at a given location within the boundary region, the given location having an epipolar line associated therewith; and

applying a transformation operation to displace pixels within the boundary region along a direction tangential to the epipolar line for the given location, the transformation operation determined based on a discrete refinement;

wherein

the first lens is characterized by a first optical center location and the second lens is characterized by a second optical center location;

the epipolar line comprises a segment configured to connect the first optical center location, the second optical center location, and the given location; and

the transformation operation reduces the disparity measure.

20. The apparatus of claim 19, wherein

the first lens and the second lens are characterized by a fish eye field of view;

and

the computer instructions, when executed, further cause the one or more physical processors to perform converting the first source image and the second source image into an equirectangular plane so that the boundary region is transformed into a straight segment; and

the transformation operation is applied to an output of the conversion of the first source image and the second source image into the equirectangular plane.