In-camera panorama image stitching assistance

• US 9,307,165 B2
• Filed: 08/06/2009
• Issued: 04/05/2016
• Est. Priority Date: 08/08/2008
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm Technologies Incorporated (Qualcomm, Inc.)
• Inventors: Levy, Noam
• Primary Examiner(s): RUBIN, BLAKE J
• Application Number:
  US12/536,728
• Publication Number:
  US 20100033553A1
• Time in Patent Office: 2,434 Days
• Field of Search: 348/36
• US Class Current: 1/1
• CPC Class Codes: G03B 37/02 with scanning movement of lens or cameras
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06V 10/10 Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 25/41 Extracting pixel data from a plurality of image sensors simultaneously picking up an image, e.g. for increasing the field of view by combining the outputs of a plurality of sensors
• CPC Class Codes: H04N 5/2624 for obtaining an image which is composed of whole input images, e.g. splitscreen

Abstract

A camera that provides for a panorama mode of operation that stitches together two or more images to create a single image with a wide format. In panorama mode, a live view of a scene is transformed from rectangular coordinates to cylindrical coordinates and displayed by the camera. Also, an overlap portion between the previous image and the next image to be captured is characterized. In real time, after the previous image is captured, the overlap portion of the previous image is also transformed from rectangular coordinates into cylindrical coordinates. Next, the camera displays an overlay of at least a portion of the overlap portion onto the live view of the next image to be captured. And this overlay can assist the user in aligning the live view of the next image to be captured with the overlap portion of the previously captured image.

Claims

1. A method for panorama stitching of images of a scene, comprising:

capturing each pixel for a real-time view, wherein each real-time pixel corresponds to a particular pair of a plurality of rectangular coordinates;

transforming each pair of rectangular coordinates that corresponds to each pixel of the real-time view into a particular pair of a plurality of cylindrical coordinates, wherein each transformed pair of cylindrical coordinates corresponds to each pixel in another real-time view of the scene and wherein the transforming is performed with respect to a line of horizon of the rectangular coordinates that determines the origin of the cylindrical coordinates;

enabling a user to adjust a vertical position of the line of horizon for the real time view;

adjusting each transformed pair of cylindrical coordinates if the vertical position of the line of horizon for the real time view is adjusted by the user;

displaying the other real-time view that corresponds to the pairs of cylindrical coordinates;

capturing each pixel for a first image of the scene, wherein each pixel of the first image corresponds to a particular pair of the plurality of rectangular coordinates;

based on the adjusted vertical position of the line of horizon, transforming each pair of rectangular coordinates for each pixel in at least a portion of the first image into a particular pair of cylindrical coordinates, wherein the portion of the first image with a transformed pair of cylindrical coordinates for each pixel is saved; and

displaying simultaneously the portion of the transformed first image and the other real-time view based on the transformed pairs of cylindrical coordinates to enable the user to align the portion of the transformed first image with the other real-time view.

2. The method of claim 1, wherein transforming each pair of rectangular coordinates that correspond to each pixel of the real-time view into a particular pair of a plurality of cylindrical coordinates includes:

for each of the plurality of cylindrical coordinates, determining correspondence with each real-time pixel in the rectangular coordinates based on interpolation by looking up each cylindrical coordinate in a look-up table and employing an interpolation device to perform interpolation to determine a corresponding coordinate in the rectangular coordinate system.

3. The method of claim 1, wherein transforming each pair of rectangular coordinates for each pixel in at least a portion of the first image into a particular pair of cylindrical coordinates includes:

for each of the cylindrical coordinates in the portion of the first transformed image, determining correspondence with each pair of rectangular coordinates for each pixel in the portion of the first image by looking up each cylindrical coordinate in a look-up table and employing an interpolation device to perform interpolation to determine a corresponding coordinate in the rectangular coordinate system.

4. The method of claim 1, further comprising:

making a determination as to whether the other real-time view is aligned with the portion of the transformed first image; and

if it is determined that the other real-time view is roughly aligned with the portion of the transformed first image, either automatically taking a shot or providing an indication to take the shot.

5. The method of claim 1, further comprising:

capturing each pixel for a second image of the scene, wherein each pixel of the second image corresponds to a particular pair of the plurality of rectangular coordinates;

transforming each pair of rectangular coordinates for each pixel in at least a portion of the second image into a particular pair of cylindrical coordinates, wherein the portion of the second image with a transformed pair of cylindrical coordinates for each pixel is saved; and

displaying simultaneously the portion of the transformed second image and the other real-time view based on the transformed pairs of cylindrical coordinates to enable alignment of the portion of the transformed second image with the other real-time view.

6. The method of claim 5, further comprising:

performing panoramic stitching to generate a panoramic mosaic based on at least the first image and the second image.

7. The method of claim 1, further comprising:

making a determination as to whether the other real-time view is aligned with the portion of the transformed first image; and

if it is determined that the other real-time view is aligned with the portion of the transformed first image, providing an indication to take a shot.

8. The method of claim 7, further comprising:

if it is determined that the other real-time view is unaligned with the portion of the transformed first image, providing an indication as to how to move a camera to align the other real-time view with the portion of the transformed first image.

9. The method of claim 7, wherein making the determination includes:

employing a random sample consensus algorithm to determine how many correspondences exist between an overlap region of the portion of the first transformed image and an overlap region of the other real-time view;

making a determination as to whether the number of correspondences is greater than a first predetermined threshold;

calculating an average absolute displacement between pixel values of the overlap region of the portion of the first transformed image and the overlap region of the other real-time view;

making a determination as to whether the average absolute displacement is less than a second predetermined threshold; and

if the number of correspondences is greater than the first predetermined threshold and the average absolute displacement is less than the second predetermined threshold, making a positive determination that the other real-time view is roughly aligned with the portion of the first transformed image.

10. The method of claim 9, wherein

the overlap region is a predetermined percentage of the transformed first image.

11. The method of claim 9, further comprising:

determining how much of the transformed first image to use as the overlap region, by determining a minimum width for which a region of the minimum width contains a predetermined number of corners.

12. A device for panorama stitching of images of a scene, comprising:

an image capture device that is arranged to capture an image; and

a processor that is configured to enable actions, including;

controlling the image capture device to capture each pixel for a real-time view, wherein each real-time pixel corresponds to a particular pair of a plurality of rectangular coordinates;

transforming each pair of rectangular coordinates that correspond to each pixel of the real-time view into a particular pair of a plurality of cylindrical coordinates, wherein each transformed pair of cylindrical coordinates corresponds to each pixel in another real-time view of the scene and wherein the transforming is performed with respect to a line of horizon of the rectangular coordinates that determines the origin of the cylindrical coordinates;

enabling a user to adjust a vertical position of the line of horizon for the real time view;

adjusting each transformed pair of cylindrical coordinates if the vertical position of the line of horizon for the real time view is adjusted by the user;

controlling display of the other real-time view that corresponds to the pairs of cylindrical coordinates;

controlling the image capture device to capture each pixel for a first image of the scene, wherein each pixel of the first image corresponds to a particular pair of the plurality of rectangular coordinates;

based on the adjusted vertical position of the line of horizon, transforming each pair of rectangular coordinates for each pixel in at least a portion of the first image into a particular pair of cylindrical coordinates, wherein the portion of the first image with a transformed pair of cylindrical coordinates for each pixel is saved; and

controlling simultaneous display of the portion of the transformed first image and the other real-time view based on the transformed pairs of cylindrical coordinates to enable alignment of the portion of the transformed first image with the other real-time view.

13. The device of claim 12, further comprising:

a look-up table; and

an interpolation device that is arranged to perform interpolation on values in the look-up table,

wherein the processor is arranged to enable transforming each pair of rectangular coordinates that correspond to each pixel of the real-time view into a particular pair of a plurality of cylindrical coordinates by controlling the interpolation device to;

look up each cylindrical coordinate in the look-up table and perform interpolation to determine a corresponding coordinate in the rectangular coordinate system.

14. The device of claim 12, further comprising:

a look-up table; and

an interpolation device that is arranged to perform interpolation on values in the look-up table,

wherein the processor is arranged to enable transforming each pair of rectangular coordinates for each pixel in at least the portion of the first image into a particular pair of cylindrical coordinates by controlling the interpolation device to;

look up each cylindrical coordinate in the look-up table and performing interpolation to determine a corresponding coordinate in the rectangular coordinate system.

15. The device of claim 12, wherein

the processor is configured to enable further actions, including:

controlling the image capture device to capture each pixel for a second image of the scene, wherein each pixel of the second image corresponds to a particular pair of the plurality of rectangular coordinates;

transforming each pair of rectangular coordinates for each pixel in at least a portion of the second image into a particular pair of cylindrical coordinates, wherein the portion of the second image with a transformed pair of cylindrical coordinates for each pixel is saved; and

controlling simultaneous display of the portion of the transformed second image and the other real-time view based on the transformed pairs of cylindrical coordinates to enable alignment of the portion of the transformed second image with the other real-time view.

16. The device of claim 12, wherein

the processor is configured to perform further actions, including:

making a determination as to whether the other real-time view is aligned with the portion of the transformed first image; and

if it is determined that the other real-time view is aligned with the portion of the transformed first image, providing an indication to take a shot.

17. The device of claim 16, wherein

the processor is configured to perform further actions, including:

if it is determined that the other real-time view is unaligned with the portion of the transformed first image, providing an indication as to how to move the image capture device to align the other real-time view with the portion of the transformed first image.

18. The device of claim 16, wherein

the processor is configured to make the determination such that making the determination includes:

employing a random sample consensus algorithm to determine how many correspondences exist between an overlap region of the portion of the first transformed image and an overlap region of the other real-time view;

making a determination as to whether the number of correspondences is greater than a first predetermined threshold;

calculating an average absolute displacement between pixel values of the overlap region of the portion of the first transformed image and the overlap region of the other real-time view;

making a determination as to whether the average absolute displacement is less than a second predetermined threshold; and

if the number of correspondences is greater than the first predetermined threshold and the average absolute displacement is less than the second predetermined threshold, making a positive determination that the other real-time view is roughly aligned with the portion of the first transformed image.

19. An article of manufacture including a processor-readable storage medium having processor-executable code encoded therein, which when executed by one or more processors, enables actions for panorama stitching of images of a scene, comprising:

capturing each pixel for a real-time view, wherein each real-time pixel corresponds to a particular pair of a plurality of rectangular coordinates;

transforming each pair of rectangular coordinates that correspond to each pixel of the real-time view into a particular pair of a plurality of cylindrical coordinates, wherein each transformed pair of cylindrical coordinates corresponds to each pixel in another real-time view of the scene and wherein the transforming is performed with respect to a line of horizon of the rectangular coordinates that determines the origin of the cylindrical coordinates;

enabling a user to adjust a vertical position of the line of horizon for the real time view;

adjusting each transformed pair of cylindrical coordinates if the vertical position of the line of horizon for the real time view is adjusted by the user;

displaying the other real-time view that corresponds to the pairs of cylindrical coordinates;

capturing each pixel for a first image of the scene, wherein each pixel of the first image corresponds to a particular pair of the plurality of rectangular coordinates;

based on the adjusted vertical position of the line of horizon, transforming each pair of rectangular coordinates for each pixel in at least a portion of the first image into a particular pair of cylindrical coordinates, wherein the portion of the first image with a transformed pair of cylindrical coordinates for each pixel is saved; and

displaying simultaneously the portion of the transformed first image and the other real-time view based on the transformed pairs of cylindrical coordinates to enable a user to align the portion of the transformed first image with the other real-time view.

20. The article of manufacture of claim 19, wherein

the processor-readable code enables transforming each pair of rectangular coordinates that correspond to each pixel of the real-time view into a particular pair of a plurality of cylindrical coordinates by controlling an interpolation device to look up each cylindrical coordinate in the look-up table and perform interpolation to determine a corresponding coordinate in the rectangular coordinate system.