Adaptive image blending operations

• US 9,098,922 B2
• Filed: 06/06/2012
• Issued: 08/04/2015
• Est. Priority Date: 06/06/2012
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Tico, Marius
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): Zhou, Zhihan
• Application Number:
  US13/490,183
• Publication Number:
  US 20130329002A1
• Time in Patent Office: 1,154 Days
• Field of Search: 348/36, 348/218.1, 348/222.1, 348/239, 382/284, 382/103, 382/154
• US Class Current: 1/1
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Special blend operations (referred to as “image seam-matching”) are described that keep the pixel values in the two images being blended the same along their transition border or seam, and smoothly increases/decreases pixel values on either side of the seam through the images'"'"' transition band (an area around the seam in which image blend operations are constrained). Image seam-matching provides many of the benefits of gradient blending (e.g., the avoidance of ghosting), without the associated computational overhead. This makes image seam-matching a particularly useful approach for real-time image processing such as during the real-time generation of wide area-of-view images. In situations where image seam-matching may be inappropriate, such as when the images being blended include long objects that span an entire overlap region(s), a mechanism is described which allows the selection of either, or both, seam-matching and cross-fading blend operations in a graceful or smooth manner.

Claims

1. A non-transitory program storage device, readable by a processor and comprising instructions stored thereon to cause the processor to:

obtain an overlap region between a left image and a right image, wherein the overlap region comprises a plurality of rows, each row having a plurality of pixels, each overlap region pixel having a corresponding pixel in the left image and a corresponding pixel in the right image, each pixel in the left and right images having a value;

identify a seam through the overlap region, the seam comprising a plurality of overlap region pixels, each overlap region pixel from a different row in the overlap region, wherein the seam divides each row in the overlap region into a left portion bounded by a left transition band boundary pixel on one end and the seam pixel on the other end and a right portion bounded by the seam pixel on one end and a right transition band boundary pixel on the other end;

determine a value for each seam pixel based on a combination of each seam pixel'"'"'s corresponding left and right image pixels;

identify a left smoothing function for each row in the overlap region, each left smoothing function having a zero value at the row'"'"'s left transition band boundary pixel and another value at the row'"'"'s seam pixel, the value of which is based on a combination of the row'"'"'s left transition band boundary pixel'"'"'s value and the row'"'"'s seam pixel value;

identify a right smoothing function for each row in the overlap region, each right smoothing function having a zero value at the row'"'"'s right transition band boundary pixel and another value at the row'"'"'s seam pixel, the value of which is based on a combination of the row'"'"'s right transition band boundary pixel'"'"'s value and the row'"'"'s seam pixel value;

apply, for each row in the overlap region, the corresponding left smoothing function to those pixels in the left image corresponding to the overlap region'"'"'s row'"'"'s left portion to generate corresponding left portion blended output image pixels;

apply, for each row in the overlap region, the corresponding right smoothing function to those pixels in the right image corresponding to the overlap region'"'"'s row'"'"'s right portion to generate corresponding right portion blended output image pixels; and

store the blended output image pixels in a memory.

2. The non-transitory program storage device of claim 1, wherein the instructions to cause the processor to determine a value for each seam pixel comprise instructions to cause the processor to:

determine a preliminary value for each seam pixel based, at least in part, on the values of the seam pixel'"'"'s corresponding pixels from the left and right images; and

filter each preliminary seam pixel value to generate a seam pixel value.

3. The non-transitory program storage device of claim 2, wherein the instructions to cause the processor to determine a preliminary value for each seam pixel comprise instructions to cause the processor to determine an average of each seam pixel'"'"'s corresponding left and right image pixel values.

4. The non-transitory program storage device of claim 1, further comprising instructions to cause the processor to display the blended output pixels on a display element.

5. The non-transitory program storage device of claim 1, wherein the instructions to obtain an overlap region further comprise instructions to determine a registration quality metric.

6. The non-transitory program storage device of claim 5, further comprising instructions to, when the registration quality metric is less than a threshold value:

obtain output from a motion sensor; and

obtain an overlap region based on the motion sensor output.

7. The non-transitory program storage device of claim 6, wherein the motion sensor output corresponds to motion that occurred at a time the left and right images were captured.

8. The non-transitory program storage device of claim 6, wherein the motion sensor comprises one of an accelerometer and a gyroscope.

9. The non-transitory program storage device of claim 6, wherein the left and right smoothing function and the right smoothing functions are different smoothing functions.

10. A hand-held electronic device, comprising:

an image capture element;

a memory operatively coupled to the image capture element and having stored therein, computer program code;

a display element operatively coupled to the memory; and

one or more processors operatively coupled to the memory and the display element and configured to execute the computer program code to—

use the image capture element to acquire first and second images,

store the first and second images in the memory,

determine an overlap region between the first and second images, wherein the overlap region comprises a plurality of rows, each row having a plurality of pixels, each overlap region pixel having a corresponding pixel in the first image and a corresponding pixel in the second image, each pixel in the first and second images having a value,

identify a seam through the overlap region, the seam comprising a plurality of overlap region pixels, each overlap region pixel from a different row in the overlap region, wherein the seam divides each row in the overlap region into a left portion bounded by a left transition band boundary pixel on one end and the seam pixel on the other end, and a second portion bounded by a right transition band boundary pixel on one end and the seam pixel on the other end,

determine a value for each seam pixel based on a combination of each seam pixel'"'"'s corresponding first and second image pixels,

identify a left smoothing function for each row in the overlap region, each left smoothing function having a zero value at the row'"'"'s left transition band boundary pixel and another value at the row'"'"'s seam pixel based on a combination of the row'"'"'s left transition band boundary pixel'"'"'s value and the row'"'"'s seam pixel value,

identify a right smoothing function for each row in the overlap region, each right smoothing function having a zero value at the row'"'"'s right transition band boundary pixel and another value at the row'"'"'s seam pixel based on a combination of the row'"'"'s right transition band boundary pixel'"'"'s value and the row'"'"'s seam pixel value,

apply, for each row in the overlap region, the corresponding left smoothing function to those pixels in the first image corresponding to the overlap region'"'"'s row'"'"'s left portion to generate corresponding left portion blended output image pixels,

apply, for each row in the overlap region, the corresponding right smoothing function to those pixels in the second image corresponding to the overlap region'"'"'s row'"'"'s right portion to generate corresponding right portion blended output image pixels, and

store the blended output image pixels in the memory.

11. The hand-held electronic device of claim 10, further comprising instructions to cause the one or more processors to display the blended output image pixels on the display element.

12. The hand-held electronic device of claim 10, wherein the instructions to obtain an overlap region further comprise instructions to determine a registration quality metric.

13. The hand-held electronic device of claim 12, further comprising instructions to cause the one or more processors to, when the registration quality metric is less than a threshold value:

obtain, concurrently with capture of the first and second images, output from a motion sensor of the hand-held electronic device; and

obtain an overlap region based on the motion sensor output.

14. The hand-held electronic device of claim 13, wherein the motion sensor comprises one of an accelerometer and a gyroscope.

15. A method to stitch multiple images into a single image, comprising:

obtaining an overlap region between a left image and a right image, wherein the overlap region comprises a plurality of rows, each row having a plurality of pixels, each overlap region pixel having a corresponding pixel in the left image and a corresponding pixel in the right image, each pixel in the left and right images having a value;

identifying a seam through the overlap region, the seam comprising a plurality of overlap region pixels, each overlap region pixel from a different row in the overlap region, wherein the seam divides each row in the overlap region into a left portion bounded by a left transition band boundary pixel on one end and the seam pixel on the other end and a right portion bounded by a right transition band boundary pixel on one end and the seam pixel on the other end;

determining a value for each seam pixel based on a combination of each seam pixel'"'"'s corresponding left and right image pixels;

identifying a left smoothing function for each row in the overlap region, each left smoothing function having a zero value at the row'"'"'s left transition band boundary pixel and another value at the row'"'"'s seam pixel based on a combination of the row'"'"'s left transition band boundary pixel'"'"'s value and the row'"'"'s seam pixel value;

identifying a right smoothing function for each row in the overlap region, each right smoothing function having a zero value at the row'"'"'s right transition band boundary pixel and another value at the row'"'"'s seam pixel based on a combination of the row'"'"'s right transition band boundary pixel'"'"'s value and the row'"'"'s seam pixel value;

applying, for each row in the overlap region, the corresponding left smoothing function to those pixels in the left image corresponding to the overlap region'"'"'s row'"'"'s left portion to generate corresponding left portion blended output image pixels;

applying, for each row in the overlap region, the corresponding right smoothing function to those pixels in the right image corresponding to the overlap region'"'"'s row'"'"'s right portion to generate corresponding right portion blended output image pixels; and

storing the blended output image pixels in a memory.

16. The method of claim 15, further comprising displaying the blended output image pixels.

17. The method of claim 15, wherein determining a value for each seam pixel comprises:

determining a preliminary value for each seam pixel based, at least in part, on the values of the seam pixel'"'"'s corresponding pixels from the left and right images; and

filtering each preliminary seam pixel value to generate a seam pixel value.

18. The method of claim 15, wherein obtaining an overlap region further comprises determining a registration quality metric.

19. The method of claim 18, further comprising, when the registration quality metric is less than a threshold value:

obtaining output from a motion sensor; and

obtaining an overlap region based on the motion sensor output.

20. The method of claim 19, wherein the motion sensor output corresponds to motion that occurred at a time the left and right images were captured.