Image blending operations

• US 8,902,335 B2
• Filed: 06/06/2012
• Issued: 12/02/2014
• Est. Priority Date: 06/06/2012
• Status: Expired due to Fees

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Doepke, Frank, Tico, Marius, Zuliani, Marco
• Primary Examiner(s): Tran, Nhan T
• Assistant Examiner(s): NGUYEN, CHAN T H
• Application Number:
  US13/489,950
• Publication Number:
  US 20130329071A1
• Time in Patent Office: 909 Days
• Field of Search: 348 36- 39, 348/239, 348/218.1, 382/284, 382/294
• US Class Current: 348/239
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors

Abstract

Procedures are described for blending images in real-time that avoid ghosting artifacts (attributable to moving objects), maintain the proper appearance of contiguous edges in the final image, and permits the use of fast (real-time) blending operations. A “guard-band” may be defined around an initially identified seam that perturbs the path of the initial seam so that both the seam and the guard-band'"'"'s edges avoid moving objects by at least a specified amount. Rapid blend operations may then be performed in the region demarcated by the guard-band. The seam may be further adjusted to bias its position toward a specified trajectory within the overlap region when there is no moving object present. If visual registration techniques are not able to provide a properly aligned overlap region, motion sensor data for the image capture device, may be used instead to facilitate blending operations.

Claims

1. A non-transitory program storage device, readable by a processor and comprising instructions stored thereon to cause the processor to:

obtain a first high-aspect ratio strip from a first image, the first high-aspect ratio strip having a plurality of pixels and a size, the size having a first dimension that is significantly greater than a second dimension;

obtain a second high-aspect ratio strip from a second image, the second high-aspect ratio strip having a plurality of pixels and a size, the size having a first dimension that is significantly greater than a second dimension;

obtain an overlap region between the first and second high-aspect ratio strips, the overlap region having a plurality of rows of pixels wherein each pixel in the overlap region has a corresponding pixel in each of the first and second high-aspect ratio strips;

identify a seam through the overlap region, the seam having a plurality of values, each value corresponding to a pixel in a row of the overlap region;

identify a guard band about the seam, the guard band extending a first number of pixels in a row to a first side and a second number of pixels in the row to a second side of each seam value, wherein the first and second number of pixels do not have to be the same number of pixels for each seam value;

blend, in real-time and for each seam value, pixels from the first image and the corresponding pixels from the second image along each row of pixels in the overlap region within the identified guard-band to produce an output image; and

store the output image in a memory.

2. The non-transitory program storage device of claim 1, wherein the instructions to cause the processor to identify a seam comprise instructions to cause the processor to apply a first filter to each pixel from the first high-aspect ratio strip in the overlap region and each corresponding pixel from the second high-aspect ratio strip to generate a first cost map, the first cost map having a plurality of values, one value for each pixel in the overlap region.

3. The non-transitory program storage device of claim 2, wherein the instructions to cause the processor to identify a guard band comprise instructions to cause the processor to apply a second filter to each of the plurality of values in the first cost map to generate a second cost map, the second cost map having a plurality of values, each value in the first cost map having a corresponding value in the second cost map, wherein the values in the second cost map identify the guard-band.

4. The non-transitory program storage device of claim 3, wherein the instructions to cause the processor to identify a seam further comprise instructions to cause the processor to identify a minimum cost path through the second cost map.

5. The non-transitory program storage device of claim 4, wherein the instructions to cause the processor to identify a seam further comprise instructions to cause the processor to enforce a continuity constraint on adjacent values of the minimum cost path.

6. The non-transitory program storage device of claim 3, wherein the instructions to cause the processor to apply a second filter further comprise instructions to cause the processor to apply a third filter to each value in the second cost map to generate a third cost map, the third filter adapted to increase each value in the second cost map based, at least in part, on a distance each value is from a specified path in the overlap region, each value so generated comprising one value in the third cost map, each value in the second cost map having a corresponding value in the third cost map.

7. The non-transitory program storage device of claim 6, wherein the instructions to cause the processor to identify a seam further comprise instructions to cause the processor to identify a minimum cost path through the third cost map.

8. The non-transitory program storage device of claim 7, wherein the instructions to cause the processor to identify a seam further comprise instructions to cause the processor to enforce a continuity constraint on adjacent values of the minimum cost path comprising the seam.

9. The non-transitory program storage device of claim 2, wherein the instructions to cause the processor to apply a first filter further comprise instructions to cause the processor to apply a second filter to each value in the first cost map to generate a second cost map, the second filter adapted to increase each value in the first cost map based, at least in part, on a distance each value is from a specified path in the overlap region, each value so generated comprising one value in the second cost map, each value in the second cost map having a corresponding value in the first cost map.

10. The non-transitory program storage device of claim 9, wherein the instructions to cause the processor to identify a seam further comprise instructions to cause the processor to identify a minimum cost path through the second cost map.

11. The non-transitory program storage device of claim 9, wherein the specified path comprises a path along a centerline of the overlap region.

12. The non-transitory program storage device of claim 1, further comprising instructions to cause the processor to display the output image on a display element.

13. The non-transitory program storage device of claim 1, wherein the instructions to cause the processor to obtain an overlap region further comprise instructions to cause the processor to obtain a registration quality value indicating a quality of the overlap region.

14. The non-transitory program storage device of claim 13, further comprising instructions to cause the processor to:

determine the registration quality value is less than a threshold value; and

in response

identify a new overlap region based, at least in part, on motion sensor output.

15. An image blending method, comprising:

receiving, by a processor, an overlap region between a first high-aspect ratio strip from a first image and a second high-aspect ratio strip from a second image, each of the first and second high-aspect ratio strips having a plurality of pixels and a size, the size having a first dimension that is significantly larger than a second dimension, the overlap region having a plurality of rows of pixels wherein each pixel in the overlap region has a corresponding pixel in each of the first and second high-aspect ratio strips;

identifying, by the processor, a seam through the overlap region, the seam having a plurality of values, each value corresponding to a pixel in a row of the overlap region;

identifying, by the processor, a guard band about the seam, the guard band extending a first number of pixels in a row to a first side and a second number of pixels in the row to a second side of each seam value, wherein the first and second number of pixels do not have to be the same number of pixels for each seam value;

blending, by the processor, in real-time and for each seam value, pixels from the first high-aspect ratio strip and the corresponding pixels from the second high-aspect ratio strip along each row of pixels in the overlap region within the identified guard-band to produce an output image; and

storing, by the processor, the output image in a memory.

16. The method of claim 15, wherein the act of identifying a seam comprises applying, by the processor, a first filter to each pixel from the first high-aspect ratio strip in the overlap region and each corresponding pixel from the second high-aspect ratio strip to generate a first cost map, the first cost map having a plurality of values, one value for each pixel in the overlap region.

17. The method of claim 16, wherein the act of identifying a guard band comprises applying, by the processor, a second filter to each of the plurality of values in the first cost map to generate a second cost map, the second cost map having a plurality of values, each value in the first cost map having a corresponding value in the second cost map, wherein the values in the second cost map identify the guard-band.

18. The method of claim 17, wherein the act of identifying a seam, further comprises:

identifying, by the processor, a minimum cost path through the second cost map; and

enforcing, by the processor, a continuity constraint on adjacent values of the minimum cost path.

19. The method of claim 16, wherein the act of applying a first filter further comprises applying, by the processor, a second filter to each value in the first cost map to generate a second cost map, the second filter adapted to increase each value in the first cost map based, at least in part, on a distance each value is from a centerline path in the overlap region, each value so generated comprising one value in the second cost map, each value in the second cost map having a corresponding value in the first cost map.

20. The method of claim 15, wherein the act of receiving an overlap region further comprises obtaining, by the processor, a registration quality value indicating a quality of the overlap region.

21. The method of claim 20, further comprising:

determining, by the processor, if the registration quality value is at least a specified value; and

identifying, by the processor, a new overlap region based, at least in part, on motion sensor output when the registration quality value is less than the specified value.

22. An electronic device, comprising:

an image capture element;

a motion sensor element;

a memory operatively coupled to the image capture element;

a display element operatively coupled to the memory; and

a processor operatively coupled to the motion sensor element, the memory and the display element, wherein the memory further includes instructions to cause the processor to

obtain a first high-aspect ratio strip from a first image and a second high-aspect ratio strip from a second image, each high-aspect ratio strip having a plurality of pixels and a size, the size having a first dimension that is significantly greater than a second dimension,

identify an overlap region between the first and second high-aspect ratio strips, the overlap region having a plurality of rows of pixels wherein each pixel in the overlap region has a corresponding pixel in each of the first and second high-aspect ratio strips,

identify a seam through the overlap region, the seam having a plurality of values, each value corresponding to a pixel in a row of the overlap region,

identify a guard band about the seam, the guard band extending one or more pixels in a row to a first side and one or more pixels in the row to a second side of each seam value,

blend, for each seam value, pixels from the first high-aspect ratio strip and the corresponding pixels from the second high-aspect ratio strip along each row of pixels in the overlap region within the identified guard-band to produce an output image, and

store the output image in the memory.

23. The electronic device of claim 22, wherein the instructions to cause the processor to identify a guard band comprise instructions to cause the processor to:

apply a first filter to each pixel from the first high-aspect ratio strip in the overlap region and each corresponding pixel from the second high-aspect ratio strip to generate a first cost map, the first cost map having a plurality of values, one value for each pixel in the overlap region; and

apply a second filter to each of the plurality of values in the first cost map to generate a second cost map, the second cost map having a plurality of values, each value in the first cost map having a corresponding value in the second cost map, wherein the values in the second cost map identify the guard-band.

24. The electronic device of claim 23, wherein the instructions to cause the processor to identify a seam further comprise instructions to cause the processor to identify a minimum cost path through the second cost map.

25. The electronic device of claim 23, wherein the instructions to cause the processor to apply a second filter further comprise instructions to cause the processor to apply a third filter to each value in the second cost map to generate a third cost map, the third filter adapted to increase each value in the second cost map based, at least in part, on a distance each value is from a specified path in the overlap region, each value so generated comprising one value in the third cost map, each value in the second cost map having a corresponding value in the third cost map.

26. The electronic device of claim 25, wherein the instructions to cause the processor to identify a seam further comprise instructions to cause the processor to:

identify a minimum cost path through the third cost map; and

enforce a continuity constraint on adjacent values of the minimum cost path comprising the seam.

27. The electronic device of claim 22, wherein the instructions to cause the processor to identify a seam comprise instructions to cause the processor to:

apply a first filter to each pixel from the first high-aspect ratio strip in the overlap region and each corresponding pixel from the second high-aspect ratio strip to generate a first cost map, the first cost map having a plurality of values, one value for each pixel in the overlap region;

apply a second filter to each value in the first cost map to generate a second cost map, the second filter adapted to increase each value in the first cost map based, at least in part, on a distance each value is from a specified path in the overlap region, each value so generated comprising one value in the second cost map, each value in the second cost map having a corresponding value in the first cost map; and

identify a minimum cost path through the second cost map.

28. The electronic device of claim 27, wherein the specified path comprises a path along a centerline through the overlap region.

29. The electronic device of claim 22, further comprising instructions to cause the processor to display the output image on the display element.

30. The electronic device of claim 22, wherein the instructions to cause the processor to identify an overlap region comprise instructions to cause the processor to obtain a registration quality value indicating a quality of the overlap region.

31. The electronic device of claim 30, further comprising instructions to cause the processor to:

determine the registration quality value less than a threshold value; and

, in response identify a new overlap region based, at least in part, on data from the motion sensor element.

32. A non-transitory program storage device, readable by a processor and comprising instructions stored thereon to cause the processor to:

obtain first and second images, each having a first plurality of pixels;

identify an overlap region between the first and second images, the overlap region having a second plurality of pixels, wherein each pixel in the overlap region has a corresponding pixel in the first image and a corresponding pixel in the second image;

apply a first filter to each corresponding pair of pixels in the first and second images that are in the overlap region to generate a first cost map, the first cost map having one value for each pixel in the overlap region;

apply a second filter to each value in the first cost map to generate a second cost map, wherein the second filter is adapted to create an area of lower values in the second cost map that are close to a moving object in the overlap region;

identify a minimum cost path through the second cost map;

blend, to generate a first output image, corresponding pairs of pixels from the first and second images corresponding to each value in the second cost map that is in an area of lower values created by the second filter;

incorporate pixels from the first image that are in the overlap region, and not in an area of lower values created by the second filter, into the first output image;

incorporate pixels from the first image that are not in the overlap region into the first output image;

incorporate pixels from the second image that are in the overlap region, and not in an area of lower values created by the second filter, into the first output image;

incorporate pixels from the second image that are not in the overlap region into the first output image; and

store the first output image in a memory.

33. The non-transitory program storage device of claim 32, wherein the instructions to cause the processor to identify a minimum cost path comprise instructions to cause the processor to apply a continuity constraint between successive values of the minimum cost path.

34. The non-transitory program storage device of claim 32, wherein the instructions to cause the processor to apply a second filter further comprise instructions to cause the processor to apply a third filter to the second cost map to generate a third cost map, wherein the third filter is adapted to lower values from the second cost map that are toward a centerline of the overlap region when no moving object is present at a corresponding location in the first and second images.

35. The non-transitory program storage device of claim 34, wherein the instructions to cause the processor to identify a minimum cost path comprise instructions to cause the processor to apply a continuity constraint between successive values of the minimum cost path.

36. The non-transitory program storage device of claim 34, wherein the instructions to cause the processor to blend, to generate a first output image, corresponding pairs of pixels from the first and second images comprise instructions to cause the processor to alpha-blend corresponding pairs of pixels from the first and second images corresponding to each value in the second cost map that is in an area of lower values created by the second filter.

37. The non-transitory program storage device of claim 32, wherein the instructions to cause the processor to obtain first and second images comprise instructions to cause the processor to obtain a registration quality value.

38. The non-transitory program storage device of claim 37, wherein the instructions to cause the processor to identify an overlap region comprise instructions to cause the processor to:

determine if the registration quality value is at least a specified value; and

identify the overlap region based, at least in part, on motion sensor output when the registration quality values is less than the specified value.