SYSTEM AND METHOD FOR ALL-IN-FOCUS IMAGING FROM MULTIPLE IMAGES ACQUIRED WITH HAND-HELD CAMERA

• US 20120076403A1
• Filed: 09/23/2010
• Published: 03/29/2012
• Est. Priority Date: 09/23/2010
• Status: Abandoned Application

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Nestares, Oscar, Zhou, Jianping, Gat, Yoram
• Application Number:
  US12/888,684
• Publication Number:
  US 20120076403A1
• Time in Patent Office: Days
• US Class Current: 382/167
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 23/743 Bracketing, i.e. taking a series of images with varying exposure conditions
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

Methods and systems to create an image in which objects at different focal depths all appear to be in focus. In an embodiment, all objects in the scene may appear in focus. Non-stationary cameras may be accommodated, so that variations in the scene resulting from camera jitter or other camera motion may be tolerated. An image alignment process may be used, and the aligned images may be blended using a process that may be implemented using logic that has relatively limited performance capability. The blending process may take a set of aligned input images and convert each image into a simplified Laplacian pyramid (LP). The LP is a data structure that includes several processed versions of the image, each version being of a different size. The set of aligned images is therefore converted into a corresponding set of LPs. The LPs may be combined into a composite LP, which may then undergo Laplacian pyramid reconstruction (LPR). The output of the LPR process is the final blended image.

Claims

1. A method, comprising:

aligning a plurality of images of the same scene, where different images have different objects in focus; and

blending the aligned images, said blending comprising;

for each image, constructing a Laplacian pyramid representing the image for each of the color components of the image in an appropriate color component representation;

constructing a composite Laplacian pyramid, based on the plurality of Laplacian pyramids corresponding to the respective plurality of images; and

performing Laplacian pyramid reconstruction on the composite Laplacian pyramid, to create a blended image wherein the different objects that were in focus in the respective images, appear in focus in the blended image.

2. The method of claim 1, wherein said construction of a Laplacian pyramid representing an image comprises:

iteratively reducing the image and saving the reduced image resulting from each reduction, creating a series of reduced images;

expanding each reduced image, creating a series of expanded images;

subtracting each expanded image from the image that served as input to the reduction that produced the reduced image used to create the expanded image, resulting in a set of difference images; and

using the difference images to create the Laplacian pyramid representing the image.

3. The method of claim 2, wherein said reduction of an image comprises:

applying a linear filter to the image;

discarding every other row of pixels from the filtered image; and

for the remaining rows of the filtered image, discarding every other pixel.

4. The method of claim 3, wherein said expansion of an image comprises:

in each row, interleaving all-zero pixel representations between pixels in the row, so that pixels in the row alternate with all-zero pixel representations;

adding rows of all-zero pixel representations between rows of the image, so that every other row of the image is a row of all-zero pixel representations; and

applying the linear filter to the result.

5. The method of claim 4, wherein said linear filter uses a mask defined by the expression

6. The method of claim 2, wherein said Laplacian pyramid reconstruction comprises:

expanding the smallest difference image;

adding the result of the expansion of the smallest difference image to the next largest difference image, resulting in a sum;

iteratively expanding the sum and adding the expanded sum to the next largest difference image to create a next sum, ultimately producing a final sum; and

using the final sum as the blended image.

7. The method of claim 1, wherein said constructing of a composite Laplacian pyramid comprises:

for each pixel to be defined in the composite Laplacian pyramid,

examining the corresponding pixel in each Laplacian pyramid representing an image;

choosing the pixel having the maximum absolute value from the set of corresponding pixels; and

using the chosen pixel as the pixel to be defined in the composite Laplacian pyramid.

8. A system, comprising:

a processor; and

a memory in communication with said processor, wherein said memory stores a plurality of processing instructions configured to direct said processor to

align a plurality of images of the same scene, where different images have different objects in focus; and

blend the aligned images, the blending comprising;

for each image, constructing a Laplacian pyramid representing the image for each of the color components of the image in an appropriate color component representation;

constructing a composite Laplacian pyramid, based on the plurality of Laplacian pyramids corresponding to the respective plurality of images; and

performing Laplacian pyramid reconstruction on the composite Laplacian pyramid, to create a blended image wherein the different objects that were in focus in the respective images, appear in focus in the blended image.

9. The system of claim 8, wherein said construction of a Laplacian pyramid representing an image comprises:

iteratively reducing the image and saving the reduced image resulting from each reduction, creating a series of reduced images;

expanding each reduced image, creating a series of expanded images;

subtracting each expanded image from the image that served as input to the reduction that produced the reduced image used to create the expanded image, resulting in a set of difference images; and

using the difference images to create the Laplacian pyramid representing the image.

10. The system of claim 9,

wherein the reduction of an image comprises:

applying a linear filter to the image;

discarding every other row of pixels from the filtered image; and

for the remaining rows of the filtered image, discarding every other pixel, and

wherein the expansion of an image comprises;

in each row, interleaving all-zero pixel representations between pixels in the row, so that pixels in the row alternate with all-zero pixel representations;

adding rows of all-zero pixel representations between rows of the image, so that every other row of the image is a row of all-zero pixel representations; and

applying the linear filter to the result.

11. The system of claim 10, wherein the linear filter uses a mask defined by the expression

12. The system of claim 9, wherein the Laplacian pyramid reconstruction comprises:

expanding the smallest difference image;

adding the result of the expansion of the smallest difference image to the next largest difference image, resulting in a sum;

iteratively expanding the sum and adding the expanded sum to the next largest difference image to create a next sum, ultimately producing a final sum; and

using the final sum as the blended image.

13. The system of claim 8, wherein the construction of a composite Laplacian pyramid comprises:

for each pixel to be defined in the composite Laplacian pyramid,

examining the corresponding pixel in each Laplacian pyramid representing an image;

choosing the pixel having the maximum absolute value from the set of corresponding pixels; and

using the chosen pixel as the pixel to be defined in the composite Laplacian pyramid.

14. A computer program product including a non-transitory computer readable medium having computer program logic stored therein, the computer program logic including:

logic to cause a processor to align a plurality of images of the same scene, where different images have different objects in focus; and

logic to cause a processor to blend the aligned images, the blending comprising;

for each image, constructing a Laplacian pyramid representing the image for each of the color components of the image in an appropriate color component representation;

constructing a composite Laplacian pyramid, based on the plurality of Laplacian pyramids corresponding to the respective plurality of images; and

performing Laplacian pyramid reconstruction on the composite Laplacian pyramid, to create a blended image wherein the different objects that were in focus in the respective images, appear in focus in the blended image.

15. The computer program product of claim 14, wherein the construction of a Laplacian pyramid representing an image comprises:

iteratively reducing the image and saving the reduced image resulting from each reduction, creating a series of reduced images;

expanding each reduced image, creating a series of expanded images;

subtracting each expanded image from the image that served as input to the reduction that produced the reduced image used to create the expanded image, resulting in a set of difference images; and

using the difference images to create the Laplacian pyramid representing the image.

16. The computer program product of claim 15, wherein the reduction of an image comprises:

applying a linear filter to the image;

discarding every other row of pixels from the filtered image; and

for the remaining rows of the filtered image, discarding every other pixel.

17. The computer program product of claim 16, wherein the expansion of an image comprises:

in each row, interleaving all-zero pixel representations between pixels in the row, so that pixels in the row alternate with all-zero pixel representations;

adding rows of all-zero pixel representations between rows of the image, so that every other row of the image is a row of all-zero pixel representations; and

applying the linear filter to the result.

18. The computer program product of claim 17, wherein the linear filter uses a mask defined by the expression

19. The computer program product of claim 15, wherein the Laplacian pyramid reconstruction comprises:

expanding the smallest difference image;

adding the result of the expansion of the smallest difference image to the next largest difference image, resulting in a sum;

iteratively expanding the sum and adding the expanded sum to the next largest difference image to create a next sum, ultimately producing a final sum; and

using the final sum as the blended image.

20. The computer program product of claim 14, wherein the constructing of a composite Laplacian pyramid comprises:

for each pixel to be defined in the composite Laplacian pyramid,

examining the corresponding pixel in each Laplacian pyramid representing an image;

choosing the pixel having the maximum absolute value from the set of corresponding pixels; and

using the chosen pixel as the pixel to be defined in the composite Laplacian pyramid.