Processing Images Having Different Focus

• US 20100165152A1
• Filed: 12/30/2008
• Published: 07/01/2010
• Est. Priority Date: 12/30/2008
• Status: Active Grant

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Lim, Jae S.
• Granted Patent: US 8,405,742 B2
• Time in Patent Office: Days
• US Class Current: 348/240.990
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 23/743 Bracketing, i.e. taking a series of images with varying exposure conditions
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

Processing images of a scene taken with different focus includes, for each of at least some of the images, determining at least one portion of the image having a predetermined characteristic. A representation of the scene is generated that includes an array of elements, where each of at least some of the elements corresponds to a determined portion of at least one of the images. An element of the array is based on a correspondence between a distance associated with a determined portion of at least one image and a focused distance associated with at least one image. The generated representation is stored or provided as output.

Claims

1. A method for processing images of a scene taken with different focus, comprising:

for each of at least some of the images, determining at least one portion of the image having a predetermined characteristic;

generating a representation of the scene comprising an array of elements, where each of at least some of the elements corresponds to a determined portion of at least one of the images, wherein an element of the array is based on a correspondence between a distance associated with a determined portion of at least one image and a focused distance associated with at least one image; and

storing the generated representation or providing the generated representation as output.

2. The method of claim 1, wherein at least some of the elements of the array each represent an intensity for a given part of the scene.

3. The method of claim 1, wherein the generated representation of the scene comprises a composite image in which a portion of the composite image representing a given part of the scene is derived from at least one of the images in which the given part of the scene is most in focus.

4. The method of claim 1, wherein an element of the array is derived from a first portion of a first of the images that represents a given part of the scene, and the correspondence between a distance associated with the first portion of the first image and a focused distance associated with the first image indicates that the given part of the scene is more in focus in the first image than in any of the other images.

5. The method of claim 4, wherein the element of the array comprises the first portion of the first image.

6. The method of claim 4, wherein the element of the array is derived from an interpolation between the first portion of the first image and a corresponding second portion of a second of the images.

7. The method of claim 1, wherein at least some of the elements of the array each represent a distance to a given part of the scene.

8. The method of claim 1, wherein the generated representation of the scene comprises a depth map in which each distance associated with a given part of the scene is derived from a focused distance associated with at least one of the images in which the given part of the scene is most in focus.

9. The method of claim 8, wherein generating the depth map comprises adjusting an initial distance estimate associated with the given part of the scene based on the focused distance.

10. The method of claim 9, wherein the initial distance estimate associated with the given part of the scene is received from an initial depth map of the scene.

11. The method of claim 1, wherein determining at least one portion of a given image having a predetermined characteristic comprises measuring a local characteristic of each of multiple portions of the given image.

12. The method of claim 11, wherein the local characteristic comprises local variation in intensity.

13. The method of claim 12, wherein an element of the array representing a distance corresponds to portions of the images representing a given part of the scene, and the element of the array is derived from a focused distance associated with a given image for which the local variation of the corresponding portion is larger than the local variation of corresponding portions of the other images.

14. The method of claim 12, wherein measuring the local characteristic comprises calculating a value of a two-dimensional variance function.

15. The method of claim 1, wherein determining at least one portion of a given image having a predetermined characteristic comprises comparing a distance associated with the given portion of the given image to a focused distance associated with the given image.

16. The method of claim 15, wherein the distance associated with the given portion of the given image is derived from a depth map of the scene.

17. The method of claim 15, wherein an element of the array is derived from a portion of an image for which the focused distance is closer to the associated distance than the focused distances associated with the other images.

18. The method of claim 1, wherein each of at least some of the portions of an image comprises one or more pixels of a digital image.

19. The method of claim 1, further comprising capturing each of the images of the scene with a different focus.

20. The method of claim 19, further comprising storing information indicating the focus at which each of the images was captured.

21. The method of claim 1, wherein each of the images represents substantially the same view of the scene.

22. A method for generating an image of a scene, comprising:

receiving a first representation of the scene comprising a first array of elements that each represents an intensity for a given part of the scene;

receiving a second representation of the scene comprising a second array of elements that each represents a distance to a given part of the scene;

generating an image of the scene in which each portion of the generated image is based on an element from the first array and an element from the second array that represent substantially the same part of the scene; and

storing the generated image or providing the generated image as output.

23. The method of claim 22, wherein generating the image of the scene comprises applying a blurring function to a plurality of intensity values from the first array based on at least one distance value from the second array and a target focused distance.

24. The method of claim 23, wherein generating the image of the scene comprises applying a blurring function to a plurality of intensity values from the first array based on a difference between a distance value from the second array and a target focused distance.

25. The method of claim 24, wherein the distance value from the second array represents a distance to a part of the scene represented by at least one of the plurality of intensity values to which the blurring function is being applied.

26. A computer-readable medium storing a computer program for processing images of a scene taken with different focus, the computer program comprising instructions for causing a computer system to:

for each of at least some of the images, determine at least one portion of the image having a predetermined characteristic;

generate a representation of the scene comprising an array of elements, where each of at least some of the elements corresponds to a determined portion of at least one of the images, wherein an element of the array is based on a correspondence between a distance associated with a determined portion of at least one image and a focused distance associated with at least one image; and

store the generated representation or provide the generated representation as output.

27. A system, comprising:

an image capture subsystem configured to capture images of a scene taken with different focus; and

an image processing subsystem configured to,

for each of at least some of the images, determine at least one portion of the image having a predetermined characteristic, and

generate a representation of the scene comprising an array of elements, where each of at least some of the elements corresponds to a determined portion of at least one of the images, wherein an element of the array is based on a correspondence between a distance associated with a determined portion of at least one image and a focused distance associated with at least one image.