Coded aperture camera with adaptive image processing

• US 8,582,820 B2
• Filed: 09/24/2010
• Issued: 11/12/2013
• Est. Priority Date: 09/24/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Kane, Paul James, Wang, Sen
• Primary Examiner(s): Johns, Andrew W
• Application Number:
  US12/889,818
• Publication Number:
  US 20120076362A1
• Time in Patent Office: 1,145 Days
• Field of Search: 382/106, 382/154, 382/173, 382/255, 382/286, 356/4.01, 356/4.04
• US Class Current: 382/106
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/571 from focus

Abstract

An image capture device is used to identify object range information, and includes: providing an image capture device, an image sensor, a coded aperture, and a lens; and using the image capture device to capture a digital image of the scene from light passing through the lens and the coded aperture, the scene having a plurality of objects. The method further includes: dividing the digital image into a set of blocks; assigning a point spread function (psf) value to each of the blocks; combining contiguous blocks in accordance with their psf values; producing a set of blur parameters based upon the psf values of the combined blocks and the psf values of the remaining blocks; producing a set of deblurred images based upon the captured image and each of the blur parameters; and using the set of deblurred images to determine the range information for the objects in the scene.

Claims

1. A method, comprising:

obtaining a digital image of a scene that includes a plurality of objects, the digital image obtained by an image capture device through a coded aperture;

segmenting the digital image into a plurality of blocks;

determining a point spread function (psf) value for each of the blocks;

grouping contiguous blocks having related psf values; and

determining a distance between the image capture device and one or more of the plurality of objects based, at least in part, on a relation between the psf values of the grouped blocks and a measured depth dependency of psf values of the image capture device.

2. The method of claim 1, wherein the step of determining a distance between the image capture device and one or more of the plurality of objects further includes:

producing a set of blur parameters based upon the psf values of the grouped blocks and the psf values of non-grouped blocks;

producing a set of deblurred images based upon the obtained digital image and each of the blur parameters; and

determining the distance between the image capture device and one or more of the plurality of objects based, at least in part, on using the set of deblurred images.

3. The method of claim 1, wherein the step of determining a psf value for each of the blocks further includes:

measuring the psf of the image capture device at a series of object and image defocus positions;

forming an initial psf estimate for each block in the captured image;

comparing the initial psf estimates to the measured psf of the image capture device; and

determining a psf value for each block in the captured image based, at least in part, on the comparison.

4. The method of claim 3, wherein the initial psf estimate is formed using a frequency content or sharpness metric.

5. The method of claim 3, wherein the initial psf estimate is formed using edge detection and edge gradient analysis.

6. The method of claim 1, wherein the step of obtaining a digital image of a scene further includes obtaining a sequence of digital images of the scene.

7. The method of claim 6, wherein the step of determining a distance between the image capture device and one or more of the plurality of objects further includes determining at least one distance for each image in the sequence.

8. A digital camera system comprising:

an image capture device for capturing one or more images of a scene, each image including a plurality of objects, the image capture device comprising;

an image sensor;

a lens for imaging the scene onto the image sensor;

a coded aperture; and

a processor-accessible memory; and

a data processing system coupled to the image capture device for obtaining one of the one or more images of the scene;

segmenting the obtained image into a plurality of blocks;

determining a point spread function (psf) value for each of the blocks;

grouping contiguous blocks having related psf values; and

determining a distance between the image capture device and one or more of the plurality of objects in the obtained image based, at least in part, on a relation between the psf values of the grouped blocks and a measured depth dependency of psf values of the image capture device.

9. The digital camera system of claim 8, wherein the step of determining the distance between the image capture device and one or more of the plurality of objects further includes:

producing a set of blur parameters based upon the psf values of the grouped blocks and the psf values of non-grouped blocks;

producing a set of deblurred images based upon the obtained digital image and each of the blur parameters; and

determining the distance between the image capture device and one or more of the plurality of objects based, at least in part, on using the set of deblurred images.

10. The digital camera system of claim 8, wherein the step of determining the psf value for each of the blocks further includes:

measuring the psf of the image capture device at a series of object and image defocus positions;

forming an initial psf estimate for each block in the captured image;

comparing the initial psf estimates to the measured psf of the image capture device; and

determining a psf value for each block in the obtained image based, at least in part, on the comparison.

11. The digital camera system of claim 10, wherein the initial psf estimate is formed using a frequency content or sharpness metric.

12. The digital camera system of claim 10, wherein the initial psf estimate is formed using edge detection and edge gradient analysis.

13. The digital camera system of claim 8, wherein the step of obtaining one of the one or more images of the scene further includes obtaining a plurality of images of the scene captured by the image capture device.

14. The digital camera system of claim 13, wherein the step of determining the distance between the image capture device and one or more of the plurality of objects further includes determining at least one distance for each image in the obtained plurality of images of the scene captured by the image capture device.

15. A non-transitory processor-accessible memory device, readable by a programmable digital computer and comprising instructions stored thereon to cause the programmable digital computer to perform a method, comprising:

obtaining a digital image of a scene that includes a plurality of objects, the digital image obtained by an image capture device through a coded aperture;

segmenting the digital image into a plurality of blocks;

determining a point spread function (psf) value for each of the blocks;

grouping contiguous blocks having related psf values; and

determining a distance between the image capture device and one or more of the plurality of objects based, at least in part, on a relation between the psf values of the grouped blocks and a measured depth dependency of psf values of the image capture device.

16. The non-transitory processor-accessible memory device of claim 15, wherein the step of determining a distance between the image capture device and one or more of the plurality of objects further includes:

producing a set of blur parameters based upon the psf values of the grouped blocks and the psf values of non-grouped blocks;

producing a set of deblurred images based upon the obtained digital image and each of the blur parameters; and

determining the distance between the image capture device and the one or more of the plurality of objects based, at least in part, on using the set of deblurred images.

17. The non-transitory processor-accessible memory device of claim 15, wherein the step of determining the psf value for each of the blocks further includes:

measuring the psf of the image capture device at a series of object and image defocus positions;

forming an initial psf estimate for each block in the captured image;

comparing the initial psf estimates to the measured psf of the image capture device; and

determining a psf value for each block in the captured image based, at least in part, on the comparison.

18. The non-transitory processor-accessible memory device of claim 16, wherein the initial psf estimate is formed using a frequency content or sharpness metric.

19. The non-transitory processor-accessible memory device of claim 16, wherein the initial psf estimate is formed using edge detection and edge gradient analysis.

20. The non-transitory processor-accessible memory device of claim 15, wherein the step of obtaining a digital image of a scene further includes obtaining a sequence of digital images obtained by the image capture device, and wherein the step of determining a distance between the image capture device and one or more of the plurality of objects further includes determining at least one distance for each image in the sequence.