Image correction using individual manipulation of microlenses in a microlens array

• US 6,967,780 B2
• Filed: 01/21/2004
• Issued: 11/22/2005
• Est. Priority Date: 01/21/2004
• Status: Expired due to Fees

• Current Assignee: The Invention Science Fund I, L.L.C. (Intellectual Ventures LLC)
• Original Assignee: Searete LLC (Intellectual Ventures LLC)
• Inventors: Myhrvold, Nathan P., Wood, Lowell L. Jr., Hillis, W. Daniel
• Primary Examiner(s): Dang, Hung Xuan
• Assistant Examiner(s): MARTINEZ, JOSEPH P
• Application Number:
  US10/764,431
• Publication Number:
  US 20050157395A1
• Time in Patent Office: 671 Days
• Field of Search: 359/619, 359/626, 348/335, 348/340, 348/345, 348/349, 348/207.9, 348/208.11, 348/222.1, 382/280, 382/284, 382/286, 382/299, 382/308, 250/200, 250/201.2, 250/201.7, 250/201.8, 250/201.9
• US Class Current: 359/619
• CPC Class Codes: G02B 3/0006 Arrays G02B3/02, G02B5/188 take precedence
• CPC Class Codes: G02B 3/0056 arranged along two different directions in a plane, e.g. honeycomb arrangement of lenses G02B3/0043 takes precedence; miniaturised objectives for electronic devices employing wafer level optics G02B13/0085
• CPC Class Codes: G02B 3/04 with continuous faces that are rotationally symmetrical but deviate from a true sphere , e.g. so called "aspheric" lenses
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction

Abstract

A system constructs a composite image using focus assessment information of image regions.

Claims

1. A method comprising:

capturing a primary image with a microlens array at a primary position, the microlens array having at least one microlens deviation that exceeds a first tolerance from a target optical property;

determining at least one out-of-focus region of the primary image;

capturing another image with at least one microlens of the microlens array at another position;

determining a focus of at least one region of the other image relative to a focus of the at least one out-of-focus region of the primary image; and

constructing a composite image in response to the at least one region of the other image having a sharper focus relative to the focus of the at least one out-of-focus region of the primary image.

2. The method of claim 1, wherein said capturing a primary image with a microlens array at a primary position, the microlens array having at least one microlens deviation that exceeds a first tolerance from a target optical property further comprises:

capturing with the microlens array having at least one microlens position that exceeds a first tolerance from at least one defined primary position.

3. The method of claim 1, wherein said capturing a primary image with a microlens array at a primary position, the microlens array having at least one microlens deviation that exceeds a first tolerance from a target optical property further comprises:

capturing with a microlens array frame having at least one frame deviation that exceeds a first tolerance from at least one defined array frame position.

4. The method of claim 1, wherein said capturing a primary image with a microlens array at a primary position, the microlens array having at least one microlens deviation that exceeds a first tolerance from a target optical property further comprises:

capturing with at least one microlens having a focal length that exceeds a first tolerance from a defined focal length.

5. The method of claim 1, wherein said capturing a primary image with a microlens array at a primary position, the microlens array having at least one microlens deviation that exceeds a first tolerance from a target optical property further comprises:

capturing with at least one microlens having a spherical aberration that exceeds a first tolerance from a defined spherical aberration.

6. The method of claim 1, wherein said capturing a primary image with a microlens array at a primary position, the microlens array having at least one microlens deviation that exceeds a first tolerance from a target optical property further comprises:

capturing with at least one microlens having a cylindrical aberration that exceeds a first tolerance from a defined cylindrical aberration.

7. The method of claim 1, wherein said capturing a primary image with a microlens array at a primary position, the microlens array having at least one microlens deviation that exceeds a first tolerance from a target optical property further comprises:

capturing the primary image at a primary focal surface location of the microlens array.

8. The method of claim 7, wherein said capturing the primary image at a primary focal surface location of the microlens array further comprises:

capturing the primary image with a photo-detector array at the primary focal surface location of the microlens array.

9. The method of claim 1, wherein said capturing another image with at least one microlens of the microlens array at another position further comprises:

mapping the at least one out-of-focus region to one or more microlenses of the microlens array;

moving at least a part of the mapped one or more microlenses of the microlens array to the other position; and

capturing the other image with a photo-detector array.

10. The method of claim 9, wherein said mapping the at least one out-of-focus region to one or more microlenses of the microlens array further comprises:

projecting mathematically from a surface of the photo-detector array to the microlens array; and

selecting the one or more microlenses of the microlens array in response to said projecting.

11. The method of claim 1, wherein said capturing another image with at least one microlens of the microlens array at another position further comprises:

moving at least a part of the microlens array to the other position.

12. The method of claim 11, wherein said moving at least a part of the microlens array to the other position further comprises:

moving at least one microlens of the microlens array to the other position; and

capturing the other image with a photo-detector array at an average primary focal surface location of the microlens array at the primary position.

13. The method of claim 12, wherein said moving at least one microlens of the microlens array to the other position further comprises:

moving the at least one microlens of the microlens array to the other position, said moving constrained by a predefined aberration from at least one defined microlens position.

14. The method of claim 1, wherein said determining at least one out-of-focus region of the primary image further comprises:

calculating a Fourier transform of at least a part of the primary image.

15. The method of claim 14, wherein said calculating a Fourier transform of at least a part of the primary image further comprises:

calculating a Fourier transform of at least one region of the primary image associated with at least one microlens.

16. The method of claim 1, wherein said determining a focus of at least one region of the other image relative to a focus of the at least one out-of-focus region of the primary image further comprises:

calculating a Fourier transform of the at least one region of the other image.

17. The method of claim 16, wherein said calculating a Fourier transform of the at least one region of the other image further comprises:

mapping at least one region of the primary image with the at least one region of the other image.

18. The method of claim 1, wherein said constructing a composite image in response to the at least one region of the other image having a sharper focus relative to the focus of the at least one out-of-focus region of the primary image further comprises:

replacing at least a part of the out-of-focus region of the primary image with at least a part of the at least one region of the other image.

19. The method of claim 18, wherein said replacing at least a part of the out-of-focus region of the primary image with at least a part of the at least one region of the other image further comprises:

utilizing at least one of tiling image processing techniques, morphing image processing techniques, blending image processing techniques, and stitching image processing techniques.

20. The method of claim 1, wherein said constructing a composite image in response to the at least one region of the other image having a sharper focus relative to the focus of the at least one out-of-focus region of the primary image further comprises:

correlating a feature of the primary image with a feature of the other image;

detecting at least one of size, color, and displacement distortion of at least one of the primary image and the other image;

correcting the detected at least one of size, color, and displacement distortion of the at least one of the primary image and the other image; and

assembling the composite image using the corrected distortion.

21. The method of claim 1, further comprising:

correcting for motion between the primary and the other image.

22. A system comprising:

a microlens array having at least one microlens deviation that exceeds a first tolerance from a target optical property;

means for capturing a primary image with a lens at a primary position;

means for determining at least one out-of-focus region of the primary image;

means for capturing another image with the lens at another position;

means for determining a focus of at least one region of the other image relative to a focus of the at least one out-of-focus region of the primary image; and

means for constructing a composite image in response to the at least one region of the other image having a sharper focus relative to the focus of the at least one out-of-focus region of the primary image.

23. The system of claim 22, wherein said microlens array having at least one microlens deviation that exceeds a first tolerance from a target optical property further comprises:

the microlens array having at least one microlens position that exceeds a first tolerance from at least one defined microlens position.

24. The system of claim 22, wherein said microlens array having at least one microlens deviation that exceeds a first tolerance from a target optical property further comprises:

a microlens array frame having at least one frame deviation that exceeds a first tolerance from at least one defined array frame position.

25. The system of claim 22, wherein said microlens array having at least one microlens deviation that exceeds a first tolerance from a target optical property further comprises:

at least one microlens having a focal length that exceeds a first tolerance from a defined focal length.

26. The system of claim 22, wherein said microlens array having at least one microlens deviation that exceeds a first tolerance from a target optical property further comprises:

at least one microlens having a spherical aberration that exceeds a first tolerance from a defined spherical aberration.

27. The system of claim 22, wherein said microlens array having at least one microlens deviation that exceeds a first tolerance from a target optical property further comprises:

at least one microlens having a cylindrical aberration that exceeds a first tolerance from a defined cylindrical aberration.

28. The system of claim 22, wherein said means for capturing a primary image with a lens at a primary position further comprises:

means for capturing the primary image at a primary focal surface location of the microlens array.

29. The system of claim 28, wherein said means for capturing the primary image at a primary focal surface location of the microlens array further comprises:

means for capturing the primary image with a photo-detector array at the primary focal surface location of the microlens array.

30. The system of claim 22, wherein said means for capturing another image with the lens at another position further comprises:

means for mapping the at least one out-of-focus region to one or more microlenses of the microlens array;

moving at least a part of the mapped one or more microlenses of the microlens array to the other position; and

capturing the other image with a photo-detector array.

31. The system of claim 30, wherein said means for mapping the at least one out-of-focus region to one or more microlenses of the microlens array further comprises:

means for projecting mathematically from a surface of the photo-detector array to the microlens array; and

means for selecting the one or more microlenses of the microlens array in response to said projecting.

32. The system of claim 22, wherein said means for mapping the at least one out-of-focus region to one or more microlenses of the microlens array further comprises:

means for moving at least a part of the microlens array to the other position.

33. The system of claim 32, wherein said means for moving at least a part of the microlens array to the other position further comprises:

means for moving at least one microlens of the microlens array to the other position; and

means for capturing the other image with a photo-detector array at an average primary focal surface location of the microlens array at the average primary position.

34. The system of claim 32, wherein said means for moving at least one microlens of the microlens array to the other position further comprises:

means for moving the at least one microlens of the microlens array to the other position, said moving constrained by a defined deviation from at least one defined microlens position.

35. The system of claim 22, wherein said means for determining at least one out-of-focus region of the primary image further comprises:

means for calculating a Fourier transform of at least a part of the primary image.

36. The system of claim 35, wherein said means for calculating a Fourier transform of at least a part of the primary image further comprises:

means for calculating a Fourier transform of at least one region of the primary image associated with at least one microlens.

37. The system of claim 22, wherein said means for determining a focus of at least one region of the other image relative to a focus of the at least one out-of-focus region of the primary image further comprises:

means for calculating a Fourier transform of at least a part of the at least one region of the other image.

38. The system of claim 37, wherein said means for calculating a Fourier transform of at least a part of the at least one region of the other image further comprises:

means for mapping at least one region of the primary image with at least one region of the other image.

39. The system of claim 37, wherein said calculating a Fourier transform of the at least one region of the other image further comprises:

means for mapping at least one region of the primary image associated with at least one specific microlens against the at least one region of the other image associated with the at least one specific microlens.

40. The system of claim 22, wherein said means for constructing a composite image in response to the at least one region of the other image having a sharper focus relative to the focus of the at least one out-of-focus region of the primary image further comprises:

means for replacing at least a part of the out-of-focus region of the primary image with at least a part of the at least one region of the other image.

41. The system of claim 40, wherein said means for replacing at least a part of the out-of-focus region of the primary image with at least a part of the at least one region of the other image further comprises:

means for utilizing at least one of tiling image processing techniques, morphing image processing techniques, blending image processing techniques, and stitching image processing techniques.

42. The system of claim 22, wherein said means for constructing a composite image in response to the at least one region of the other image having a sharper focus relative to the focus of the at least one out-of-focus region of the primary image further comprises:

means for correlating a feature of the primary image with a feature of the other image;

means for detecting at least one of size, color, and displacement distortion of at least one of the primary image and the other image;

means for correcting the detected at least one of size, color, and displacement distortion of the at least one of the primary image and the other image; and

means for assembling the composite image using the corrected distortion.

43. The system of claim 22, wherein said means for constructing a composite image in response to the at least one region of the other image having a sharper focus relative to the focus of the at least one out-of-focus region of the primary image further comprises:

means for correcting for motion between the primary and the other image.