Volumetric imaging using "virtual'' lenslets

• US 20050185062A1
• Filed: 02/24/2004
• Published: 08/25/2005
• Est. Priority Date: 02/24/2004
• Status: Active Grant

• Current Assignee: The Invention Science Fund I, L.L.C. (Intellectual Ventures LLC)
• Original Assignee: Searete LLC (Intellectual Ventures LLC)
• Inventors: Myhrvold, Nathan P., Wood, Lowell L. Jr., Hillis, W. Daniel
• Granted Patent: US 7,232,221 B2
• Time in Patent Office: Days
• US Class Current: 348/218.100
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 13/218 using spatial multiplexing
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses

Abstract

A system constructs an image using virtual lenslets.

Claims

1. A method comprising:

capturing a primary image representation with a photo-detector array at a primary position;

associating a primary set of discrete portions of the primary image representation with one or more defined virtual lenslets of a lens;

moving the photo-detector array to another position;

capturing another image representation with the photo-detector array at the other position;

associating another set of discrete portions of the other image representation with the one or more defined virtual lenslets of the lens; and

assembling a volumetric image from those virtual lenslet associated portions of the primary and the other image representation that have relatively sharper focuses.

2. The method of claim 1, wherein said capturing a primary image representation with a photo-detector array at a primary position further comprises:

extracting at least one of a red, blue, or green color component of the primary image.

3. The method of claim 2, wherein said extracting at least one of a red, blue, or green color component of the primary image further comprises:

numerically filtering the primary image.

4. The method of claim 1, wherein said associating a primary set of discrete portions of the primary image representation with one or more defined virtual lenslets of a lens further comprises:

obtaining one or more projections of the one or more defined virtual lenslets through a geometric surface in 3-space corresponding to an imaging surface of the photo-detector array at the primary position.

5. The method of claim 4, wherein said obtaining one or more projections of the one or more defined virtual lenslets through a geometric surface in 3-space corresponding to an imaging surface of the photo-detector array at the primary position further comprises:

defining one or more geometric shapes at a predefined position of a lens; and

calculating a projection of the one or more defined geometric shapes to the geometric surface in 3-space corresponding to an imaging surface of the photo-detector array at the primary position.

6. The method of claim 1, wherein said moving the photo-detector array to another position further comprises:

tilting the CCD detector array about a defined axis of tilt.

7. The method of claim 6, wherein said tilting the CCD detector array about a defined axis of tilt further comprises:

tilting a portion of the photo-detector array forward of the defined axis of tilt.

8. The method of claim 6, wherein said tilting the CCD detector array about a defined axis of tilt further comprises:

tilting a portion of the photo-detector array rearward of the defined axis of tilt.

9. The method of claim 1, wherein said moving the photo-detector array to another position further comprises:

rotating the photo-detector array about a defined axis of rotation.

10. The method of claim 1, wherein said capturing another image representation with the photo-detector array at the other position further comprises:

extracting at least one of a red, blue, or green color component of the other image.

11. The method of claim 10, wherein said extracting at least one of a red, blue, or green color component of the other image further comprises:

numerically filtering the other image.

12. The method of claim 1, wherein said associating another set of discrete portions of the other image representation with the one or more defined virtual lenslets of the lens further comprises:

obtaining one or more projections of the one or more defined virtual lenslets through a geometric surface in 3-space corresponding to an imaging surface of the photo-detector array at the other position.

13. The method of claim 12, wherein said obtaining one or more projections of the one or more defined virtual lenslets through a geometric surface in 3-space corresponding to an imaging surface of the photo-detector array at the other position further comprises:

defining one or more geometric shapes at a predefined position of a lens; and

calculating a projection of the one or more defined geometric shapes to the geometric surface in 3-space corresponding to an imaging surface of the photo-detector array at the other position.

14. The method of claim 1, wherein said assembling a volumetric image from those virtual lenslet associated portions of the primary and the other image representation that have relatively sharper focuses further comprises:

determining an image score for each of the lenslet-associated portions; and

storing the image score.

15. The method of claim 14, wherein said determining an image score for each of the lenslet-associated portions further comprises:

calculating a Fourier transform of each of the lenslet associated portions.

16. The method of claim 1, wherein said assembling a volumetric image from those virtual lenslet associated portions of the primary and the other image representation that have relatively sharper focuses further comprises:

correlating a feature of the primary image with a feature of the other image;

detecting at least one of size, color, or displacement deviation of at least one of the primary image or the other image;

correcting the detected at least one of size, color, or displacement deviation of the at least one of the primary image or the other image; and

assembling the composite image using the corrected deviation.

17. The method of claim 1, wherein said assembling a volumetric image from those virtual lenslet associated portions of the primary and the other image representation that have relatively sharper focuses further comprises:

correlating a feature of a virtual lenslet of the primary image with a feature of a virtual lenslet of the other image;

detecting at least one of size, color, or displacement deviation of at least one virtual lenslet of the primary image or at least one virtual lenslet of the other image;

correcting the detected at least one of size, color, or displacement deviation of the at least one virtual lenslet of the primary image or the at least one virtual lenslet of the other image; and

assembling the composite image using the corrected deviation.

18. A system comprising:

means for capturing a primary image representation with a photo-detector array at a primary position;

means for associating a primary set of discrete portions of the primary image representation with one or more defined virtual lenslets of a lens;

means for moving the photo-detector array to another position;

means for capturing another image representation with the photo-detector array at the other position;

means for associating another set of discrete portions of the other image representation with the one or more defined virtual lenslets of the lens; and

means for assembling a volumetric image from those virtual lenslet associated portions of the primary and the other image representations that have relatively sharper focuses.

19. The system of claim 18, wherein said means for capturing a primary image representation with a photo-detector array at a primary position further comprises:

means for extracting at least one of a red, blue, or green color component of the primary image.

20. The system of claim 19, wherein said means for extracting at least one of a red, blue, or green color component of the primary image further comprises:

means for numerically filtering the primary image.

21. The system of claim 18, wherein said means for associating a primary set of discrete portions of the primary image representation with one or more defined virtual lenslets of a lens further comprises:

means for obtaining one or more projections of the one or more defined virtual lenslets through a geometric surface in 3-space corresponding to an imaging surface of the photo-detector array at the primary position.

22. The system of claim 21, wherein said means for obtaining one or more projections of the one or more defined virtual lenslets through a geometric surface in 3-space corresponding to an imaging surface of the photo-detector array at the primary position further comprises:

means for defining one or more geometric shapes at a predefined position of a lens; and

means for calculating a projection of the one or more defined geometric shapes to the geometric surface in 3-space corresponding to an imaging surface of the photo-detector array at the primary position.

23. The system of claim 18, wherein said means for moving the photo-detector array to another position further comprises:

means for tilting the CCD detector array about a defined axis of tilt.

24. The system of claim 23, wherein said means for tilting the CCD detector array about a defined axis of tilt further comprises:

means for tilting a portion of the photo-detector array forward of the defined axis of tilt.

25. The system of claim 23, wherein said means for tilting the CCD detector array about a defined axis of tilt further comprises:

means for tilting a portion of the photo-detector array rearward of the defined axis of tilt.

26. The system of claim 18, wherein said means for moving the photo-detector array to another position further comprises:

means for rotating the photo-detector array about a defined axis of rotation.

27. The system of claim 18, wherein said means for capturing another image representation with the photo-detector array at the other position further comprises:

means for extracting at least one of a red, blue, or green color component of the other image.

28. The system of claim 27, wherein said means for extracting at least one of a red, blue, or green color component of the other image further comprises:

means for numerically filtering the other image.

29. The system of claim 18, wherein said means for associating another set of discrete portions of the other image representation with the one or more defined virtual lenslets of the lens further comprises:

means for obtaining one or more projections of the one or more defined virtual lenslets through a geometric surface in 3-space corresponding to an imaging surface of the photo-detector array at the other position.

30. The system of claim 29, wherein said means for obtaining one or more projections of the one or more defined virtual lenslets through a geometric surface in 3-space corresponding to an imaging surface of the photo-detector array at the other position further comprises:

means for defining one or more geometric shapes at a predefined position of a lens; and

means for calculating a projection of the one or more defined geometric shapes to the geometric surface in 3-space corresponding to an imaging surface of the photo-detector array at the other position.

31. The system of claim 18, wherein said means for assembling a volumetric image from those virtual lenslet associated portions of the primary and the other image representations that have relatively sharper focuses further comprises:

means for determining an image score for each of the lenslet associated portions; and

means for storing the image score.

32. The system of claim 31, wherein said means for determining an image score for each of the lenslet associated portions further comprises:

means for calculating a Fourier transform of each of the lenslet associated portions.

33. The system of claim 18, wherein said means for assembling a volumetric image from those virtual lenslet associated portions of the primary and the other image representations that have relatively sharper focuses further comprises:

means for correlating a feature of the primary image with a feature of the other image;

means for detecting at least one of size, color, or displacement deviation of at least one of the primary image and the other image;

means for correcting the detected at least one of size, color, or displacement deviation of the at least one of the primary image or the other image; and

means for assembling the composite image using the corrected deviation.

34. The system of claim 18, wherein said means for assembling a volumetric image from those virtual lenslet associated portions of the primary and the other image representations that have relatively sharper focuses further comprises:

means for correlating a feature of a virtual lenslet of the primary image with a feature of a virtual lenslet of the other image;

means for detecting at least one of size, color, or displacement deviation of at least one virtual lenslet of the primary image or at least one virtual lenslet of the other image;

means for correcting the detected at least one of size, color, or displacement deviation of the at least one virtual lenslet of the primary image or the at least one virtual lenslet of the other image; and

means for assembling the composite image using the corrected deviation.

35. A system comprising:

a lens associated with one or more virtual lenslets;

a controller configured to position a photo-detector array at a primary and another position;

an image capture unit configured to capture a primary image at the primary position and another image at the other position; and

a volumetric image construction unit configured to utilize at least a part of the one or more virtual lenslets in association with at least one of the primary image or the other image.

36. The system of claim 35, wherein said controller configured to position a photo-detector array at a primary and another position further comprises:

a transducer system having a control signal input operably coupled with said controller and a motion output operably coupled with said photo-detector array.

37. The system of claim 36, wherein said transducer system further comprises an electric motor operably coupled to move said photo-detector array.

38. The system of claim 36, wherein said transducer system further comprises an electric motor operably coupled to distort said photo-detector array.

39. The system of claim 35, wherein said image capture unit configured to capture a primary image at the primary position and another image at the other position further comprises:

circuitry for obtaining one or more projections of the one or more virtual lenslets through a geometric surface in 3-space corresponding to an imaging surface of the photo-detector array.

40. The system of claim 35, wherein said volumetric image construction unit configured to utilize at least a part of the one or more virtual lenslets in association with at least one of the primary image or the other image further comprises:

circuitry for utilizing at least a part of the one or more virtual lenslets in association with at least one of the primary image or the other image said circuitry including at least one of electrical circuitry having at least one discrete electrical circuit, electrical circuitry having at least one integrated circuit, electrical circuitry having at least one application specific integrated circuit, electrical circuitry having a general purpose computing device configured by a computer program, electrical circuitry having a memory device, or electrical circuitry having a communications device.