OPTIMIZATION OF OPTICAL SYSTEMS FOR IMPROVED LIGHT FIELD CAPTURE AND MANIPULATION

• US 20140146201A1
• Filed: 05/06/2013
• Published: 05/29/2014
• Est. Priority Date: 05/09/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Lytro Incorporated
• Inventors: Knight, Timothy, Pitts, Colvin, Akeley, Kurt, Romanenko, Yuriy, Craddock, Carl (Warren)
• Granted Patent: US 9,300,932 B2
• Time in Patent Office: Days
• US Class Current: 348/231.99
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/12 with one sensor only
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 23/957 Light-field or plenoptic cameras or camera modules
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/447 by preserving the colour pattern with or without loss of information
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

According to various embodiments of the present invention, the optical systems of light field capture devices are optimized so as to improve captured light field image data. Optimizing optical systems of light field capture devices can result in captured light field image data (both still and video) that is cheaper and/or easier to process. Optical systems can be optimized to yield improved quality or resolution when using cheaper processing approaches whose computational costs fit within various processing and/or resource constraints. As such, the optical systems of light field cameras can be optimized to reduce size and/or cost and/or increase the quality of such optical systems.

Claims

1. In a light field image capture device, the light field image capture device including at least one sensor and processing circuitry, the processing circuitry configured to process captured light field image data to generate visual output, a method for reducing the size of captured light field image data, the method comprising:

capturing light field image data at the at least one sensor, the light field image data representing output of a microlens array;

determining a read-out mode of the at least one sensor;

pre-processing the captured light field image data in accordance with a the determined read-out mode to reduce the data size of the captured light field image data; and

sending the pre-processed captured light field image data to the processing circuitry.

2. The method of claim 1, wherein pre-processing the captured light field image data comprises performing at least one selected from the group consisting of:

sub-sampling, binning, and resampling on the captured light field image data.

3. The method of claim 2, wherein pre-processing the captured light field image data comprises filtering the captured light field image data prior to performing at least one selected from the group consisting of:

sub-sampling, binning, and resampling the captured light field image data.

4. The method of claim 1, wherein pre-processing the captured light field image data comprises skipping at least one pixel on the at least one sensor, based on light field coordinates of the at least one pixel.

5. The method of claim 1, further comprising, prior to pre-processing the captured light field image data:

buffering at least a portion of the captured light field image data in local memory at the at least one sensor; and

accessing the at least a portion of the captured light field image data from the local memory.

6. The method of claim 1, wherein pre-processing the captured light field image data in accordance with a determined read-out mode comprises processing the captured light field image data in accordance with a read-out mode configured on a per-pixel basis.

7. The method of claim 6, wherein processing the captured light field image data in accordance with a read-out mode configured on a per-pixel basis comprises using at least one of:

a bitmask;

at least one bitmask tile;

at least one weight image tile; and

a per-pixel offset;

to process pixels on a per-pixel basis.

8. The method of claim 1, wherein pre-processing the captured light field image data comprises using at least one of variable bit depths and variable pixel maximum values based on the light field coordinates of at least one pixel on the at least one sensor.

9. The method of claim 1, wherein pre-processing the captured light field image data comprises performing light field aware processing on the light field image data.

10. The method of claim 1, wherein pre-processing the captured light field image data comprises:

processing at least one frame of the captured light field image data in accordance with the determined read-out mode; and

processing at least one other frame of the captured light field image data in accordance with a second different read-out mode.

11. The method of claim 1, wherein pre-processing the captured light field image data comprises reading out the captured light field image data from a buffer in a plurality of different passes.

12. The method of claim 1, wherein the at least one sensor include a charge-coupled device (CCD) image sensor, and wherein pre-processing the captured light field image data accordance with a specified read-out mode comprises implementing the specified read-out mode using the charge-coupled device (CCD) image sensor.

13. The method of claim 12, wherein implementing the specified read-out mode using the charge-coupled device (CCD) image sensor comprises implementing a reduced-resolution read-out mode using a charge-coupled device (CCD) image sensor with interlaced scan.

14. In a light field image capture device, the light field image capture device including a microlens array, at least one sensor, a disk pattern formed on the sensor, a color filter array (CFA) grid layout, and processing circuitry, the processing circuitry configured to process captured light field image data to generate visual output, the disk pattern and the color filter array (CFA) interoperating to convert output from the microlens array into captured light field image data, a method for modifying the color filter array (CFA), the method comprising:

accessing the configuration of the disk pattern;

determining how to modify the color filter array (CFA) to improve the quality of the color information captured from the output of the microlens array;

modifying the color filter array (CFA) in accordance with the determination;

capturing light field image data from the output of the microlens array in accordance with the disk pattern and the modified color filter array (CFA); and

sending the captured light field image data to the processing circuitry.

15. The method of claim 14, wherein modifying the color filter array (CFA) in accordance with the determination comprises modifying the color filter array so that no color filter is applied to at least one pixel.

16. The method of claim 14, wherein modifying the color filter array (CFA) in accordance with the determination comprises modifying color filter pass bands for at least one pixel.

17. The method of claim 14, wherein modifying the color filter array (CFA) in accordance with the determination comprises randomizing a color filter array (CFA) pattern.

18. In a light field image capture device, the light field image capture device including at least one sensor and processing circuitry, the processing circuitry configured to process captured light field image data to generate visual output, a method for modifying pixel properties of the at least one sensor, the method comprising:

capturing light field image data at the at least one sensor, the light field image data representing output of a microlens array, the light field image data including properties for each of a plurality of pixels of the at least one sensor;

for each of at least one pixel in the at least one sensor, and based on light coordinates of the at least one pixel, modifying the pixel by adjusting at least one property of the pixel to compensate for non-uniform sampling due to other characteristics of the light field image capture device; and

sending the captured light field image data, including the modified at least one pixel, to the processing circuitry.

19. The method of claim 18, wherein modifying the pixel by adjusting at least one property of the pixel to compensate for non-uniform sampling due to other characteristics of the light field image capture device comprises adjusting at least one property of the pixel to compensate for factors related to at least one of:

image sensor angular response, a microlens array, and a main lens.

20. The method of claim 18, wherein modifying the pixel by adjusting at least one property of the pixel to compensate for non-uniform sampling due to other characteristics of the light field image capture device comprises adjusting at least one of:

exposure duration, gain, and bias for the pixel.

21. The method of claim 18, wherein modifying the pixel by adjusting at least one property of the pixel to compensate for non-uniform sampling due to other characteristics of the light field image capture device comprises implementing pixel adjustment using at least one of:

post-fabrication location trimming of a semiconductor, a digital or analog signal, and fusing a scale or offset into a semiconductor.

22. In a light field image capture device, the light field image capture device including an aperture, a main lens, a microlens array, at least one sensor, and processing circuitry, the lenses on the microlens array arranged in a regular pattern, the pixels on the at least one sensor arranged in a regular pattern, the microlens array and at least one sensor interoperating to capture light field image data from light passing through the aperture and the main lens, a method for perturbing at least one regular pattern of the light field image capture device to increase the quality of captured light field image data, the method comprising:

accessing light that has passed through the aperture and the main leans;

introducing random variations into at least one of;

the regular pattern of lenses in the microlens array and the regular pattern of pixels in the at least one sensor;

subsequent to introduction of the random variations, capturing light field image data from the accessed light; and

sending the captured light field image data to the processing circuitry.

23. The method of claim 22, wherein introducing random variations into at least one of:

the regular pattern of lenses and the regular pattern of pixels comprises introducing a jitter pattern into at least one of;

the regular pattern of lenses and the regular pattern of pixels.

24. The method of claim 23, wherein introducing a jitter pattern into at least one of the regular pattern of lenses and the regular pattern of pixels comprises introducing a jitter pattern into the regular pattern of lenses that extends across the entire microlens array.

25. The method of claim 23, wherein introducing a jitter pattern into at least one of the regular pattern of lenses and the regular pattern of pixels comprises introducing a jitter pattern into the regular pattern of pixels that extends across the entirety of at least one sensor.

26. The method of claim 23, wherein introducing a jitter pattern into at least one of the regular pattern of lenses and the regular pattern of pixels comprises introducing a jitter pattern tile into the regular pattern of lenses, the jitter pattern tile being repeated a plurality of times across the entire microlens array.

27. The method of claim 23, wherein introducing a jitter pattern into at least one of the regular pattern of lenses and the regular pattern of pixels comprises introducing a jitter pattern tile into the regular pattern of pixels, the jitter pattern tile being repeated a plurality of times across the entire at least one sensor.

28. An optical system, comprising:

a main lens;

a microlens array, the microlens array including a plurality of microlenses; and

at least one image sensor, each image sensor having a plurality of pixels;

wherein the main lens, the microlens array, and the at least one image sensor are configured and arranged relative to one another so that at least one of the horizontal pitch and vertical pitch of disk images on the one of more light image sensors equals an integer number of pixels.

29. The optical system of claim 28, wherein the main lens, the microlens array, and the at least one image sensor are configured and arranged relative to one another so that the disk images are a square lattice of N×

N pixels.

30. The optical system of claim 29, wherein the main lens, the microlens array, and the one or more light image sensors are configured and arranged relative to one another so as to transpose an array of disks representation of light field image data into an equivalent array of sub-aperture images representation of the light field image data.

31. The optical system of claim 30, wherein the main lens, the microlens array, and the one or more light image sensors are configured and arranged to transpose an array of disks representation of light field image data into an equivalent array of sub-aperture images representation of the light field image data on at least one of:

raw domain data and full color data.

32. The optical system of claim 30, wherein the main lens, the microlens array, and the one or more light image sensors are configured and arranged to reorder light field image data into reordered light field image data that is directly interpretable as a 2D image.

33. The optical system of claim 28, wherein the microlens array has a hexagonal layout.

34. A method for determining a configuration for a microlens array that is to be positioned over an image sensor for use in a light field image capture device, the method comprising:

accessing characteristics of the microlens array;

accessing characteristics of the image sensor, including the read-out modes of the image sensor;

determining an optimal configuration for the microlens array based on the characteristics of the microlens array and the characteristics of the image sensor, including the read-out modes of the image sensor; and

storing the optimal configuration for application to the microlens array during manufacture of the light field capture device.

35. The method of claim 34, wherein determining an optimal configuration for the microlens array comprises determining an optimal position for the microlens array over the image sensor.

36. The method of claim 34, wherein determining an optimal configuration for the microlens array comprises determining an optimal orientation for the microlens array over the image sensor.

37. The method of claim 34, wherein determining an optimal configuration for the microlens array comprises considering at least one physical rotation of the microlens array about an axis orthogonal to the image sensor.

38. The method of claim 34, wherein determining an optimal configuration for the microlens array comprises considering at least one physical translation of the microlens array in a plane parallel to the image sensor plane.

39. The method of claim 34, wherein determining an optimal configuration for the microlens array comprises considering at least one read-out mode of the sensor

40. The method of claim 39, wherein each read-out modes is selected from the group consisting of:

binning, skipping, resampling, and scaling.

41. A computer program product for use in a light field image capture device, the light field image capture device including at least one sensor and processing circuitry, the processing circuitry configured to process captured light field image data to generate visual output, the computer program product for implementing a method for reducing the size of captured light field image data, the computer program product comprising at least one computer storage device having stored thereon computer-executable instructions that, when executed at a processor, cause the light field image capture device to:

capture light field image data at the at least one sensor, the light field image data representing output of a microlens array;

determine a read-out mode of the at least one sensor;

pre-process the captured light field image data in accordance with a the determined read-out mode to reduce the data size of the captured light field image data; and

send the pre-processed captured light field image data to the processing circuitry.

42. The computer program product of claim 41, wherein the computer program product is configured to cause the light field image capture device to pre-process the captured light field image data by:

processing at least one frame of the captured light field image data in accordance with the determined read-out mode; and

processing at least one other frame of the captured light field image data in accordance with a second different read-out mode.

43. A computer program product for use in a light field image capture device, the light field image capture device including a microlens array, at least one sensor, a disk pattern formed on the sensor, a color filter array (CFA) grid layout, and processing circuitry, the processing circuitry configured to process captured light field image data to generate visual output, the disk pattern and the color filter array (CFA) interoperating to convert output from the microlens array into captured light field image data, the computer program product comprising at least one computer storage device having stored thereon computer-executable instructions that, when executed at a processor, cause the light field image capture device to:

access the configuration of the disk pattern;

determine how to modify the color filter array (CFA) to improve the quality of the color information captured from the output of the microlens array;

modify the color filter array (CFA) in accordance with the determination;

capture light field image data from the output of the microlens array in accordance with the disk pattern and the modified color filter array (CFA); and

send the captured light field image data to the processing circuitry.

44. A computer program product for use in a light field image capture device, the light field image capture device including at least one sensor and processing circuitry, the processing circuitry configured to process captured light field image data to generate visual output, the computer program product comprising at least one computer storage device having stored thereon computer-executable instructions that, when executed at a processor, cause the light field image capture device to:

capture light field image data at the at least one sensor, the light field image data representing output of a microlens array, the light field image data including properties for each of a plurality of pixels of the at least one sensor;

for each of at least one pixel in the at least one sensor, and based on light coordinates of the at least one pixel, modify the pixel by adjusting at least one property of the pixel to compensate for non-uniform sampling due to other characteristics of the light field image capture device; and

send the captured light field image data, including the modified at least one pixel, to the processing circuitry.

45. The computer program product of claim 44, wherein the computer program product is configured to cause the light field image capture device to modify the pixel by adjusting at least one property of the pixel to compensate for factors related to at least one of:

image sensor angular response, a microlens array, and a main lens.

46. The computer program product of claim 44, wherein the computer program product is configured to cause the light field image capture device to modify the pixel by adjusting at least one of:

exposure duration, gain, and bias for the pixel.

47. A computer program product for use in a light field image capture device, the light field image capture device including an aperture, a main lens, a microlens array, at least one sensor, and processing circuitry, the lenses on the microlens array arranged in a regular pattern, the pixels on the at least one sensor arranged in a regular pattern, the microlens array and at least one sensor interoperating to capture light field image data from light passing through the aperture and the main lens, the computer program product comprising at least one computer storage device having stored thereon computer-executable instructions that, when executed at a processor, cause the light field image capture device to:

access light that has passed through the aperture and the main leans;

introduce random variations into at least one of;

the regular pattern of lenses in the microlens array and the regular pattern of pixels in the at least one sensor;

subsequent to introduction of the random variations, capture light field image data from the accessed light; and

send the captured light field image data to the processing circuitry.

48. A light field image capture device, comprising:

a microlens array;

at least one sensor, configured to capture light field image data representing output of the microlens array;

a pre-processor, configured to determine a read-out mode of the at least one sensor and to pre-process the captured light field image data in accordance with a the determined read-out mode to reduce the data size of the captured light field image data; and

processing circuitry, configured to process the pre-processed captured light field image data to generate visual output.

49. The device of claim 48, wherein the pre-processor is configured to pre-process the captured light field image data by:

processing at least one frame of the captured light field image data in accordance with the determined read-out mode; and

processing at least one other frame of the captured light field image data in accordance with a second different read-out mode.

50. A light field image capture device, comprising:

a microlens array;

at least one sensor, configured to capture light field image data representing output of the microlens array, the light field image data comprising a disk pattern formed on the sensor, the at least one sensor having a color filter array (CFA) grid layout;

a pre-processor, configured to access the configuration of the disk pattern and to determine how to modify the color filter array (CFA) to improve the quality of the color information captured from the output of the microlens array;

a CFA modifier, configured to modify the color filter array (CFA) in accordance with the determination; and

processing circuitry, configured to process the pre-processed light field image data to generate visual output;

wherein the disk pattern and the color filter array (CFA) interoperate to convert output from the microlens array into captured light field image data;

and wherein the at least one sensor captures light field image data from the output of the microlens array in accordance with the disk pattern and the modified color filter array (CFA).

51. A light field image capture device, comprising:

a microlens array;

at least one sensor, configured to capture light field image data representing output of the microlens array, the light field image data including properties for each of a plurality of pixels of the at least one sensor;

a pre-processor, configured to, for each of at least one pixel in the at least one sensor, and based on light coordinates of the at least one pixel, modify the pixel by adjusting at least one property of the pixel to compensate for non-uniform sampling due to other characteristics of the light field image capture device; and

processing circuitry, configured to process the pre-processed light field image data to generate visual output.

52. A light field image capture device, comprising:

an aperture;

a main lens;

a microlens array having lenses arranged in a regular pattern;

at least one sensor having pixels arranged in a regular pattern, the at least one sensor configured to capture light field image data representing output of the microlens array, the microlens array and the at least one sensor interoperating to capture light field image data from light passing through the aperture and the main lens;

a pre-processor, configured to access light that has passed through the aperture and the main leans and to introduce random variations into at least one of;

the regular pattern of lenses in the microlens array and the regular pattern of pixels in the at least one sensor;

processing circuitry, configured to, subsequent to introduction of the random variations, capture light field image data from the accessed light and process the pre-processed light field image data to generate visual output.