LIGHT-FIELD ABERRATION CORRECTION

• US 20150178923A1
• Filed: 12/17/2014
• Published: 06/25/2015
• Est. Priority Date: 12/24/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Lytro Incorporated
• Inventors: Liang, Chia-Kai, Pitts, Colvin, Craddock, Carl (Warren), Myhre, Graham Butler
• Granted Patent: US 9,628,684 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/957 Light-field or plenoptic cameras or camera modules
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"

Abstract

According to various embodiments, the system and method disclosed herein serve to at least partially compensate for departures of an actual main lens of a light-field camera from the properties of an ideal main lens. Light-field data may be captured and processed through the use of product calibration data and unit calibration data. The product calibration data may be descriptive of departure of a main lens design of the light-field camera from an ideal main lens design. The unit calibration data may be descriptive of departure of the actual main lens of the light-field camera from the main lens design. Corrected light-field data may be generated as a result of the processing, and may be used to generate a light-field image.

Claims

1. A method for processing light-field data captured by a light-field image capture device having a sensor, a main lens, and a plurality of microlenses, the method comprising:

in a processor, receiving light-field data and metadata from the light-field image capture device;

in the processor, receiving product calibration data descriptive of a design departure of a main lens design of the main lens from an ideal main lens design;

in the processor, receiving unit calibration data descriptive of a manufacturing departure of the main lens from the main lens design;

in the processor, using the metadata, the product calibration data, and the unit calibration data to obtain corrected light-field data that at least partially compensates for the design departure and the manufacturing departure;

in the processor, using the corrected light-field data during light field processing to create at least one output image; and

initiating storage, in a data store, of the at least one output image.

2. The method of claim 1, wherein each output image comprises a light-field image.

3. The method of claim 1, further comprising, prior to receiving the product calibration data, obtaining the product calibration data by:

selecting a plurality of samples, each of which comprises a zoom setting and a focus setting for the light-field image capture device;

in a processor, receiving a main lens design of the main lens and a sensor design of the sensor;

in the processor, for each sample, raytracing light-field capture at the sample in a virtual light-field capture device comprising the main lens design and the sensor design to generate a raw mapping table that maps each of a plurality of sensor locations on the sensor to ideal light ray coordinates; and

initiating storage, in a data store, of product calibration data obtained from the raw mapping tables of the plurality of samples.

4. The method of claim 3, further comprising, at the processor and for each raw mapping table, generating a model that mathematically approximates the raw mapping table;

wherein initiating storage of the product calibration data comprises storing the model for each of the plurality of samples.

5. The method of claim 4, wherein, for each raw mapping table, generating the model comprises generating a selection from the group consisting of:

a polynomial function that can be used to approximate the raw mapping table, wherein the polynomial function can be applied to obtain the ideal light ray coordinates; and

a low-resolution mapping table obtained by resampling and averaging data from the raw mapping table at a lower resolution such that, from the low-resolution mapping table, the ideal light ray coordinates are obtainable via interpolation.

6. The method of claim 1, further comprising, prior to receiving the unit calibration data, obtaining the unit calibration data by:

selecting a plurality of samples, each of which comprises a zoom setting and a focus setting for the light-field image capture devices;

in a processor, receiving a main lens design of the main lens and a sensor design of the sensor;

in the processor, for each sample, generating a mapping function that maps the sample to non-ideal light ray coordinates; and

initiating storage, in a data store, of unit calibration data obtained from the mapping functions of the plurality of samples.

7. The method of claim 1, wherein using the metadata, the product calibration data, and the unit calibration data to obtain corrected light-field data comprises:

for each of a plurality of sensor locations on the sensor, using the unit calibration data to obtain non-ideal light ray coordinates based on the light-field data.

8. The method of claim 7, wherein using the metadata, the product calibration data, and the unit calibration data to obtain corrected light-field data further comprises:

for each of the plurality of sensor locations on the sensor, using a reverse mapping function obtained from the product calibration data to obtain ideal light ray coordinates from the non-ideal light ray coordinates.

9. The method of claim 8, wherein using the metadata, the product calibration data, and the unit calibration data to obtain corrected light-field data further comprises:

at the processor, receiving, from the metadata, an actual zoom setting and an actual focus setting that were used to capture the light-field data;

locating a portion of the product calibration data that most closely pertains to the actual zoom setting and the actual focus setting; and

obtaining the reverse mapping function based on the portion of the product calibration data.

10. The method of claim 8, wherein using the corrected light-field data to create the at least one output image comprises using the ideal light ray coordinates for each of the plurality of sensor locations to carry out one or more image processing steps selected from the group consisting of:

digital refocusing;

extended depth of field projection;

general projection;

light-field filtering;

denoising;

light-field resampling; and

light-field image compression.

11. The method of claim 1, wherein the each output image comprises an epipolar image, the method further comprising:

at the processor, identifying edges within the epipolar image;

computing slopes of the edges;

computing straightness levels of the edges; and

at the processor, using the slopes and the straightness levels to obtain a correction score indicative of a degree to which the corrected light-field data compensates for the design departure and the manufacturing departure.

12. A method for obtaining product calibration data for a plurality of light-field image capture device, each of which has a sensor, a main lens, and a plurality of microlenses, the method comprising:

selecting a plurality of samples, each of which comprises a zoom setting and a focus setting for the light-field image capture devices;

in a processor, receiving a main lens design of the main lens and a sensor design of the sensor;

in the processor, for each sample, raytracing light-field capture at the sample in a virtual light-field capture device comprising the main lens design and the sensor design to generate a raw mapping table that maps each of a plurality of sensor locations on the sensor to ideal light ray coordinates; and

in a data store, storing product calibration data obtained from the raw mapping tables of the plurality of samples.

13. The method of claim 12, further comprising, at the processor and for each raw mapping table, generating a model that mathematically approximates the raw mapping table;

wherein storing the product calibration data comprises storing the model for each of the plurality of samples.

14. The method of claim 13, wherein, for each of the plurality of samples, storing the model comprises occupying a storage space smaller than would be required to store the raw mapping table.

15. The method of claim 13, wherein, for each raw mapping table, generating the model comprises generating a polynomial function that can be used to approximate the raw mapping table, wherein the polynomial function can be applied to obtain the ideal light ray coordinates.

16. The method of claim 13, wherein, for each raw mapping table, generating the model comprises resampling and averaging data from the raw mapping table at a lower resolution to provide a low-resolution mapping table from which the ideal light ray coordinates are obtainable via interpolation.

17. A computer program product for processing light-field data captured by a light-field image capture device having a sensor, a main lens, and a plurality of microlenses, the computer program product comprising:

a non-transitory storage medium; and

computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of;

receiving light-field data and metadata from the light-field image capture device;

receiving product calibration data descriptive of a design departure of a main lens design of the main lens from an ideal main lens design;

receiving unit calibration data descriptive of a manufacturing departure of the main lens from the main lens design;

using the metadata, the product calibration data, and the unit calibration data to obtain corrected light-field data that at least partially compensates for the design departure and the manufacturing departure;

using the corrected light-field data during light field processing to create at least one output image; and

causing a data store to store the at least one output image.

18. The computer program product of claim 17, wherein each output image comprises a light-field image.

19. The computer program product of claim 17, wherein the computer program code is further configured to cause the at least one processor, prior to receiving the product calibration data, to obtain the product calibration data by:

selecting a plurality of samples, each of which comprises a zoom setting and a focus setting for the light-field image capture device;

receiving a main lens design of the main lens and a sensor design of the sensor;

for each sample, raytracing light-field capture at the sample in a virtual light-field capture device comprising the main lens design and the sensor design to generate a raw mapping table that maps each of a plurality of sensor locations on the sensor to ideal light ray coordinates; and

initiating storage of product calibration data obtained from the raw mapping tables of the plurality of samples.

20. The computer program product of claim 19, wherein the computer program code is further configured to cause the at least one processor, for each raw mapping table, to generate a model that mathematically approximates the raw mapping table;

wherein the compute program code configured to cause the at least one processor to initiate storage of the product calibration data comprises computer program code configured to cause the at least one processor to initiate storage of the model for each of the plurality of samples.

21. The computer program product of claim 17, wherein the computer program code is further configured to cause the at least one processor, prior to receiving the unit calibration data, to obtain the unit calibration data by:

selecting a plurality of samples, each of which comprises a zoom setting and a focus setting for the light-field image capture devices;

receiving a main lens design of the main lens and a sensor design of the sensor;

for each sample, generating a mapping function that maps the sample to non-ideal light ray coordinates; and

causing a data store to store unit calibration data obtained from the mapping functions of the plurality of samples.

22. The computer program product of claim 17, wherein the computer program code configured to cause the at least one processor to use the metadata, the product calibration data, and the unit calibration data to obtain corrected light-field data comprises computer program code configured to cause the at least one processor to perform the steps of:

receiving, from the metadata, an actual zoom setting and an actual focus setting that were used to capture the light-field data;

locating a portion of the product calibration data that most closely pertains to the actual zoom setting and the actual focus setting;

obtaining a reverse mapping function based on the portion of the product calibration data;

for each of a plurality of sensor locations on the sensor, using the unit calibration data to obtain non-ideal light ray coordinates based on the light-field data; and

for each of the plurality of sensor locations on the sensor, using the reverse mapping function to obtain ideal light ray coordinates from the non-ideal light ray coordinates.

23. The computer program product of claim 22, wherein the computer program code configured to cause the at least one processor to use the corrected light-field data to create the at least one output image comprises computer program code configured to cause the at least one processor to use the ideal light ray coordinates for each of the plurality of sensor locations to carry out one or more image processing steps selected from the group consisting of:

digital refocusing;

extended depth of field projection;

general projection;

light-field filtering;

denoising;

light-field resampling; and

light-field image compression.

24. The computer program product of claim 17, wherein each output image comprises an epipolar image, wherein the computer program code is further configured to cause the at least one processor to perform the steps of:

identifying edges within the epipolar image;

computing slopes of the edges;

computing straightness levels of the edges; and

using the slopes and the straightness levels to obtain a correction score indicative of a degree to which the corrected light-field data compensates for the design departure and the manufacturing departure.

25. A computer program product for obtaining product calibration data for a plurality of light-field image capture device, each of which has a sensor, a main lens, and a plurality of microlenses, the computer program product comprising:

a non-transitory storage medium; and

computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of;

selecting a plurality of samples, each of which comprises a zoom setting and a focus setting for the light-field image capture devices;

receiving a main lens design of the main lens and a sensor design of the sensor;

for each sample, raytracing light-field capture at the sample in a virtual light-field capture device comprising the main lens design and the sensor design to generate a raw mapping table that maps each of a plurality of sensor locations on the sensor to ideal light ray coordinates; and

causing a data store to store product calibration data obtained from the raw mapping tables of the plurality of samples.

26. The computer program product of claim 25, wherein the computer program code is further configured to cause the at least one processor, for each raw mapping table, to generate a model that mathematically approximates the raw mapping table;

wherein the computer program code configured to cause the data store to store the product calibration data comprises computer program code configured to cause the data store to store the model for each of the plurality of samples.

27. The computer program product of claim 26, wherein, the computer program code configured to cause the at least one processor, for each raw mapping table, to generate the model comprises computer program code configured to cause the at least one processor to perform a step selected from the group consisting of:

generating a polynomial function that can be used to approximate the raw mapping table, wherein the polynomial function can be applied to obtain the ideal light ray coordinates; and

resampling and averaging data from the raw mapping table at a lower resolution to provide a low-resolution mapping table from which the ideal light ray coordinates are obtainable via interpolation.

28. A system for processing light-field data captured by a light-field image capture device having a sensor, a main lens, and a plurality of microlenses, the system comprising:

a processor, configured to;

receive light-field data and metadata from the light-field image capture device;

receive product calibration data descriptive of a design departure of a main lens design of the main lens from an ideal main lens design;

receive unit calibration data descriptive of a manufacturing departure of the main lens from the main lens design;

use the metadata, the product calibration data, and the unit calibration data to obtain corrected light-field data that at least partially compensates for the design departure and the manufacturing departure; and

use the corrected light-field data during light field processing to create at least one output image; and

a data store, communicatively coupled to the processor, configured to;

store the at least one output image.

29. The system of claim 28, wherein each output image comprises a light-field image.

30. The system of claim 28, wherein the processor is further configured, prior to receiving the product calibration data, to obtain the product calibration data by:

selecting a plurality of samples, each of which comprises a zoom setting and a focus setting for the light-field image capture device;

receiving a main lens design of the main lens and a sensor design of the sensor;

for each sample, raytracing light-field capture at the sample in a virtual light-field capture device comprising the main lens design and the sensor design to generate a raw mapping table that maps each of a plurality of sensor locations on the sensor to ideal light ray coordinates; and

initiating storage of product calibration data obtained from the raw mapping tables of the plurality of samples.

31. The system of claim 30, wherein the processor is further configured, for each raw mapping table, to generate a model that mathematically approximates the raw mapping table;

wherein the processor is further configured to initiate storage of the product calibration data by initiating storage of the model for each of the plurality of samples.

32. The system of claim 28, wherein the processor is further configured, prior to receiving the unit calibration data, to obtain the unit calibration data by:

selecting a plurality of samples, each of which comprises a zoom setting and a focus setting for the light-field image capture devices;

receiving a main lens design of the main lens and a sensor design of the sensor;

for each sample, generating a mapping function that maps the sample to non-ideal light ray coordinates; and

initiating storage of unit calibration data obtained from the mapping functions of the plurality of samples.

33. The system of claim 28, wherein the processor is further configured to use the metadata, the product calibration data, and the unit calibration data to obtain corrected light-field data comprises by:

receiving, from the metadata, an actual zoom setting and an actual focus setting that were used to capture the light-field data;

locating a portion of the product calibration data that most closely pertains to the actual zoom setting and the actual focus setting;

obtaining a reverse mapping function based on the portion of the product calibration data;

for each of a plurality of sensor locations on the sensor, using the unit calibration data to obtain non-ideal light ray coordinates based on the light-field data; and

for each of the plurality of sensor locations on the sensor, using the reverse mapping function to obtain ideal light ray coordinates from the non-ideal light ray coordinates.

34. The system of claim 33, wherein the processor is further configured to use the corrected light-field data to create the at least one output image by using the ideal light ray coordinates for each of the plurality of sensor locations to carry out one or more image processing steps selected from the group consisting of:

digital refocusing;

extended depth of field projection;

general projection;

light-field filtering;

denoising;

light-field resampling; and

light-field image compression.

35. The system of claim 28, wherein each output image comprises an epipolar image, wherein the processor is further configured to:

identify edges within the epipolar image;

compute slopes of the edges;

compute straightness levels of the edges; and

use the slopes and the straightness levels to obtain a correction score indicative of a degree to which the corrected light-field data compensates for the design departure and the manufacturing departure.

36. A system for obtaining product calibration data for a plurality of light-field image capture device, each of which has a sensor, a main lens, and a plurality of microlenses, the system comprising:

a processor configured to;

select a plurality of samples, each of which comprises a zoom setting and a focus setting for the light-field image capture devices;

receive a main lens design of the main lens and a sensor design of the sensor; and

for each sample, raytracing light-field capture at the sample in a virtual light-field capture device comprising the main lens design and the sensor design to generate a raw mapping table that maps each of a plurality of sensor locations on the sensor to ideal light ray coordinates; and

a data store, communicatively coupled to the processor, configured to;

store product calibration data obtained from the raw mapping tables of the plurality of samples.

37. The system of claim 36, wherein the processor is further configured, for each raw mapping table, to generate a model that mathematically approximates the raw mapping table;

wherein the data store is further configured to store the product calibration data by storing the model for each of the plurality of samples.

38. The system of claim 37, wherein, for each raw mapping table, the processor is further configured to generate the model by performing a step selected from the group consisting of:

generating a polynomial function that can be used to approximate the raw mapping table, wherein the polynomial function can be applied to obtain the ideal light ray coordinates; and

resampling and averaging data from the raw mapping table at a lower resolution to provide a low-resolution mapping table from which the ideal light ray coordinates are obtainable via interpolation.