CONFIGURING TWO-DIMENSIONAL IMAGE PROCESSING BASED ON LIGHT-FIELD PARAMETERS

• US 20140176592A1
• Filed: 10/10/2013
• Published: 06/26/2014
• Est. Priority Date: 02/15/2011
• Status: Abandoned Application

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Lytro Incorporated
• Inventors: Wilburn, Bennett, Pitts, Colvin, Liang, Chia-Kai, Knight, Timothy, Poon, Tony Yip Pang, Carroll, Robert
• Application Number:
  US14/051,263
• Publication Number:
  US 20140176592A1
• Time in Patent Office: Days
• US Class Current: 345/589
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 23/673 based on contrast or high frequency components of image signals, e.g. hill climbing method
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/957 Light-field or plenoptic cameras or camera modules
• CPC Class Codes: H04N 23/958 for extended depth of field imaging

Abstract

According to various embodiments, the present may be used to apply a wide variety of processes to a two-dimensional image generated from light-field data. One or more parameters, such as light-field parameters and/or device capture parameters may be included in metadata of the two-dimensional image, and may be retrieved and processed to determine the appropriate value(s) of a first setting of the process. The process may be applied uniformly, or with variation across subsets of the two-dimensional image, down to individual pixels. The process may be a noise filtering process, an image sharpening process, a color adjustment process, a tone curve process, a contrast adjustment process, a saturation adjustment process, a gamma adjustment process, a combination thereof, or any other known process that may be desirable for enhancing two-dimensional images.

Claims

1. A method for processing a two-dimensional image projected from light-field data, comprising:

at a processor, retrieving a two-dimensional image projected from light-field data;

at the processor, retrieving at least one parameter associated with the two-dimensional image;

at the processor, based on the parameter, determining a first setting of a process; and

at the processor, applying the process with the first setting to the two-dimensional image to generate a processed two-dimensional image.

2. The method of claim 1, wherein the parameter describes the picture being generated from the light-field.

3. The method of claim 1, wherein the parameter is derived from the light-field data.

4. The method of claim 3, wherein the process comprises a non-photorealistic rendering technique selected from the group consisting of:

a magnification process by which an in-focus region of the two-dimensional image is magnified relative to a defocused region of the two-dimensional image;

an artistic simulation process by which the two-dimensional image is modified to simulate a painting with a brush stroke size that is larger in a defocused region of the two-dimensional image than in a focused region of the two-dimensional image;

a stippling filter that uses larger stipples in a defocused region of the two-dimensional image than in a focused region of the two-dimensional image; and

an edge-preserving smoothing filter with a larger radius in a defocused region of the two-dimensional image than in a focused region of the two-dimensional image.

5. The method of claim 1, wherein the process is selected from the group consisting of:

a noise filtering process;

an image sharpening process;

a color adjustment process;

a tone curve process;

a contrast adjustment process;

a saturation adjustment process; and

a gamma adjustment process.

6. The method of claim 1, wherein the parameter is selected from the group consisting of:

a target refocus depth applicable to the entire two-dimensional image;

a measured lambda value at a pixel of the two-dimensional image, wherein the lambda value indicates a distance perpendicular to a microlens array of a light-field capture device used to capture the light-field data;

a difference between the measured lambda and a target refocus depth at a pixel of the two-dimensional image; and

a click-to-focus depth value at a pixel of the two-dimensional image, wherein the click-to-focus depth value comprises a depth selected by a user for interactive refocusing of the two-dimensional image.

7. The method of claim 6, wherein the parameter comprises a measured lambda value at a pixel of the two-dimensional image, wherein the lambda value indicates a distance perpendicular to a microlens array of a light-field capture device used to capture the light-field data, and wherein the process comprises an image gain adjustment process that adjusts brightness of the pixel based on the lambda value.

8. The method of claim 6, wherein the parameter comprises a measured lambda value at a pixel of the two-dimensional image, wherein the lambda value indicates a distance perpendicular to a microlens array of a light-field capture device used to capture the light-field data, wherein the method further comprises:

using the parameter to determine a degree of high-frequency detail in the two-dimensional image;

and wherein the first setting is determined based the degree of high-frequency detail.

9. The method of claim 8, wherein using the parameter to determine a degree of high-frequency detail in the two-dimensional image comprises:

above a high threshold or below a low threshold, determining that the pixel has relatively little high-frequency detail; and

responsive to the lambda value at the pixel being above the low threshold and below the high threshold, determining that the pixel has a relatively higher amount of high-frequency detail.

10. The method of claim 1, wherein the parameter comprises a center-of-perspective parameter, wherein the process is selected from the group consisting of:

a vignetting filter that modifies the two-dimensional image to simulate changes in viewpoint when looking through a lens; and

a focus breathing filter that modifies the two-dimensional image to simulate magnification and/or reduction as a focus of a lens system is adjusted.

11. The method of claim 1, wherein the two-dimensional image comprises metadata comprising the parameter.

12. The method of claim 1, further comprising, prior to applying the process to the two-dimensional image:

at the processor, applying a reconstruction filter to the two-dimensional image to reduce aliasing artifacts of the two-dimensional image and/or increase sharpness of the two-dimensional image.

13. The method of claim 12, wherein the process comprises a noise filter, wherein applying the process to the two-dimensional image comprises reducing a noise level of the processed two-dimensional image.

14. The method of claim 13, wherein the noise filter comprises an unsharp mask, wherein the first setting comprises a blur kernel width of the unsharp mask, wherein determining the first setting comprises:

if the reconstruction filter used a low width, selecting a low width for the blur kernel; and

if the reconstruction filter used a high width, selecting a high width for the blur kernel.

15. The method of claim 14, further comprising:

based on the parameter, determining a second setting of the process;

and wherein the second setting comprises an unsharp amount of the unsharp mask, wherein the unsharp amount comprises a multiple of a high-pass image to be added to the two-dimensional image.

16. The method of claim 1, wherein the first setting is applicable to all pixels of the two-dimensional image, wherein applying the process to the two-dimensional image comprises applying the process with the first setting to all pixels of the two-dimensional image.

17. The method of claim 1, further comprising:

based on the parameter, determining a second setting of the process;

and wherein applying the process to the two-dimensional image comprises;

applying the process with the first setting to a first pixel of the two-dimensional image; and

applying the process with the second setting to a second pixel of the two-dimensional image.

18. The method of claim 17, wherein the two-dimensional image comprises an extended depth-of-field (EDOF) image comprising a non-planar virtual focal surface.

19. The method of claim 1, further comprising, prior to retrieving the two-dimensional image:

retrieving a two-dimensional calibration image projected from calibration light-field data;

performing the method on the two-dimensional calibration image with a plurality of values of the first setting to generate a processed two-dimensional calibration image; and

using the processed two-dimensional calibration image to determine which of the plurality values of the first setting should be used with each of a plurality of values of the parameter.

20. A computer program product for processing a two-dimensional image projected from light-field data, comprising:

a non-transitory computer-readable storage medium; and

computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of;

retrieving a two-dimensional image projected from light-field data;

retrieving at least one parameter associated with the two-dimensional image;

based on the parameter, determining a first setting of a process; and

applying the process with the first setting to the two-dimensional image to generate a processed two-dimensional image.

21. The computer program product of claim 20, wherein the parameter describes the picture being generated from the light-field.

22. The computer program product of claim 20, wherein the parameter is derived from the light-field data.

23. The computer program product of claim 22, wherein the process comprises a non-photorealistic rendering technique selected from the group consisting of:

a magnification process by which an in-focus region of the two-dimensional image is magnified relative to a defocused region of the two-dimensional image;

an artistic simulation process by which the two-dimensional image is modified to simulate a painting with a brush stroke size that is larger in a defocused region of the two-dimensional image than in a focused region of the two-dimensional image;

a stippling filter that uses larger stipples in a defocused region of the two-dimensional image than in a focused region of the two-dimensional image; and

an edge-preserving smoothing filter with a larger radius in a defocused region of the two-dimensional image than in a focused region of the two-dimensional image.

24. The computer program product of claim 20, wherein the process is selected from the group consisting of:

a noise filtering process;

an image sharpening process;

a color adjustment process;

a tone curve process;

a contrast adjustment process;

a saturation adjustment process; and

a gamma adjustment process.

25. The computer program product of claim 20, wherein the parameter is selected from the group consisting of:

a target refocus depth applicable to the entire two-dimensional image;

a measured lambda value at a pixel of the two-dimensional image, wherein the lambda value indicates a distance perpendicular to a microlens array of a light-field capture device used to capture the light-field data;

a difference between the measured lambda and a target refocus depth at a pixel of the two-dimensional image; and

a click-to-focus depth value at a pixel of the two-dimensional image, wherein the click-to-focus depth value comprises a depth selected by a user for interactive refocusing of the two-dimensional image.

26. The computer program product of claim 25, wherein the parameter comprises a measured lambda value at a pixel of the two-dimensional image, wherein the lambda value indicates a distance perpendicular to a microlens array of a light-field capture device used to capture the light-field data, and wherein the process comprises an image gain adjustment process that adjusts brightness of the pixel based on the lambda value.

27. The computer program product of claim 25, wherein the parameter comprises a measured lambda value at a pixel of the two-dimensional image, wherein the lambda value indicates a distance perpendicular to a microlens array of a light-field capture device used to capture the light-field data, and wherein the computer program code is further configured to cause the processor to perform the step of:

using the parameter to determine a degree of high-frequency detail in the two-dimensional image;

wherein the first setting is determined based the degree of high-frequency detail;

and wherein using the parameter to determine a degree of high-frequency detail in the two-dimensional image comprises;

responsive to the lambda value at the pixel being above a high threshold or below a low threshold, determining that the pixel has relatively little high-frequency detail; and

responsive to the lambda value at the pixel being above the low threshold and below the high threshold, determining that the pixel has a relatively higher amount of high-frequency detail.

28. The computer program product of claim 20, wherein the computer program code is further configured to cause the processor to perform the step of:

applying a reconstruction filter to the two-dimensional image to reduce aliasing artifacts of the two-dimensional image and/or increase sharpness of the two-dimensional image;

wherein the process comprises a noise filter;

and wherein applying the process to the two-dimensional image comprises reducing a noise level of the processed two-dimensional image.

29. The computer program product of claim 20, wherein the computer program code is further configured to cause the processor to perform the step of:

based on the parameter, determining a second setting of the process;

and wherein applying the process to the two-dimensional image comprises;

applying the process with the first setting to a first pixel of the two-dimensional image; and

applying the process with the second setting to a second pixel of the two-dimensional image.

30. A system for processing a two-dimensional image projected from light-field data, comprising:

a processor configured to;

retrieve a two-dimensional image projected from light-field data;

retrieve at least one parameter associated with the two-dimensional image;

based on the parameter, determine a first setting of a process; and

apply the process with the first setting to the two-dimensional image to generate a processed two-dimensional image.

31. The system of claim 30, wherein the parameter describes the picture being generated from the light-field.

32. The system of claim 30, wherein the parameter is derived from the light-field data.

33. The system of claim 32, wherein the process comprises a non-photorealistic rendering technique selected from the group consisting of:

a magnification process by which an in-focus region of the two-dimensional image is magnified relative to a defocused region of the two-dimensional image;

an artistic simulation process by which the two-dimensional image is modified to simulate a painting with a brush stroke size that is larger in a defocused region of the two-dimensional image than in a focused region of the two-dimensional image;

a stippling filter that uses larger stipples in a defocused region of the two-dimensional image than in a focused region of the two-dimensional image; and

an edge-preserving smoothing filter with a larger radius in a defocused region of the two-dimensional image than in a focused region of the two-dimensional image.

34. The system of claim 30, wherein the process is selected from the group consisting of:

a noise filtering process;

an image sharpening process;

a color adjustment process;

a tone curve process;

a contrast adjustment process;

a saturation adjustment process; and

a gamma adjustment process.

35. The system of claim 30, wherein the parameter is selected from the group consisting of:

a target refocus depth applicable to the entire two-dimensional image;

a measured lambda value at a pixel of the two-dimensional image, wherein the lambda value indicates a distance perpendicular to a microlens array of a light-field capture device used to capture the light-field data;

a difference between the measured lambda and a target refocus depth at a pixel of the two-dimensional image; and

a click-to-focus depth value at a pixel of the two-dimensional image, wherein the click-to-focus depth value comprises a depth selected by a user for interactive refocusing of the two-dimensional image.

36. The system of claim 35, wherein the parameter comprises a measured lambda value at a pixel of the two-dimensional image, wherein the lambda value indicates a distance perpendicular to a microlens array of a light-field capture device used to capture the light-field data, and wherein the process comprises an image gain adjustment process that adjusts brightness of the pixel based on the lambda value.

37. The system of claim 35, wherein the parameter comprises a measured lambda value at a pixel of the two-dimensional image, wherein the lambda value indicates a distance perpendicular to a microlens array of a light-field capture device used to capture the light-field data, wherein the processor is further configured to:

use the parameter to determine a degree of high-frequency detail in the two-dimensional image;

wherein the first setting is determined based the degree of high-frequency detail;

and wherein using the parameter to determine a degree of high-frequency detail in the two-dimensional image comprises;

responsive to the lambda value at the pixel being above a high threshold or below a low threshold, determining that the pixel has relatively little high-frequency detail; and

responsive to the lambda value at the pixel being above the low threshold and below the high threshold, determining that the pixel has a relatively higher amount of high-frequency detail.

38. The system of claim 30, wherein processor is further configured to:

apply a reconstruction filter to the two-dimensional image to reduce aliasing artifacts of the two-dimensional image and/or increase sharpness of the two-dimensional image;

wherein the process comprises a noise filter;

and wherein applying the process to the two-dimensional image comprises reducing a noise level of the processed two-dimensional image.

39. The system of claim 30, wherein the processor is further configured to:

based on the parameter, determine a second setting of the process;

and wherein applying the process to the two-dimensional image comprises;

applying the process with the first setting to a first pixel of the two-dimensional image; and

applying the process with the second setting to a second pixel of the two-dimensional image.