Systems and methods for correcting user identified artifacts in light field images

• US 9,214,013 B2
• Filed: 09/16/2013
• Issued: 12/15/2015
• Est. Priority Date: 09/14/2012
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: Pelican Imaging Corporation
• Inventors: Venkataraman, Kartik, Ciurea, Florian
• Primary Examiner(s): Patel, Kanjibhai
• Application Number:
  US14/028,278
• Publication Number:
  US 20140079336A1
• Time in Patent Office: 820 Days
• Field of Search: 382/254, 382274-275, 382/312
• US Class Current: 1/1
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal

Abstract

Systems and methods for correction of user identified artifacts in light field images are disclosed. One embodiment of the invention is a method for correcting artifacts in a light field image rendered from a light field obtained by capturing a set of images from different viewpoints and initial depth estimates for pixels within the light field using a processor configured by an image processing application, where the method includes: receiving a user input indicating the location of an artifact within said light field image; selecting a region of the light field image containing the indicated artifact; generating updated depth estimates for pixels within the selected region; and re-rendering at least a portion of the light field image using the updated depth estimates for the pixels within the selected region.

Claims

1. A method for correcting artifacts in a light field image rendered from a light field obtained by capturing a set of images from different viewpoints and initial depth estimates for pixels within the light field using a processor configured by an image processing application and comprising:

receiving a user input indicating the location of an artifact within said light field image;

selecting a region of the light field image containing the indicated artifact;

generating updated depth estimates for pixels within the selected region by;

receiving a user input indicating a specific depth; and

determining updated depth estimates for pixels in the selected region of said light field image containing said artifact based upon said specific depth by updating the depth estimate of a given pixel within the selected region containing said artifact with said specific depth when the given pixel is associated with a confidence value that satisfies a threshold;

and

re-rendering at least a portion of the light field image using the updated depth estimates for the pixels within the selected region.

2. The method of claim 1, wherein the rendering of the light field image utilizes initial visibility information with respect to pixels in the light field and the method further comprises:

generating updated visibility information for pixels within the selected region;

wherein re-rendering at least a portion of the light field image further comprises using the updated visibility information for the pixels within the selected region.

3. The method of claim 2, further comprising:

receiving a user input validating the re-rendering of the at least a portion of the light field image; and

updating the light field image, the initial depth estimates, and the initial visibility information.

4. The method of claim 1, wherein the set of images includes images captured in a plurality of color channels.

5. The method of claim 1, wherein the light field image is rendered from the light field and the depth estimates using a super-resolution process.

6. The method of claim 1, wherein receiving a user input indicating the location of an artifact within said light field image comprises receiving a user selection of at least one pixel in said light field image.

7. The method of claim 6, wherein selecting a region of the light field image containing the indicated artifact further comprises identifying additional pixels within the neighborhood of the at least one selected pixel in said light field image.

8. The method of claim 7, wherein identifying additional pixels further comprises identifying pixels within a predetermined neighborhood surrounding each of the at least one selected pixel in said light field image.

9. The method of claim 7, wherein identifying additional pixels further comprises identifying pixels within a neighborhood surrounding each of the at least one selected pixel in said light field image that adapts based upon the local characteristics of the pixels in the neighborhood of the at least one selected pixel.

10. The method of claim 1, wherein the rendering of the light field image utilizes initial visibility information with respect to pixels in the light field and the method further comprises:

generating updated visibility information for pixels within the selected region by;

re-rendering a portion of said light field image multiple times by applying different visibility information to the light field; and

determining updated visibility information for pixels in the selected region of said light field image containing said artifact based upon receipt of a user input selecting one of said multiple renderings of said portion of said light field image.

11. The method of claim 1, wherein the rendering of the light field image utilizes initial visibility information with respect to pixels in the light field and the method further comprises:

generating updated depth estimates and visibility information for pixels within the selected region by;

searching a plurality of combinations of depth and visibilities for pixels within the selected region of said light field image containing said artifact; and

selecting one of said plurality of combinations of depth and visibilities based upon the selection resulting in the best overall match between the pixels in the selected region of said light field image and corresponding pixels within the light field, where the corresponding pixels within the light field are identified based upon the selected depth and visibilities.

12. The method of claim 1, wherein re-rendering at least a portion of the light field image using the updated depth estimates for the pixels within the selected region further comprises using a super-resolution process to re-render said at least a portion of the light field image.

13. The method of claim 1, wherein re-rendering at least a portion of the light field image using the updated depth estimates for the pixels within the selected region further comprises shifting pixels within the selected region along epipolar lines based upon the updated depth estimates.

14. The method of claim 1, wherein:

receiving a user input indicating a specific depth comprises;

receiving a user input identifying of a region of said light field image that does not contain said artifact; and

indicating the specific depth by averaging the initial depths of pixels within the identified region that does not contain said artifact.

15. A method for correcting artifacts in a light field image rendered from a light field obtained by capturing a set of images from different viewpoints and initial depth estimates for pixels within the light field using a processor configured by an image processing application, the initial depth estimates including confidence metrics describing the reliability of the initial depth estimates, the method comprising:

receiving a user input indicating the location of an artifact within said light field image, the user input selecting at least one pixel in said light field image;

selecting a region of the light field image containing the indicated artifact;

identifying additional pixels within the neighborhood of the at least one selected pixel in said light field image, the additional pixels having depth estimates with associated confidence metrics, the additional pixels'"'"' confidence metrics satisfying a threshold;

generating updated depth estimates for pixels within the selected region; and

re-rendering at least a portion of the light field image using the updated depth estimates for the pixels within the selected region.

16. The method of claim 15, wherein generating updated depth estimates for pixels within the selected region further comprises:

receiving a user input identifying of a region of said light field image that does not contain said artifact; and

determining an updated depth estimate based upon the initial depth estimates of pixels within the identified region of said light field image.

17. The method of claim 16, wherein receiving a user input identifying a region of said light field image that does not contain said artifact further comprises receiving a user selection of at least one pixel in said light field image.

18. The method of claim 17, wherein receiving a user input identifying a region of said light field image that does not contain said artifact further comprises identifying additional pixels within the neighborhood of the at least one selected pixel in said light field image.

19. The method of claim 18, wherein identifying additional pixels further comprises identifying pixels within a predetermined neighborhood surrounding each of the at least one selected pixel in said light field image.

20. The method of claim 18, wherein identifying additional pixels further comprises identifying pixels within a neighborhood surrounding each of the at least one selected pixel in said light field image that adapts based upon the local characteristics of the pixels in the neighborhood of the at least one selected pixel.

21. The method of claim 15, wherein generating updated depth estimates further comprises:

iteratively re-rendering a portion of said light field image containing said artifact in response to received user inputs indicating a plurality of potential depths; and

determining updated depth estimates for pixels in the selected region of said light field image containing said artifact based upon receipt of a user input validating one of said plurality of potential depths.

22. A method for correcting artifacts in a light field image rendered from a light field obtained by capturing a set of images from different viewpoints and initial depth estimates for pixels within the light field using a processor configured by an image processing application and comprising:

receiving a user input indicating the location of an artifact within said light field image;

selecting a region of the light field image containing the indicated artifact;

generating updated depth estimates for pixels within the selected region by;

receiving a user input identifying of a region of said light field image that does not contain said artifact;

determining an updated depth estimate based upon the initial depth estimates of pixels within the identified region of said light field image that does not contain said artifact by averaging the initial depths of pixels within the identified region that does not contain said artifact; and

updating the depth estimates of pixels within the selected region containing said artifact;

and

re-rendering at least a portion of the light field image using the updated depth estimates for the pixels within the selected region.

23. The method of claim 22, wherein:

the initial depth estimates include confidence metrics describing the reliability of the initial depth estimates; and

averaging the initial depths of pixels within the identified region further comprises filtering pixels from the averaging process that have depth estimates with confidence metrics that satisfy a threshold level of reliability.

24. The method of claim 22, wherein:

the confidence metrics comprise a set of confidence values associated with the depth information assigned to at least one pixel within the light field image; and

updating the depth estimates of pixels within the selected region containing said artifact further comprises;

comparing the confidence value associated with a given pixel within the selected region containing said artifact to a second confidence value associated with at least one pixel in the identified region of the light field image that does not contain said artifact; and

when the difference between the first and second confidence values satisfies a threshold, updating the depth estimate of the given pixel with said average depth estimate.

25. The method of claim 22, wherein averaging the initial depths of pixels within the identified region further comprises filtering pixels from the averaging process that are outliers relative to the other pixels within the identified region that does not contain said artifact.

26. An image processing system, comprising:

a processor; and

memory containing an image processing application;

wherein the image processing application configures the processor to;

receive a user input indicating the location of an artifact within a light field image rendered from a light field obtained by capturing a set of images from different viewpoints and initial depth estimates for pixels within the light field;

select a region of the light field image containing an indicated artifact;

generate updated depth estimates for pixels within a selected region by;

receiving a user input indicating a specific depth; and

determining updated depth estimates for pixels in the selected region of said light field image containing said artifact based upon said specific depth by updating the depth estimate of a given pixel within the selected region containing said artifact with said specific depth when the given pixel is associated with a confidence value that satisfies a threshold; and

re-render at least a portion of a light field image using the updated depth estimates for the pixels within the selected region.

27. The image processing system of claim 26, further comprising:

an array camera module;

wherein the image processing application configures the processor to;

capture a set of images from different viewpoints;

generate initial depth estimates based upon the disparity between corresponding pixels in a captured set of images;

render a light field image using a super-resolution process based upon a set of images and initial depth estimates; and

store a rendered light field image in said memory.

28. The image processing system of claim 26, wherein:

receiving a user input indicating a specific depth comprises;

receiving a user input identifying of a region of said light field image that does not contain said artifact; and

indicating the specific depth by averaging the initial depths of pixels within the identified region that does not contain said artifact.