Systems and method for performing depth based image editing

• US 9,754,422 B2
• Filed: 05/06/2015
• Issued: 09/05/2017
• Est. Priority Date: 02/21/2012
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: Fotonation Cayman Limited (Adeia Inc.)
• Inventors: McMahon, Andrew Kenneth John, Venkataraman, Kartik, Mullis, Robert
• Primary Examiner(s): Crawford, Jacinta M
• Assistant Examiner(s): ROBINSON, TERRELL M
• Application Number:
  US14/705,914
• Publication Number:
  US 20150235476A1
• Time in Patent Office: 853 Days
• Field of Search: 345419
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/21 involving computational photography

Abstract

Systems and methods for the manipulation of captured light fields and captured light field image data in accordance with embodiments of the invention are disclosed. In one embodiment of the invention, a system for manipulating captured light field image data includes a processor, a display, a user input device, and a memory, wherein a depth map includes depth information for one or more pixels in the image data, and wherein an image manipulation application configures the processor to display a first synthesized image, receive user input data identifying a region within the first synthesized image, determine boundary data for the identified region using the depth map, receive user input data identifying at least one action, and perform the received action using the boundary data and the captured light field image data.

Claims

1. A system for manipulating captured light field image data, comprising:

a processor;

a display connected to the processor and configured to display images;

a user input device connected to the processor and configured to generate user input data in response to user input; and

memory connected to the processor and containing an image manipulation application, a synthesized image, and associated depth information; and

wherein the image manipulation application directs the processor to;

display the synthesized image using the display;

receive user input data from the user input device identifying at least one pixel identifying a region within the synthesized image;

identify an object within the synthesized image based upon color and intensity values, and the associated depth information for the identified at least one pixel, where at least one boundary of the object is determined by separately clustering each given pixel based upon depth, color, and intensity;

receive user input data identifying at least one image processing operation; and

perform the image processing operation on the identified object within the synthesized image.

2. The system of claim 1, wherein the image processing operation is performed as a function of the depth information associated with the synthesized image.

3. The system of claim 1, wherein the image processing operation is selected from the group consisting of cutting pixels corresponding to the identified object from the synthesized image, copying pixels corresponding to the identified object, re-coloring pixels corresponding to the identified object within the synthesized image, and scaling the identified object within the synthesized image.

4. The system of claim 1, wherein the identified object is a set of adjacent pixels in a synthesized image related based on corresponding associated depth information.

5. The system of claim 1, wherein the image manipulation application directs the processor to utilize depth information for the identified at least one pixel along with color and intensity values to determine at least one boundary for the identified object.

6. The system of claim 4, wherein the image manipulation application directs the processor to utilize grouping based upon at least one of depth, color and intensity to determine the at least one boundary for the identified object.

7. The system of claim 4, wherein the image manipulation application directs the processor to utilize edge detection based upon at least one of depth, color and intensity to determine the at least one boundary for the identified object.

8. The system of claim 1, wherein the input device is a touchscreen device.

9. The system of claim 1, wherein:

the image processing operation is a refocus action; and

the image manipulation application further configures the processor to perform the refocus action by synthesizing a second image using a synthetic aperture having a focal plane determined based upon the depth of the identified object.

10. The system of claim 9, wherein the image manipulation application directs the processor to:

receive blur modification data from the user interface, where the blur modification data specifies a depth of field for the synthetic aperture; and

synthesize the second image using the focal plane, and the blur modification data.

11. The system of claim 1, wherein the synthesized image and associated depth information are stored in memory in a light field image data file.

12. The system of claim 11, wherein the light field image data file also includes at least one auxiliary map selected from the group consisting of a confidence map, an edge map, and a missing pixel map.

13. A method for manipulating captured light field image data, comprising:

displaying a synthesized image using a display;

receiving user input data from a user input device connected to the processor, where the user input data identifies at least one pixel identifying a region within the synthesized image;

identifying an object within the synthesized image using a processor directed by an image manipulation application based upon color and intensity values, and the depth information for the identified at least one pixel obtained from a depth map associated with the synthesized image, where at least one boundary of the object is determined by separately clustering each given pixel based upon depth, color, and intensity;

receiving user input data identifying at least one image processing operation; and

performing the image processing operation on the identified object within the synthesized image using the processor directed by the image manipulation application.

14. The method of claim 13, wherein the image processing operation is performed as a function of the depth information associated with the synthesized image.

15. The method of claim 13, wherein the image processing operation is selected from the group consisting of cutting pixels corresponding to the identified object from the synthesized image, copying pixels corresponding to the identified object, re-coloring pixels corresponding to the identified object within the synthesized image, and scaling the identified object within the synthesized image.

16. The method of claim 13, wherein the identified object is a set of adjacent pixels in a synthesized image related based on corresponding associated depth information.

17. The method of claim 13, wherein the image processing operation is a refocus action and the method further comprises synthesizing a second image using a synthetic aperture having a focal plane determined based upon the depth of the identified object using the processor directed by the image manipulation application.

18. The method of claim 17, further comprising:

receiving blur modification data from the user interface, where the blur modification data specifies a depth of field for the synthetic aperture; and

synthesizing the second image using the processor directed by the image manipulation application based upon the focal plane, and the blur modification data.