Systems and method for performing depth based image editing

• US 10,311,649 B2
• Filed: 09/01/2017
• Issued: 06/04/2019
• Est. Priority Date: 02/21/2012
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Limited (Adeia Inc.)
• Inventors: McMahon, Andrew Kenneth John, Venkataraman, Kartik, Mullis, Robert
• Primary Examiner(s): Drennan, Barry
• Assistant Examiner(s): Robinson, Terrell M
• Application Number:
  US15/694,700
• Publication Number:
  US 20170365104A1
• Time in Patent Office: 641 Days
• Field of Search: 345419
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/21 involving computational photography

Abstract

Systems and methods for the manipulation of captured light fields and captured light field image data in accordance with embodiments of the invention are disclosed. In one embodiment of the invention, a system for manipulating captured light field image data includes a processor, a display, a user input device, and a memory, wherein a depth map includes depth information for one or more pixels in the image data, and wherein an image manipulation application configures the processor to display a first synthesized image, receive user input data identifying a region within the first synthesized image, determine boundary data for the identified region using the depth map, receive user input data identifying at least one action, and perform the received action using the boundary data and the captured light field image data.

Claims

1. An array camera system for capturing and manipulating captured image data, comprising:

an array camera comprising a plurality of cameras, where each camera includes separate optics, and a plurality of sensor elements, and each camera is configured to independently capture an image of a scene;

a processor;

a display connected to the processor and configured to display images;

a memory connected to the processor;

software connected to the processor and directs the processor to;

capture images;

store the captured images in the memory; and

generate a depth map having depth information associated with the captured images of the scene; and

an image manipulation application within the memory that directs the processor to;

select a collection of pixels within at least one captured image based upon depth information, wherein the collection of pixels is selected by identifying a boundary of the collection of pixels based upon color and intensity values, and the associated depth information, where the boundary is determined by separately clustering each given pixel based upon depth, color, and intensity;

modify the pixels of the selected collection of pixels of the at least one captured image;

copy the modified pixels of the selected collection of pixels; and

paste the modified pixels of the selected collection of pixels into another image.

2. The system of claim 1, wherein the image manipulation application directs the processor to modify at least one of:

brightness, luminosity, color values and transparency values of the pixels of the selected collection of pixels of the at least one captured image.

3. The system of claim 1, wherein the collection of pixels is an object.

4. The system of claim 1, wherein the collection of pixels excludes the pixels of an object.

5. The system of claim of claim 1, wherein the image manipulation application directs the processor to modify a focus of the selected collection of pixels of at least one captured image by:

identifying a focal plane of the selected collection of pixels using a depth of field; and

refocusing the pixels of the selected collection of pixels using the depth of field, the identified focal plane, and blur modification data, where the blur modification data affects bokeh of the selected collection of pixels.

6. The system of claim 5, wherein the depth of field is determined by at least one of:

utilizing the associated depth information of the at least one captured image, setting the depth of field to a default value, and using a user-provided depth of field through a user input device connected to the processor and configured to provide user generated input data.

7. The system of claim 1, wherein the image manipulation application further directs the processor to:

display the captured image using the display; and

identify the selected collection of pixels within the displayed image.

8. The system of claim 1 further comprising a user input device connected to the processor and configured to provide user generated input data, wherein identifying the collection of pixels within the displayed image comprises receiving user input data from the user input device identifying at least one pixel of the selected collection of pixels within the captured image.

9. The system of claim 1 further comprising an eyeball tracking device, wherein the associated depth of the selected collection of pixels is determined based upon where a user is focusing and dynamically adjusted as the user focuses at different parts of the captured image.

10. The system of claim 1, further comprising:

a user input device connected to the processor and configured to generate user input data in response to user input;

wherein the image manipulation application further configures the processor to receive user input data.

11. The system of claim 10, wherein the input device is a touchscreen device.

12. The system of claim 10, wherein the received input data comprises the selected collection of pixels.

13. The system of claim 10, wherein the received input data comprises modification of the pixels of the selected collection of pixels.

14. The system of claim 1, wherein the captured image and associated depth information are stored in memory in an image data file.

15. The system of claim 14, wherein the image data file also includes at least one auxiliary map selected from a group consisting of a confidence map, an edge map, and a missing pixel map.

16. The system of claim 1, wherein the image into which the selected collection of pixels having modified pixels is copied is another image of the scene.

17. A method for manipulating captured image data using an array camera having a processor, comprising:

capturing, using the processor directed by software, a plurality of images of a scene;

generating, using the processor directed by the software, a depth map having depth information associated with the captured images of the scene;

selecting, using the processor directed by an image manipulation application, a collection of pixels within at least one captured image, based upon the depth information, wherein the collection of pixels is selected by identifying a boundary of the collection of pixels based upon color and intensity values, and the associated depth information, where the boundary is determined by separately clustering each given pixel based upon depth, color, and intensity;

modifying, using the processor directed by an image manipulation application, the pixels of the selected collection of pixels of the at least one captured image;

copying, using the processor directed by an image manipulation application, the modified pixels of the selected collection of pixels; and

pasting, using the processor directed by an image manipulation application, the modified pixels of the selected collection of pixels into another image.

18. The method of claim 17 wherein modifying, using the processor directed by an image manipulation application, the pixels of the selected collection of pixels comprises modifying at least one of:

brightness, luminosity, color values or transparency values of the pixels.

19. The method of claim 17, wherein the collection of pixels is an object.

20. The method of claim 17, wherein the collection of pixels excludes an object.

21. The method of claim of claim 17 further comprising:

identifying, using the processor directed by an image manipulation application, a focal plane of the selected collection of pixels using a depth of field; and

refocusing, using the processor directed by an image manipulation application, the pixels of the selected collection of pixels using the depth of field, the identified focal plane, and blur modification data, where the blur modification data affects bokeh of the selected collection of pixels.

22. The method of claim 21, wherein the depth of field is determined by at least one of:

utilizing the associated depth information of the at least one captured image, setting the depth of field to a default value, and using a user-provided depth of field through a user input device connected to the processor and configured to provide user generated input data.

23. The method of claim 17, further comprising:

displaying, using the processor directed by an image manipulation application, the captured image using the display; and

identifying, using the processor directed by an image manipulation application, the selected collection of pixels within the displayed image.

24. The method of claim 23, wherein identifying the collection of pixels within the displayed image comprises receiving user input data from a user input device identifying at least one pixel of the object within the captured image, wherein the user input device is connected to the processor and configured to provide user generated input data.

25. The method of claim 23, wherein the associated depth of the selected collection of pixels is determined, using an eyeball tracking device, based upon where a user is focusing and dynamically adjusted as the user focuses at different parts of the captured image.

26. The method of claim 17, further comprising receiving user input data via a user input device connected to the processor and configured to generate user input data in response to user input.

27. The method of claim 26, wherein the input device is a touchscreen device.

28. The method of claim 26, wherein the received input data comprises a selected collection of pixels.

29. The method of claim 26, wherein the received input data comprises modification of the pixels of the selected collection of pixels.

30. The method of claim 17 further comprising storing the captured image and associated depth information in memory in an image data file.

31. The method of claim 30, wherein the image data file also includes at least one auxiliary map selected from a group consisting of a confidence map, an edge map, and a missing pixel map.

32. The method of claim 17, wherein the other image is an image of the scene.