Depth-based image blurring

• US 10,552,947 B2
• Filed: 11/28/2017
• Issued: 02/04/2020
• Est. Priority Date: 06/26/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Liang, Chia-Kai, Oberheu, Kent, Akeley, Kurt, Girod, Garrett, Karnad, Nikhil, Benevides, Francis A.
• Primary Examiner(s): Mehta, Bhavesh M
• Assistant Examiner(s): Lemieux, Ian L
• Application Number:
  US15/824,574
• Publication Number:
  US 20180082405A1
• Time in Patent Office: 798 Days
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation

Abstract

An image such as a light-field image may be processed to provide depth-based blurring. The image may be received in a data store. At an input device, first and second user input may be received to designate a first focus depth and a second focus depth different from the first focus depth, respectively. A processor may identify one or more foreground portions of the image that have one or more foreground portion depths, each of which is less than the first focus depth. The processor may also identify one or more background portions of the image that have one or more background portion depths, each of which is greater than the second focus depth. The processor may also apply blurring to the one or more foreground portions and the one or more background portions to generate a processed image, which may be displayed on a display device.

Claims

1. A method for processing an image, the method comprising:

in a data store, receiving the image;

on a display device, displaying a range bar comprising a first slider and a second slider;

in an input device, receiving a first user input that designates a first focus depth by detecting motion selecting to move the first slider along the range bar to a first location corresponding to the first focus depth;

in the input device, receiving a second user input that designates a second focus depth different from the first focus depth by detecting motion selecting to move the second slider along the range bar to a second location corresponding to the second focus depth;

in a processor, identifying one or more foreground portions of the image, wherein the one or more foreground portions have one or more foreground portion depths, each of which is less than the first focus depth;

in the processor, identifying one or more background portions of the image, wherein the one or more background portions have one or more background portion depths, each of which is greater than the second focus depth;

in the processor, applying blurring to the one or more foreground portions and to the one or more background portions to generate a processed image; and

on the display device, displaying the processed image.

2. The method of claim 1, further comprising, in the data store, receiving a depth map indicative of a plurality of depths at which a plurality of objects were disposed, relative to a camera used to capture the image, during capture of the image;

wherein identifying the one or more foreground portions comprises determining, based on the depth map, that the one or more foreground portions have the one or more foreground portion depths;

and wherein identifying the one or more background portions comprises determining, based on the depth map, that the one or more background portions have the one or more background portion depths.

3. The method of claim 2, wherein the image comprises a light-field image captured with a light-field image capture device.

4. The method of claim 1, further comprising, on the display device:

displaying a graphical user interface comprising the first slider and the second slider;

in response to receipt of the first user input, displaying motion of the first slider to the first location; and

in response to receipt of the second user input, displaying motion of the second slider to the second location.

5. The method of claim 1, wherein the first focus depth is less than the second focus depth such that one or more intermediate portions of the image have one or more intermediate portion depths that are greater than the first focus depth and less than the second focus depth;

and wherein generating the processed image comprises rendering the one or more intermediate portions of the image without blurring.

6. The method of claim 1, wherein the first focus depth is greater than the second focus depth such one or more of the one or more foreground portions of the image are also included in the one or more background portions of the image;

and wherein generating the processed image comprises applying blurring to an entirety of the image.

7. The method of claim 1, wherein applying the blurring to the one or more foreground portions and to the one or more background portions comprises applying a noncircular blur shape to the one or more foreground portions and to the one or more background portions.

8. The method of claim 1, wherein applying the blurring to the one or more foreground portions and to the one or more background portions comprises:

applying a first noncircular blur shape to one or more first subsets of the one or more foreground portions and the one or more background portions that are at a first depth; and

applying a second noncircular blur shape to one or more second subsets of the one or more foreground portions and the one or more background portions that are at a second depth different from the first depth.

9. The method of claim 1, wherein applying the blurring to the one or more foreground portions and to the one or more background portions comprises applying the blurring to each pixel of a plurality of pixels of the image based, at least partially, on at least one of an X coordinate of the pixel, and a Y coordinate of the pixel.

10. The method of claim 9, further comprising, in the input device, prior to applying the blurring to the one or more foreground portions and to the one or more background portions, receiving a third user input that designates a tilt angle at which a focus plane of the image is to be tilted;

wherein applying the blurring to the one or more foreground portions and to the one or more background portions further comprises applying the blurring to portions of the image that are displaced from the focus plane.

11. The method of claim 10, further comprising, in the input device, prior to applying the blurring to the one or more foreground portions and to the one or more background portions, receiving a fourth user input that designates a rotation angle at which the focus plane is to be rotated.

12. A non-transitory computer-readable medium for processing an image, comprising instructions stored thereon, that when executed by a processor, perform the steps of:

causing a data store to receive the image;

causing a display device to display a range bar comprising a first slider and a second slider;

causing an input device to receive a first user input that designates a first focus depth by detecting motion selecting to move the first slider along the range bar to a first location corresponding to the first focus depth;

causing the input device to receive a second user input that designates a second focus depth different from the first focus depth by detecting motion selecting to move the second slider along the range bar to a second location corresponding to the second focus depth;

identifying one or more foreground portions of the image, wherein the one or more foreground portions have one or more foreground portion depths, each of which is less than the first focus depth;

identifying one or more background portions of the image, wherein the one or more background portions have one or more background portion depths, each of which is greater than the second focus depth;

applying blurring to the one or more foreground portions and to the one or more background portions to generate a processed image; and

causing the display device to display the processed image.

13. The non-transitory computer-readable medium of claim 12, further comprising instructions stored thereon, that when executed by a processor, cause the data store to receive a depth map indicative of a plurality of depths at which a plurality of objects were disposed, relative to a camera used to capture the image, during capture of the image;

wherein identifying the one or more foreground portions comprises determining, based on the depth map, that the one or more foreground portions have the one or more foreground portion depths;

and wherein identifying the one or more background portions comprises determining, based on the depth map, that the one or more background portions have the one or more background portion depths.

14. The non-transitory computer-readable medium of claim 12, further comprising instructions stored thereon, that when executed by a processor, perform the steps of:

causing the display device, in response to receipt of the first user input, to display motion of the first slider to a first location corresponding to the first focus depth; and

causing the display device, in response to receipt of the second user input, to display motion of the second slider to a second location corresponding to the second focus depth;

wherein the input device comprises a pointing device.

15. The non-transitory computer-readable medium of claim 12, wherein the first focus depth is less than the second focus depth such that one or more intermediate portions of the image have one or more intermediate portion depths that are greater than the first focus depth and less than the second focus depth;

and wherein generating the processed image comprises rendering the one or more intermediate portions of the image without blurring.

16. The non-transitory computer-readable medium of claim 12, wherein applying the blurring to the one or more foreground portions and to the one or more background portions comprises applying the blurring to each pixel of a plurality of pixels of the image based, at least partially, on at least one of an X coordinate of the pixel, and a Y coordinate of the pixel.

17. The non-transitory computer-readable medium of claim 16, further comprising instructions stored thereon, that when executed by a processor, cause the input device, prior to applying the blurring to the one or more foreground portions and to the one or more background portions, to receive a third user input that designates a tilt angle at which a focus plane of the image is to be tilted;

wherein applying the blurring to the one or more foreground portions and to the one or more background portions further comprises applying the blurring to portions of the image that are displaced from the focus plane.

18. A system for processing an image, the system comprising:

a data store configured to receive the image;

a display device, communicatively coupled to a processor, configured to display a range bar comprising a first slider on the range bar and a second slider on the range bar;

an input device configured to;

receive a first user input that designates a first focus depth by detecting motion selecting to move the first slider along the range bar to a first location corresponding to the first focus depth; and

receive a second user input that designates a second focus depth different from the first focus depth by detecting motion selecting to move the second slider along the range bar to a second location;

wherein the processor is configured to;

identify one or more foreground portions of the image, wherein the one or more foreground portions have one or more foreground portion depths, each of which is less than the first focus depth;

identify one or more background portions of the image, wherein the one or more background portions have one or more background portion depths, each of which is greater than the second focus depth; and

apply blurring to the one or more foreground portions and to the one or more background portions to generate a processed image; and

wherein the display device is further configured to display the processed image.

19. The system of claim 18, wherein the data store is further configured to receive a depth map indicative of a plurality of depths at which a plurality of objects were disposed, relative to a camera used to capture the image, during capture of the image;

wherein the processor is further configured to identify the one or more foreground portions by determining, based on the depth map, that the one or more foreground portions have the one or more foreground portion depths;

and wherein the processor is further configured to identify the one or more background portions by determining, based on the depth map, that the one or more background portions have the one or more background portion depths.

20. The system of claim 18, wherein the display device is further configured to:

in response to receipt of the first user input, display motion of the first slider to the first location; and

in response to receipt of the second user input, display motion of the second slider to the second location;

wherein the input device comprises a pointing device.

21. The system of claim 18, wherein the first focus depth is less than the second focus depth such that one or more intermediate portions of the image have one or more intermediate portion depths that are greater than the first focus depth and less than the second focus depth;

and wherein the processor is further configured to generate the processed image by rendering the one or more intermediate portions of the image without blurring.

22. The system of claim 18, wherein the processor is further configured to apply the blurring to the one or more foreground portions and to the one or more background portions by applying the blurring to each pixel of a plurality of pixels of the image based, at least partially, on at least one of an X coordinate of the pixel, and a Y coordinate of the pixel.

23. The system of claim 22, wherein the input device is further configured, prior to application of the blurring to the one or more foreground portions and to the one or more background portions, to receive a third user input that designates a tilt angle at which a focus plane of the image is to be tilted;

wherein the processor is further configured to apply the blurring to the one or more foreground portions and to the one or more background portions by applying the blurring to portions of the image that are displaced from the focus plane.

24. A method for processing an image, the method comprising:

in a data store, receiving the image;

in an input device, receiving a first user input that designates a first focus depth;

in the input device, receiving a second user input that designates a second focus depth different from the first focus depth;

in a processor, identifying one or more foreground portions of the image, wherein the one or more foreground portions have one or more foreground portion depths, each of which is less than the first focus depth;

in the processor, identifying one or more background portions of the image, wherein the one or more background portions have one or more background portion depths, each of which is greater than the second focus depth, wherein the first focus depth is greater than the second focus depth such one or more of the one or more foreground portions of the image are also included in the one or more background portions of the image;

in the processor, applying blurring to an entirety of the image to generate a processed image; and

on a display device, displaying the processed image.

25. The method of claim 24, further comprising, on the display device:

displaying a graphical user interface comprising a first element and a second element;

in response to receipt of the first user input, displaying motion of the first element to a first location corresponding to the first focus depth; and

in response to receipt of the second user input, displaying motion of the second element to a second location corresponding to the second focus depth.

26. The method of claim 25, wherein displaying the graphical user interface comprises displaying a range bar;

wherein the first element comprises a first slider on the range bar;

wherein the second element comprises a second slider on the range bar;

wherein the input device comprises a pointing device;

wherein receiving the first user input comprises, with the pointing device, detecting motion selecting to move the first slider along the range bar to the first location;

and wherein receiving the second user input comprises, with the pointing device, detecting motion selecting to move the second slider along the range bar to the second location.

27. A method for processing an image, the method comprising:

in a data store, receiving the image;

in an input device, receiving a first user input that designates a first focus depth;

in the input device, receiving a second user input that designates a second focus depth different from the first focus depth;

in a processor, identifying one or more foreground portions of the image, wherein the one or more foreground portions have one or more foreground portion depths, each of which is less than the first focus depth;

in the processor, identifying one or more background portions of the image, wherein the one or more background portions have one or more background portion depths, each of which is greater than the second focus depth;

in the processor, generating a processed image by applying a first noncircular blur shape to one or more first subsets of the one or more foreground portions and the one or more background portions that are at a first depth and applying a second noncircular blur shape to one or more second subsets of the one or more foreground portions and the one or more background portions that are at a second depth different from the first depth; and

on a display device, displaying the processed image.

28. The method of claim 27, further comprising, in the data store, receiving a depth map indicative of a plurality of depths at which a plurality of objects were disposed, relative to a camera used to capture the image, during capture of the image;

wherein identifying the one or more foreground portions comprises determining, based on the depth map, that the one or more foreground portions have the one or more foreground portion depths;

and wherein identifying the one or more background portions comprises determining, based on the depth map, that the one or more background portions have the one or more background portion depths.

29. The method of claim 27, wherein the first focus depth is less than the second focus depth such that one or more intermediate portions of the image have one or more intermediate portion depths that are greater than the first focus depth and less than the second focus depth;

and wherein generating the processed image comprises rendering the one or more intermediate portions of the image without blurring.

30. The method of claim 27, further comprising, on the display device:

displaying a graphical user interface comprising a first element and a second element;

in response to receipt of the first user input, displaying motion of the first element to a first location corresponding to the first focus depth; and

in response to receipt of the second user input, displaying motion of the second element to a second location corresponding to the second focus depth;

wherein the first element comprises a first slider on the range bar;

wherein the second element comprises a second slider on the range bar;

wherein the input device comprises a pointing device;

wherein receiving the first user input comprises, with the pointing device, detecting motion selecting to move the first slider along the range bar to the first location;

and wherein receiving the second user input comprises, with the pointing device, detecting motion selecting to move the second slider along the range bar to the second location.

31. A system for processing an image, the system comprising:

a data store configured to receive the image;

an input device configured to;

receive a first user input that designates a first focus depth; and

receive a second user input that designates a second focus depth different from the first focus depth;

a processor, communicatively coupled to the data store and the input device, configured to;

identify one or more foreground portions of the image, wherein the one or more foreground portions have one or more foreground portion depths, each of which is less than the first focus depth;

identify one or more background portions of the image, wherein the one or more background portions have one or more background portion depths, each of which is greater than the second focus depth; and

generate a processed image by applying a first noncircular blur shape to one or more first subsets of the one or more foreground portions and the one or more background portions that are at a first depth and applying a second noncircular blur shape to one or more second subsets of the one or more foreground portions and the one or more background portions that are at a second depth different from the first depth; and

a display device, communicatively coupled to the processor, configured to display the processed image.