IMAGE BLUR BASED ON 3D DEPTH INFORMATION

• US 20140009585A1
• Filed: 03/14/2013
• Published: 01/09/2014
• Est. Priority Date: 07/03/2012
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: CAMPBELL, Scott Patrick, ADSUMILLI, Balineedu Chowdary
• Granted Patent: US 9,185,387 B2
• Time in Patent Office: Days
• US Class Current: 348/47
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10148 Varying focus
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/571 from focus
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0088 Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image
• CPC Class Codes: H04N 2013/0096 Synchronisation or controlling aspects
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

Blurring is simulated in post-processing for captured images. A 3D image is received from a 3D camera, and depth information in the 3D image is used to determine the relative distances of objects in the image. One object is chosen as the subject of the image, and an additional object in the image is identified. Image blur is applied to the identified additional object based on the distance between the 3D camera and the subject object, the distance between the subject object and the additional object, and a virtual focal length and virtual f-number.

Claims

1. A method for applying image blur based on depth information, comprising:

receiving a 3D image captured by a 3D camera, the 3D image comprising a plurality of objects and depth information for each of the plurality of objects;

identifying one of the plurality of objects as a subject object;

determining a first distance between the 3D camera and the subject object based on the depth information associated with the subject object;

determining a second distance between the subject object and a first additional object of the plurality of objects based on the depth information associated with the first additional object and the depth information associated with the subject object;

receiving a virtual f-number and a virtual focal length;

calculating a first blur factor for the first additional object, the first blur factor based on the first distance, the second distance, the virtual f-number, and the virtual focal length; and

applying a first image blur to the first additional object based on the first blur factor.

2. The method of claim 1, wherein the 3D image comprises a first image and a second image, the first image captured by a first camera and the second image captured by a second camera, two cameras separated by a separation distance and pointed in the same direction.

3. The method of claim 2, wherein the depth information for each of the plurality of objects in the 3D image comprises an offset between the object'"'"'s position in the first image and the object'"'"'s position in the second image.

4. The method of claim 1, wherein the depth information for each of the plurality of objects comprises parallax information for each of the plurality of objects.

5. The method of claim 1, wherein identifying the first object of the 3D image as the subject object comprises receiving a user input identifying the first object as the subject.

6. The method of claim 1, wherein applying the first image blur to the first additional object comprises:

generating an image layer comprising an object region associated with the first additional object and a transparent region;

generating a blur spot, a size of the blur spot based on the first blur factor;

convolving the blur spot with the image layer, the convolution operation generating a blurred image layer, the blurred image layer comprising a blurred object region with a blurred first additional object and a transparent region; and

applying a transparency function to a boundary between the blurred object region and the transparent region.

7. The method of claim 1, further comprising:

determining a third distance between the subject object and a second additional object of the plurality of objects based on the depth information associated with the second additional object and the depth information associated with the subject object;

calculating a second blur factor for the second additional object, the second blur factor based on the first distance, the third distance, the virtual f-number, and the virtual focal length; and

applying a second image blur to the second additional object based on the value of the second blur factor.

8. The method of claim 7, wherein the first additional object is closer in distance to the 3D camera than the subject object, and wherein the subject object is closer in distance to the 3D camera than the second additional object.

9. A system for applying an image blur based on depth information, comprising:

a non-transitory computer-readable medium storing computer program instructions and configured to be executed by a processor, the instructions for;

receiving a 3D image from a 3D camera, the 3D image comprising a plurality of objects and depth information associated with each object;

identifying one of the plurality of objects as a subject object;

determining a first distance between the 3D camera and the subject object based on the depth information associated with the subject object;

determining a second distance between the subject object and a first additional object of the plurality of objects based on the depth information associated with the first additional object and the depth information associated with the subject object;

receiving a virtual f-number and a virtual focal length;

calculating a first blur factor for the first additional object, the first blur factor based on the first distance, the second distance, the virtual f-number, and the virtual focal length; and

applying a first image blur to the first additional object based on the first blur factor; and

a processor configured to execute the computer program instructions.

10. The system of claim 9, wherein the 3D camera comprises a first camera and a second camera, the two cameras separated by a separation distance and pointed in the same direction, wherein each of the two cameras comprises an image sensor.

11. The system of claim 10, wherein the 3D depth information for each of the plurality of objects in the 3D image comprises an offset between the object'"'"'s position relative to the image sensor of the first camera and the object'"'"'s position relative to the image sensor of the second camera.

12. The system of claim 9, wherein identifying the first object of the 3D image as the subject object comprises receiving a user input identifying the first object.

13. The system of claim 9, wherein applying the first image blur to the first additional object comprises:

generating an image layer comprising an object region associated with the first additional object and a transparent region;

generating a blur spot, a size of the blur spot based on the first blur factor;

convolving the blur spot with the image layer, the convolution operation generating a blurred image layer, the blurred image layer comprising a blurred object region with a blurred first additional object and a transparent region; and

applying a transparency function to a boundary between the blurred object region and the transparent region.

14. The system of claim 9, wherein the computer-readable medium further comprises instructions for:

determining a third distance between the subject object and a second additional object of the plurality of objects based on the depth information associated with the second additional object and the depth information associated with the subject object;

calculating a second blur factor for the second additional object, the second blur factor based on the first distance, the third distance, the virtual f-number, and the virtual focal length; and

applying a second image blur to the second additional object based on the second blur factor.

15. The system of claim 14, wherein the first additional object is closer in distance to the 3D camera than the subject object, and wherein the subject object is closer in distance to the 3D camera than the second additional object.

16. A non-transitory computer-readable medium storing computer program instructions and configured to be executed by a processor, the instructions for:

receiving a 3D image captured by a 3D camera, the 3D image comprising a plurality of objects and depth information for each of the plurality of objects;

identifying one of the plurality of objects as a subject object;

determining a first distance between the 3D camera and the subject object based on the depth information associated with the subject object;

determining a second distance between the subject object and a first additional object of the plurality of objects based on the depth information associated with the first additional object and the depth information associated with the subject object;

receiving a virtual f-number and a virtual focal length;

calculating a first blur factor for the first additional object, the first blur factor based on the first distance, the second distance, the virtual f-number, and the virtual focal length; and

applying a first image blur to the first additional object based on the first blur factor.

17. The computer-readable medium of claim 16, wherein the 3D image comprises a first image and a second image, the first image captured by a first camera and the second image captured by a second camera, two cameras separated by a separation distance and pointed in the same direction.

18. The computer-readable medium of claim 17, wherein the depth information for each of the plurality of objects in the 3D image comprises an offset between the object'"'"'s position in the first image and the object'"'"'s position in the second image.

19. The computer-readable medium of claim 16, wherein identifying the first object of the 3D image as the subject object comprises receiving a user input identifying the first object as the subject.

20. The computer-readable medium of claim 16, wherein applying the first image blur to the first additional object comprises:

generating an image layer comprising an object region associated with the first additional object and a transparent region;

generating a blur spot, a size of the blur spot based on the first blur factor;

convolving the blur spot with the image layer, the convolution operation generating a blurred image layer, the blurred image layer comprising a blurred object region with a blurred first additional object and a transparent region; and

applying a transparency function to a boundary between the blurred object region and the transparent region.

21. The computer-readable medium of claim 16, further comprising instructions for:

determining a third distance between the subject object and a second additional object of the plurality of objects based on the depth information associated with the second additional object and the depth information associated with the subject object;

calculating a second blur factor for the second additional object, the second blur factor based on the first distance, the third distance, the virtual f-number, and the virtual focal length; and

applying a second image blur to the second additional object based on the value of the second blur factor.

22. The computer-readable medium of claim 21, wherein the first additional object is closer in distance to the 3D camera than the subject object, and wherein the subject object is closer in distance to the 3D camera than the second additional object.

23. A method for applying image blur based on depth information, comprising:

receiving an image taken by a camera, the image comprising a plurality of objects;

identifying one of the plurality of objects as a subject object;

determining a first distance between the camera and the subject object;

determining a second distance between the subject object and a first additional object of the plurality of objects;

receiving an f-number and a focal length; and

applying a first image blur to the first additional object based on the first distance, the second distance, the f-number, and the focal length.