Image blur based on 3D depth information

• US 10,015,469 B2
• Filed: 10/05/2015
• Issued: 07/03/2018
• Est. Priority Date: 07/03/2012
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Campbell, Scott Patrick, Adsumilli, Balineedu Chowdary
• Primary Examiner(s): Haque, Md
• Assistant Examiner(s): Haque, Nazmul
• Application Number:
  US14/875,690
• Publication Number:
  US 20160029004A1
• Time in Patent Office: 1,002 Days
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10148 Varying focus
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/571 from focus
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0088 Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image
• CPC Class Codes: H04N 2013/0096 Synchronisation or controlling aspects
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

Blurring is simulated in post-processing for captured images. A 3D image is received from a 3D camera, and depth information in the 3D image is used to determine the relative distances of objects in the image. One object is chosen as the subject of the image, and an additional object in the image is identified. Image blur is applied to the identified additional object based on the distance between the 3D camera and the subject object, the distance between the subject object and the additional object, and a virtual focal length and virtual f-number.

Claims

1. A method for applying image blur based on depth information, comprising:

receiving a 3D image captured by a 3D camera, the 3D image including capture of an object and including the depth information for the object, the object located at an object distance from the 3D camera when captured within the 3D image;

identifying a focus distance of a focal plane for the 3D image;

identifying a blur distance between the focal plane and the object, the blur distance comprising a difference between the object distance and the focus distance, the object distance determined based on the depth information;

calculating a blur factor for the object based on the blur distance, wherein;

based on the object distance being less than the focus distance, a first blur factor is calculated based on a given value of the blur distance;

based on the object distance being greater than the focus distance, a second blur factor is calculated based on the given value of the blur distance, the first blur factor being greater than the second blur factor; and

generating a blurred image by blurring a portion of the 3D image including the object by the blur factor such that the portion of the 3D image is blurred based on the object distance between the 3D camera and the object when the 3D image was captured.

2. The method of claim 1, wherein the 3D image comprises a first image and a second image, the first image captured by a first camera and the second image captured by a second camera, the first camera and the second camera separated by a separation distance and pointed in a same direction.

3. The method of claim 2, wherein the object distance is determined based on an offset between the object'"'"'s position in the first image and the object'"'"'s position in the second image.

4. The method of claim 1, wherein the object distance comprises parallax information for the object.

5. The method of claim 1, wherein the object is selected by a user from a plurality of objects included in the 3D image.

6. The method of claim 1, wherein the focal plane is selected by a user.

7. The method of claim 1, wherein blurring the portion of the 3D image including the object comprises:

generating an image layer comprising an object region associated with the object and a transparent region;

generating a blur spot, a size of the blur spot based on a size of the object region;

convolving the blur spot with the image layer, the convolution operation generating a blurred image layer, the blurred image layer comprising a blurred object region and the transparent region; and

applying a transparency function to a boundary between the blurred object region and the transparent region.

8. A system for applying image blur based on depth information, comprising:

a non-transitory computer-readable storage medium storing executable computer instructions that, when executed by a processor, perform steps comprising;

receiving a 3D image captured by a 3D camera, the 3D image including capture of an object and including the depth information for the object, the object located at an object distance from the 3D camera when captured within the 3D image;

identifying a focus distance of a focal plane for the 3D image;

identifying a blur distance between the focal plane and the object, the blur distance comprising a difference between the object distance and the focus distance, the object distance determined based on the depth information;

calculating a blur factor for the object based on the blur distance, wherein;

based on the object distance being less than the focus distance, a first blur factor is calculated based on a given value of the blur distance;

based on the object distance being greater than the focus distance, a second blur factor is calculated based on the given value of the blur distance, the first blur factor being greater than the second blur factor; and

generating a blurred image by blurring a portion of the 3D image including the object by the blur factor such that the portion of the 3D image is blurred based on the object distance between the 3D camera and the object when the 3D image was captured; and

a processor configured to execute the instructions.

9. The system of claim 8, wherein the 3D image comprises a first image and a second image, the first image captured by a first camera and the second image captured by a second camera, the first camera and the second camera separated by a separation distance and pointed in a same direction.

10. The system of claim 9, wherein the object distance is determined based on an offset between the object'"'"'s position in the first image and the object'"'"'s position in the second image.

11. The system of claim 8, wherein the object distance comprises parallax information for the object.

12. The system of claim 8, wherein the object is selected by a user from a plurality of objects included in the 3D image.

13. The system of claim 8, wherein the focal plane is selected by a user.

14. The system of claim 8, wherein blurring the portion of the 3D image including the object comprises:

generating an image layer comprising an object region associated with the object and a transparent region;

generating a blur spot, a size of the blur spot based on a size of the object region;

convolving the blur spot with the image layer, the convolution operation generating a blurred image layer, the blurred image layer comprising a blurred object region and the transparent region; and

applying a transparency function to a boundary between the blurred object region and the transparent region.

15. A non-transitory computer-readable storage medium storing executable computer instructions for applying image blur based on depth information, the instructions configured to, when executed by a processor, perform steps comprising:

receiving a 3D image captured by a 3D camera, the 3D image including capture of an object and including the depth information for the object, the object located at an object distance from the 3D camera when captured within the 3D image;

identifying a focus distance of a focal plane for the 3D image;

identifying a blur distance between the focal plane and the object, the blur distance comprising a difference between the object distance and the focus distance, the object distance determined based on the depth information;

calculating a blur factor for the object based on the blur distance, wherein;

based on the object distance being less than the focus distance, a first blur factor is calculated based on a given value of the blur distance;

based on the object distance being greater than the focus distance, a second blur factor is calculated based on the given value of the blur distance, the first blur factor being greater than the second blur factor; and

generating a blurred image by blurring a portion of the 3D image including the object by the blur factor such that the portion of the 3D image is blurred based on the object distance between the 3D camera and the object when the 3D image was captured.

16. The non-transitory computer-readable storage medium of claim 15, wherein the 3D image comprises a first image and a second image, the first image captured by a first camera and the second image captured by a second camera, the first camera and the second camera separated by a separation distance and pointed in a same direction.

17. The non-transitory computer-readable storage medium of claim 16, wherein the object distance is determined based on an offset between the object'"'"'s position in the first image and the object'"'"'s position in the second image.

18. The non-transitory computer-readable storage medium of claim 15, wherein the object distance comprises parallax information for the object.

19. The non-transitory computer-readable storage medium of claim 15, wherein the object is selected by a user from a plurality of objects included in the 3D image.

20. The non-transitory computer-readable storage medium of claim 15, wherein the focal plane is selected by a user.

21. The non-transitory computer-readable storage medium of claim 15, wherein blurring the portion of the 3D image including the object comprises:

generating an image layer comprising an object region associated with the object and a transparent region;

generating a blur spot, a size of the blur spot based on a size of the object region;

convolving the blur spot with the image layer, the convolution operation generating a blurred image layer, the blurred image layer comprising a blurred object region and the transparent region; and

applying a transparency function to a boundary between the blurred object region and the transparent region.