Method and system for rendering simulated depth-of-field visual effect

• US 9,418,400 B2
• Filed: 06/18/2013
• Issued: 08/16/2016
• Est. Priority Date: 06/18/2013
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Gruen, Holger, Sakharnykh, Nikolay
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US13/921,161
• Publication Number:
  US 20140368494A1
• Time in Patent Office: 1,155 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2200/21 involving computational photography
• CPC Class Codes: G06T 2207/20056 Discrete and fast Fourier transform, [DFT, FFT]
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/10 using non-spatial domain filtering
• CPC Class Codes: G06T 5/75 Unsharp masking

Abstract

Systems and methods for rendering depth-of-field visual effect on images with high computing efficiency and performance. A diffusion blurring process and a Fast Fourier Transform (FFT)-based convolution are combined to achieve high-fidelity depth-of-field visual effect with Bokeh spots in real-time applications. The brightest regions in the background of an original image are enhanced with Bokeh effect by virtue of FFT convolution with a convolution kernel. A diffusion solver can be used to blur the background of the original image. By blending the Bokeh spots with the image with gradually blurred background, a resultant image can present an enhanced depth-of-field visual effect. The FFT-based convolution can be computed with multi-threaded parallelism.

Claims

1. A computer implemented method of rendering an image, the method comprising:

accessing a first image represented in a space domain, said first image comprising a first section and a second section;

applying a blur operator on said second section to generate a second image;

identifying a plurality of target regions from said second section in said first image;

converting said plurality of target regions from said space domain to a frequency domain by performing a convolution thereon;

restoring said plurality of target regions in said first image from said frequency domain to said space domain to produce a third image; and

blending said second image and said third image to produce a resultant image.

2. The computer implemented method of claim 1, wherein said converting and said performing a convolution comprises performing a Fast Fourier Transform (FFT) convolution on a convolution kernel with said plurality of target regions, and wherein said convolution kernel represents a hexagon camera aperture.

3. The computer implemented method of claim 2, wherein a size of said convolution kernel corresponds to a circle of confusion (CoC) at an infinite distance.

4. The computer implemented method of claim 2 further comprising configuring a plurality of execution threads to perform said FFT convolution in parallel.

5. The computer implemented method of claim 1, wherein said identifying said plurality of target regions comprises:

down-sampling said first image to a downsized image;

filtering said downsized image with a bright pass filter based on a configurable luminance threshold.

6. The computer implemented method of claim 1, wherein said first section corresponds to an in-focus subject represented in said first image, and wherein further said second section corresponds to a background represented in said first image.

7. The computer implemented method of claim 1, wherein said blur operator comprises a diffusion depth-of-field solver, and wherein further said resultant image comprises Bokeh effect components.

8. A system comprising

a processor;

a memory coupled to said processor and storing an image processing program, said image processing program comprising instructions that cause said processor to perform a method of generating an image, said method comprising;

accessing a first image;

identify a first portion and a second portion of said first image;

selecting a plurality of regions from said second portion of said first image, said plurality of regions comprising luminous regions on said first image;

performing a Fast Fourier Transform (FFT) convolution on said plurality of regions with a convolution kernel representing a geometric shape to produce an intermediate image;

performing an inverse FFT convolution on said intermediate image to produce a second image;

applying a blur operator on said second portion of said first image and preserving said first portion to produce a third image; and

combining said second image and said third image to produce a resultant image.

9. The system of claim 8, wherein said blur operator comprises a diffusion depth-of-field (DOF) solver, and wherein said resultant image presents a Bokeh effect.

10. The system of claim 8, wherein said Fast Fourier Transform (FFT) convolution converts said plurality of regions from a space domain to a frequency domain, and wherein said convolution kernel represents a camera aperture shape with a size equal to a circle of confusion (CoC) at an infinite distance of said first image.

11. The system of claim 8, wherein said selecting said plurality of regions comprises:

down-sampling said second portion of said first image to a reduced-resolution image, and applying a bright pass filter on said reduced-resolution image based on a luminance threshold.

12. The system of claim 8, wherein said method further comprises configuring a plurality of execution threads to perform said FFT convolution and said inverse FFT convolution respectively in parallel.

13. The system of claim 8, wherein said first portion corresponds to an in-focus subject, and wherein said second portion corresponds to a background.

14. A non-transitory computer-readable storage medium comprising instructions, when executed by a processor, cause the processor to implement a method of generating an image, said method comprising:

accessing a first image;

identify a first portion and a second portion of said first image;

selecting a plurality of regions from said second portion of said first image, said plurality of regions comprising luminous regions on said first image;

performing a Fast Fourier Transform (FFT) convolution on said plurality of regions with a convolution kernel representing a geometric shape to produce an intermediate image;

performing an inverse FFT convolution on said intermediate image to produce a second image;

applying a blur operator on said second portion of said first image and preserving said first portion to produce a third image; and

combining said second image and said third image to produce a resultant image.

15. The non-transitory computer-readable storage medium of claim 14, wherein said blur operator comprises a diffusion depth-of-field (DOF) solver, and wherein said resultant image presents a Bokeh effect.

16. The non-transitory computer-readable storage medium of claim 14, wherein said Fast Fourier Transform (FFT) convolution converts said plurality of regions from a space domain to a frequency domain.

17. The non-transitory computer-readable storage medium of claim 14, wherein said selecting said plurality of regions comprises:

down-sampling said second portion of said first image to a reduced-resolution image, and applying a bright pass filter on said reduced-resolution image based on a luminance threshold.

18. The non-transitory computer-readable storage medium of claim 14, wherein said method further comprises configuring a plurality of execution threads to perform said FFT convolution and said inverse FFT convolution respectively in parallel.

19. The non-transitory computer-readable storage medium of claim 14, wherein said first portion corresponds to an in-focus subject represented in said first image, and wherein said second portion corresponds to a background represented in said first image.

20. The non-transitory computer-readable storage medium of claim 14, wherein said convolution kernel represents a camera aperture shape with a size equal to a circle of confusion (CoC) at an infinite distance of said first image.