Performing spatiotemporal filtering

• US 10,600,167 B2
• Filed: 01/18/2018
• Issued: 03/24/2020
• Est. Priority Date: 01/18/2017
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Schied, Christoph H., Salvi, Marco, Kaplanyan, Anton S., Lefohn, Aaron Eliot, Burgess, John Matthew, Patney, Anjul, Wyman, Christopher Ryan
• Primary Examiner(s): Hon, Ming Y
• Application Number:
  US15/874,611
• Publication Number:
  US 20180204307A1
• Time in Patent Office: 796 Days
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/20182 Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning

Abstract

A method, computer readable medium, and system are disclosed for performing spatiotemporal filtering. The method includes the steps of applying, utilizing a processor, a temporal filter of a filtering pipeline to a current image frame, using a temporal reprojection, to obtain a color and auxiliary information for each pixel within the current image frame, providing the auxiliary information for each pixel within the current image frame to one or more subsequent filters of the filtering pipeline, and creating a reconstructed image for the current image frame, utilizing the one or more subsequent filters of the filtering pipeline.

Claims

1. A method comprising:

applying, utilizing a processor, a temporal filter of a filtering pipeline to a current image frame to obtain a color and auxiliary information for each pixel within the current image frame, including for each pixel within the current image frame;

performing back-projection of a first sample of the pixel in the current image frame to a screen space location in a prior frame,

determining a second sample located at the screen space location in the prior frame,

verifying that the first sample and the second sample are located on a same surface, and

computing the color and the auxiliary information for the pixel based on the first sample and the second sample, responsive to verifying that the first sample and the second sample are located on a same surface;

providing the auxiliary information for each pixel within the current image frame to one or more subsequent filters of the filtering pipeline; and

creating a reconstructed image for the current image frame, utilizing the one or more subsequent filters of the filtering pipeline.

2. The method of claim 1, wherein the current image frame includes a number of samples per pixel (spp) below a predetermined threshold.

3. The method of claim 1, wherein the current image frame results from a Monte Carlo sampling method.

4. The method of claim 1, wherein the prior frame is a previously filtered image frame.

5. The method of claim 1, wherein the auxiliary information computed for the pixel based on the first sample and the second sample includes a color variance for the pixel.

6. The method of claim 1, wherein the auxiliary information computed for the pixel based on the first sample and the second sample includes a luminance variance for the pixel.

7. The method of claim 1, wherein the auxiliary information computed for the pixel based on the first sample and the second sample includes frequency analysis information associated with nearby geometry.

8. The method of claim 1, wherein the auxiliary information computed for the pixel based on the first sample and the second sample includes a confidence metric of the color for the pixel.

9. The method of claim 1, wherein the one or more subsequent filters include one or more additional temporal filters.

10. The method of claim 1, wherein the one or more subsequent filters include one or more spatial filters.

11. The method of claim 1, wherein the auxiliary information is used as input for parameters of the one or more subsequent filters to adjust the one or more subsequent filters based on the auxiliary information.

12. The method of claim 1, wherein information output from one subsequent filter is used as input to another subsequent filter within the current image frame.

13. The method of claim 1, wherein information output from one subsequent filter is used as input to another subsequent filter within a subsequent image frame.

14. The method of claim 1, wherein the creating includes storing pixel information in a history buffer for future image frames.

19. The method of claim 1, wherein the temporal filter is applied to the current image frame with respect to a plurality of prior frames, and wherein the temporal filter performs temporal anti-aliasing.

20. The method of claim 1, wherein verifying that the first sample and the second sample are located on a same surface includes verifying consistency between the first sample and the second sample with respect to depth, object-space normal, and mesh ID.

15. A system comprising:

a processor that is configured to;

apply a temporal filter of a filtering pipeline to a current image frame to obtain a color and auxiliary information for each pixel within the current image frame, including for each pixel within the current image frame;

performing back-projection of a first sample of the pixel in the current image frame to a screen space location in a prior frame,

determining a second sample located at the screen space location in the prior frame,

verifying that the first sample and the second sample are located on a same surface, and

computing the color and the auxiliary information for the pixel based on the first sample and the second sample, responsive to verifying that the first sample and the second sample are located on a same surface;

provide the auxiliary information for each pixel within the current image frame to one or more subsequent filters of the filtering pipeline; and

create a reconstructed image for the current image frame, utilizing the one or more subsequent filters of the filtering pipeline.

16. The system of claim 15, wherein the prior frame is a previously filtered image frame.

17. The system of claim 15, wherein the auxiliary information is used as input for parameters of the one or more subsequent filters.

18. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, causes the processor to perform steps comprising:

applying a temporal filter of a filtering pipeline to a current image frame to obtain a color and auxiliary information for each pixel within the current image frame, including for each pixel within the current image frame;

performing back-projection of a first sample of the pixel in the current image frame to a screen space location in a prior frame,

determining a second sample located at the screen space location in the prior frame,

verifying that the first sample and the second sample are located on a same surface, and

computing the color and the auxiliary information for the pixel based on the first sample and the second sample, responsive to verifying that the first sample and the second sample are located on a same surface;

providing the auxiliary information for each pixel within the current image frame to one or more subsequent filters of the filtering pipeline; and

creating a reconstructed image for the current image frame, utilizing the one or more subsequent filters of the filtering pipeline.