ADAPTIVE FILTERING MECHANISM TO REMOVE ENCODING ARTIFACTS IN VIDEO DATA

• US 20140369613A1
• Filed: 06/14/2013
• Published: 12/18/2014
• Est. Priority Date: 06/14/2013
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Avadhanam, Niranjan, Boddeti, Ravi Kumar
• Granted Patent: US 9,118,932 B2
• Time in Patent Office: Days
• US Class Current: 382/233
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/86 involving reduction of coding artifacts, e.g. of blockiness

Abstract

A method includes determining, through a processor and/or a hardware engine, edge pixels and flat pixels of a video frame of a video sequence during decoding thereof or post-processing associated with the decoding based on a predetermined threshold, and quantifying spatial correlation of pixels of the video frame around edges thereof to estimate strength of ringing artifacts and spatial and temporal persistence thereof across the video frame and across video frames of the video sequence. The method also includes adaptively and spatially filtering the pixels around the edges of the video frame, adaptively and temporally filtering the video frame, and blending an output of the adaptive spatial filtering and the adaptive temporal filtering to generate an output with suppressed ringing artifacts, spatial and temporal persistence thereof and artifacts resulting from the cumulative effect of compression therein.

Claims

1. A method comprising:

determining, through at least one of a processor of a data processing device communicatively coupled to a memory and a hardware engine, edge pixels and flat pixels of a video frame of a video sequence during one of decoding thereof and post-processing associated with the decoding based on a predetermined threshold, the edge pixels being associated with at least one edge of the video frame around which there is a change in intensity level above a threshold value thereof and the flat pixels being associated with at least one area of the video frame around which the change in the intensity level is below the threshold value;

quantifying, through the at least one of the processor and the hardware engine, spatial correlation of pixels of the video frame around the at least one edge to estimate a strength of ringing artifacts therein, persistence of the ringing artifacts within the video frame and temporal persistence thereof across the video frame and another video frame of the video sequence based on the determined edge pixels and the flat pixels and a raw output of the decoding;

adaptively and spatially filtering, through an edge preserving filter implemented through at least one of;

executing instructions on the processor and the hardware engine, the pixels around the at least one edge of the video frame based on metadata associated with the video sequence, the estimated strength of the ringing artifacts, the persistence of the ringing artifacts within the video frame and the temporal persistence of the ringing artifacts across the video frame and the another video frame;

adaptively and temporally filtering, through the at least one of the processor and the hardware engine, the video frame to mitigate artifacts resulting from a cumulative effect of compression during encoding of the video sequence; and

blending, through the at least one of the processor and the hardware engine, an output of the adaptive spatial filtering and the adaptive temporal filtering to generate an output with suppressed ringing artifacts, spatial and temporal persistence thereof and artifacts resulting from the cumulative effect of compression therein.

2. The method of claim 1, wherein determining the edge pixels and the flat pixels further comprises at least one of:

estimating the at least one edge of the video frame;

marking at least one pixel surrounding an edge pixel also as an edge pixel to widen a contour of the at least one edge;

computing a range for each sub-block of the video frame as a difference between a maximum and a minimum of intensity levels of pixels therewithin;

comparing the range for the each sub-block to another predetermined threshold to determine the flat pixels and the edge pixels therewithin; and

preserving the edge pixels within the each sub-block.

3. The method of claim 2, further comprising preventing filtering of the each sub-block when a flat pixel percentage and an edge pixel percentage thereof are respectively above a first threshold value and a second threshold value.

4. The method of claim 2, comprising computing an approximate gradient of an intensity level function of the video frame using a Sobel operator as part of the estimation of the at least one edge thereof.

5. The method of claim 1, comprising at least one of:

implementing a bilateral filter having a spatial filter component and a range filter component as the edge preserving filter; and

implementing a first-order Infinite Impulse Response (IIR) filter as a component performing the adaptive temporal filtering, the IIR filter being configured to generate an output by recursively applying alpha-blending of the video frame with a previous temporally filtered video frame of the video sequence.

6. The method of claim 5, further comprising adapting the bilateral filter through varying at least one of a deviation parameter associated with the spatial filter component and a radiometric deviation parameter associated with the range filter component.

7. The method of claim 6, comprising implementing the spatial filter component and the range filter component as Gaussian functions.

8. A data processing device comprising:

a memory; and

a processor communicatively coupled to the memory, the processor being configured to execute instructions to;

determine edge pixels and flat pixels of a video frame of a video sequence during one of decoding thereof and post-processing associated with the decoding based on a predetermined threshold, the edge pixels being associated with at least one edge of the video frame around which there is a change in intensity level above a threshold value thereof and the flat pixels being associated with at least one area of the video frame around which the change in the intensity level is below the threshold value,

quantify spatial correlation of pixels of the video frame around the at least one edge to estimate a strength of ringing artifacts therein, persistence of the ringing artifacts within the video frame and temporal persistence thereof across the video frame and another video frame of the video sequence based on the determined edge pixels and the flat pixels and a raw output of the decoding,

adaptively and spatially filter, through an edge preserving filter, the pixels around the at least one edge of the video frame based on metadata associated with the video sequence, the estimated strength of the ringing artifacts, the persistence of the ringing artifacts within the video frame and the temporal persistence of the ringing artifacts across the video frame and the another video frame,

adaptively and temporally filter the video frame to mitigate artifacts resulting from a cumulative effect of compression during encoding of the video sequence, and

blend an output of the adaptive spatial filtering and the adaptive temporal filtering to generate an output with suppressed ringing artifacts, spatial and temporal persistence thereof and artifacts resulting from the cumulative effect of compression therein.

9. The data processing device of claim 8, wherein the processor is configured to execute instructions to determine the edge pixels and the flat pixels based on:

estimating the at least one edge of the video frame,

marking at least one pixel surrounding an edge pixel also as an edge pixel to widen a contour of the at least one edge thereof,

computing a range for each sub-block of the video frame as a difference between a maximum and a minimum of intensity levels of pixels therewithin,

comparing the range for the each sub-block to another predetermined threshold to determine the flat pixels and the edge pixels therewithin, and

preserving the edge pixels within the each sub-block.

10. The data processing device of claim 9, wherein the processor is further configured to execute instructions to prevent filtering of the each sub-block when a flat pixel percentage and an edge pixel percentage thereof are respectively above a first threshold value and a second threshold value.

11. The data processing device of claim 9, wherein the processor is configured to execute instructions to compute an approximate gradient of an intensity level function of the video frame using a Sobel operator as part of the estimation of the at least one edge thereof.

12. The data processing device of claim 8, wherein the processor is configured to execute instructions associated with at least one of:

a bilateral filter having a spatial filter component and a range filter component as the edge preserving filter, and

a first-order IIR filter as a component performing the adaptive temporal filtering, the IIR filter being configured to generate an output by recursively applying alpha-blending of the video frame with a previous temporally filtered video frame of the video sequence.

13. The data processing device of claim 12, wherein the processor is further configured to execute instructions to adapt the bilateral filter through varying at least one of a deviation parameter associated with the spatial filter component and a radiometric deviation parameter associated with the range filter component.

14. The data processing device of claim 13, wherein the processor is configured to execute instructions associated with the spatial filter component and the range filter being Gaussian functions.

15. A system comprising:

a data source configured to generate video data encoded as a video sequence; and

at least one of a hardware engine and a client device communicatively coupled to the data source, the at least one of the hardware engine and the client device being configured to;

determine edge pixels and flat pixels of a video frame of the video sequence during one of decoding thereof and post-processing associated with the decoding based on a predetermined threshold, the edge pixels being associated with at least one edge of the video frame around which there is a change in intensity level above a threshold value thereof and the flat pixels being associated with at least one area of the video frame around which the change in the intensity level is below the threshold value,

quantify spatial correlation of pixels of the video frame around the at least one edge to estimate a strength of ringing artifacts therein, persistence of the ringing artifacts within the video frame and temporal persistence thereof across the video frame and another video frame of the video sequence based on the determined edge pixels and the flat pixels and a raw output of the decoding,

adaptively and spatially filter, through an edge preserving filter, the pixels around the at least one edge of the video frame based on metadata associated with the video sequence, the estimated strength of the ringing artifacts, the persistence of the ringing artifacts within the video frame and the temporal persistence of the ringing artifacts across the video frame and the another video frame,

adaptively and temporally filter the video frame to mitigate artifacts resulting from a cumulative effect of compression during encoding of the video sequence, and

blend an output of the adaptive spatial filtering and the adaptive temporal filtering to generate an output with suppressed ringing artifacts, spatial and temporal persistence thereof and artifacts resulting from the cumulative effect of compression therein.

16. The system of claim 15, wherein the at least one of the hardware engine and the client device is configured to determine the edge pixels and the flat pixels based on:

estimating the at least one edge of the video frame,

marking at least one pixel surrounding an edge pixel also as an edge pixel to widen a contour of the at least one edge thereof,

computing a range for each sub-block of the video frame as a difference between a maximum and a minimum of intensity levels of pixels therewithin,

comparing the range for the each sub-block to another predetermined threshold to determine the flat pixels and the edge pixels therewithin, and

preserving the edge pixels within the each sub-block.

17. The system of claim 16, wherein the at least one of the hardware engine and the client device is further configured to prevent filtering of the each sub-block when a flat pixel percentage and an edge pixel percentage thereof are respectively above a first threshold value and a second threshold value.

18. The system of claim 16, wherein the at least one of the hardware engine and the client device is configured to compute an approximate gradient of an intensity level function of the video frame using a Sobel operator as part of the estimation of the at least one edge thereof.

19. The system of claim 15, wherein at least one of:

the edge preserving filter is a bilateral filter having a spatial filter component and a range filter component, and

a component performing the adaptive temporal filtering is a first-order IIR filter, the IIR filter being configured to generate an output by recursively applying alpha-blending of the video frame with a previous temporally filtered video frame of the video sequence.

20. The system of claim 19, wherein the at least one of the hardware engine and the client device is further configured to adapt the bilateral filter through varying at least one of a deviation parameter associated with the spatial filter component and a radiometric deviation parameter associated with the range filter component.