Real-time video denoising method and terminal during coding, and non-volatile computer readable storage medium

• US 10,587,874 B2
• Filed: 09/08/2017
• Issued: 03/10/2020
• Est. Priority Date: 11/18/2015
• Status: Active Grant

• Current Assignee: Tencent Technology Company Limited (Tencent Holdings Limited)
• Original Assignee: Tencent Technology Company Limited (Tencent Holdings Limited)
• Inventors: Deng, Haibo
• Primary Examiner(s): Rahman, Mohammad J
• Application Number:
  US15/699,710
• Publication Number:
  US 20170374363A1
• Time in Patent Office: 914 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 2207/20052 Discrete cosine transform [DCT]
• CPC Class Codes: G06T 5/10 using non-spatial domain filtering
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/18 the unit being a set of transform coefficients
• CPC Class Codes: H04N 19/48 using compressed domain processing techniques other than decoding, e.g. modification of transform coefficients, variable length coding [VLC] data or run-length data motion estimation in a transform domain H04N19/547; processing of decoded motion vectors H04N19/513
• CPC Class Codes: H04N 19/52 by predictive encoding
• CPC Class Codes: H04N 19/593 involving spatial prediction techniques
• CPC Class Codes: H04N 19/625 using discrete cosine transform [DCT]
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 21/2402 Monitoring of the downstream path of the transmission network, e.g. bandwidth available traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876

Abstract

A video denoising method includes: while continuing to receive a video stream, performing multi-stage denoising processing on a respective frame, including: detecting a change in a current network status of a network connection; and, in response to detecting the change in the current network status: adjusting a current value for a predefine flatness threshold for detecting a flat area within the respective frame of the image stream in accordance with the change in the current network status of the network connection; identifying one or more flat areas within the respective image frame in accordance with the predefined flatness threshold; and performing, using a predefined filter, denoising processing on the one or more flat areas that have been identified in accordance with the predefined flatness threshold.

Claims

1. A video denoising method, comprising:

at an electronic device having one or more processors and memory;

receiving a video stream comprising a sequence of image frames from a video source;

while receiving the video stream from the video source, performing denoising on a respective frame of the sequence of image frames in the video stream to obtain a processed frame, including;

detecting, at the electronic device, a bandwidth change in a current network status of a network connection between the electronic device and a remote destination device;

determining a current value for a predefine flatness threshold for detecting a flat area within the respective frame of the video stream in accordance with the bandwidth change in the current network status of the network connection, wherein a difference between any two adjacent pixel values in the flat area is smaller than the predefined flatness threshold;

identifying one or more flat areas within the respective frame in accordance with the predefined flatness threshold that has been determined in accordance with the bandwidth change in the current network status of the network connection; and

performing, using a predefined filter, a first denoising process on the one or more flat areas that have been identified in accordance with the predefined flatness threshold that has been adjusted in accordance with the bandwidth change in the current network status of the network connection;

after performing the first denoising process, determining a prediction type of the respective frame;

in accordance with a determination that the prediction type of the respective frame is a forward predicted frame (P-frame), performing a second denoising process during compression encoding of the respective frame, including;

in accordance with a determination that a respective sub-macroblock of the respective frame is an inter sub-macroblock, performing an integer discrete cosine transform (DCT) operation on the respective inter sub-macroblock to obtain low-frequency component coefficients and high-frequency component coefficients of the respective inter sub-macroblock;

 

determining a component offset value in accordance with the bandwidth change in the current network status of the network connection;

subtracting the component offset value from each of the high-frequency component coefficients of the respective inter sub-macroblock of the respective frame if an absolute value of a respective high-frequency component coefficient is greater than the component offset value; and

sending the processed frame to the remote destination device.

2. The method according to claim 1, wherein adjusting the current value for the predefined flatness threshold for detecting a flat area within the respective frame includes:

increasing the current value for the predefined flatness threshold in accordance with a determination that the bandwidth change in the current network status of the network connection is a decrease in the bandwidth of the network connection below a first bandwidth threshold.

3. The method according to claim 2, wherein adjusting the current value for the predefined flatness threshold for detecting a flat area within the respective frame includes:

decreasing the current value for the predefined flatness threshold in accordance with a determination that the bandwidth change in the current network status of the network connection is an increase in the bandwidth of the network connection above a second bandwidth threshold.

4. The method according to claim 1, wherein performing denoising processing during compression encoding of the respective frame further includes:

in accordance with a determination that the respective sub-macroblock is an intra sub macroblock, forgoing performing the integer discrete cosine transform and the denoising processing on the respective sub-macroblock.

5. An electronic device for video denoising, comprising:

one or more processors and memory including instructions, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising;

receiving a video stream comprising a sequence of image frames from a video source;

while receiving the video stream from the video source, performing denoising on a respective frame of the sequence of image frames in the video stream to obtain a processed frame, including;

detecting, at the electronic device, a bandwidth change in a current network status of a network connection between the electronic device and a remote destination device;

determining a current value for a predefine flatness threshold for detecting a flat area within the respective frame of the video stream in accordance with the bandwidth change in the current network status of the network connection, wherein a difference between any two adjacent pixel values in the flat area is smaller than the predefined flatness threshold;

identifying one or more flat areas within the respective frame in accordance with the predefined flatness threshold that has been determined in accordance with the bandwidth change in the current network status of the network connection; and

performing, using a predefined filter, a first denoising process on the one or more flat areas that have been identified in accordance with the predefined flatness threshold that has been adjusted in accordance with the bandwidth change in the current network status of the network connection;

after performing the first denoising process, determining a prediction type of the respective frame;

in accordance with a determination that the prediction type of the respective frame is a forward predicted frame (P-frame), performing a second denoising process during compression encoding of the respective frame, including;

in accordance with a determination that a respective sub-macroblock of the respective frame is an inter sub-macroblock, performing an integer discrete cosine transform (DCT) operation on the respective inter sub-macroblock to obtain low-frequency component coefficients and high-frequency component coefficients of the respective inter sub-macroblock;

 

determining a component offset value in accordance with the bandwidth change in the current network status of the network connection;

subtracting the component offset value from each of the high-frequency component coefficients of the respective inter sub-macroblock of the respective frame if an absolute value of a respective high-frequency component coefficient is greater than the component offset value; and

sending the processed frame to the remote destination device.

6. The electronic device according to claim 5, wherein adjusting the current value for the predefined flatness threshold for detecting a flat area within the respective frame includes:

increasing the current value for the predefined flatness threshold in accordance with a determination that the bandwidth change in the current network status of the network connection is a decrease in the bandwidth of the network connection below a first bandwidth threshold.

7. The electronic device according to claim 6, wherein adjusting the current value for the predefined flatness threshold for detecting a flat area within the respective frame includes:

decreasing the current value for the predefined flatness threshold in accordance with a determination that the bandwidth change in the current network status of the network connection is an increase in the bandwidth of the network connection above a second bandwidth threshold.

8. The electronic device according to claim 5, wherein performing denoising processing during compression encoding of the respective frame further includes:

in accordance with a determination that the respective sub-macroblock is an intra sub macroblock, forgoing performing the integer discrete cosine transform and the denoising processing on the respective sub-macroblock.

9. A non-transitory computer-readable storage medium storing instructions, the instructions, when executed by one or more processors, cause the processors to perform operations comprising:

receiving a video stream comprising a sequence of image frames from a video source;

while receiving the video stream from the video source, performing denoising on a respective frame of the sequence of image frames in the video stream to obtain a processed frame, including;

detecting, at the electronic device, a bandwidth change in a current network status of a network connection between the electronic device and a remote destination device;

determining a current value for a predefine flatness threshold for detecting a flat area within the respective frame of the video stream in accordance with the bandwidth change in the current network status of the network connection, wherein a difference between any two adjacent pixel values in the flat area is smaller than the predefined flatness threshold;

identifying one or more flat areas within the respective frame in accordance with the predefined flatness threshold that has been determined in accordance with the bandwidth change in the current network status of the network connection; and

performing, using a predefined filter, a first denoising process on the one or more flat areas that have been identified in accordance with the predefined flatness threshold that has been adjusted in accordance with the bandwidth change in the current network status of the network connection;

after performing the first denoising process, determining a prediction type of the respective frame;

in accordance with a determination that the prediction type of the respective frame is a forward predicted frame (P-frame), performing a second denoising process during compression encoding of the respective frame, including;

in accordance with a determination that a respective sub-macroblock of the respective frame is an inter sub-macroblock, performing an integer discrete cosine transform (DCT) operation on the respective inter sub-macroblock to obtain low-frequency component coefficients and high-frequency component coefficients of the respective inter sub-macroblock;

determining a component offset value in accordance with the bandwidth change in the current network status of the network connection;

subtracting the component offset value from each of the high-frequency component coefficients of the respective inter sub-macroblock of the respective frame if an absolute value of a respective high-frequency component coefficient is greater than the component offset value; and

sending the processed frame to the remote destination device.

10. The computer-readable storage medium according to claim 9, wherein adjusting the current value for the predefined flatness threshold for detecting a flat area within the respective frame includes:

increasing the current value for the predefined flatness threshold in accordance with a determination that the bandwidth change in the current network status of the network connection is a decrease in the bandwidth of the network connection below a first bandwidth threshold.

11. The computer-readable storage medium according to claim 10, wherein adjusting the current value for the predefined flatness threshold for detecting a flat area within the respective frame includes:

decreasing the current value for the predefined flatness threshold in accordance with a determination that the bandwidth change in the current network status of the network connection is an increase in the bandwidth of the network connection above a second bandwidth threshold.