REAL-TIME VIDEO DENOISING METHOD AND TERMINAL DURING CODING, AND NON-VOLATILE COMPUTER READABLE STORAGE MEDIUM

• US 20170374363A1
• Filed: 09/08/2017
• Published: 12/28/2017
• Est. Priority Date: 11/18/2015
• Status: Active Grant

• Current Assignee: Tencent Technology Company Limited (Tencent Holdings Limited)
• Original Assignee: Tencent Technology Company Limited (Tencent Holdings Limited)
• Inventors: DENG, Haibo
• Granted Patent: US 10,587,874 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 2207/20052 Discrete cosine transform [DCT]
• CPC Class Codes: G06T 5/10 using non-spatial domain filtering
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/18 the unit being a set of transform coefficients
• CPC Class Codes: H04N 19/48 using compressed domain processing techniques other than decoding, e.g. modification of transform coefficients, variable length coding [VLC] data or run-length data motion estimation in a transform domain H04N19/547; processing of decoded motion vectors H04N19/513
• CPC Class Codes: H04N 19/52 by predictive encoding
• CPC Class Codes: H04N 19/593 involving spatial prediction techniques
• CPC Class Codes: H04N 19/625 using discrete cosine transform [DCT]
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 21/2402 Monitoring of the downstream path of the transmission network, e.g. bandwidth available traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876

Abstract

A video denoising method includes: while continuing to receive a video stream, performing multi-stage denoising processing on a respective frame, including: detecting a change in a current network status of a network connection; and, in response to detecting the change in the current network status: adjusting a current value for a predefine flatness threshold for detecting a flat area within the respective frame of the image stream in accordance with the change in the current network status of the network connection; identifying one or more flat areas within the respective image frame in accordance with the predefined flatness threshold; and performing, using a predefined filter, denoising processing on the one or more flat areas that have been identified in accordance with the predefined flatness threshold.

Claims

1. A video denoising method, comprising:

at an electronic device having one or more processors and memory;

receiving an image stream comprising a sequence of image frames from a video source; and

while continuing to receive the video stream from the video source, performing multi-stage denoising processing on a respective frame of the sequence of images frames in the video stream, including;

detecting a change in a current network status of a network connection between the electronic device and a remote destination device for the image stream, the change in the current network status includes a change in a bandwidth of the network connection; and

in response to detecting the change in the current network status of the network connection between the electronic device and the remote destination device for the image stream;

adjusting a current value for a predefine flatness threshold for detecting a flat area within the respective frame of the image stream in accordance with the change in the current network status of the network connection;

identifying one or more flat areas within the respective image frame in accordance with the predefined flatness threshold that has been adjusted in accordance with the change in the current network status of the network connection; and

performing, using a predefined filter, denoising processing on the one or more flat areas that have been identified in accordance with the predefined flatness threshold that has been adjusted in accordance with the change in the network status of the network connection.

2. The method according to claim 1, wherein, when a different between pixel values of a respective pixel and an adjacent pixel thereof is below the predefined flatness threshold, the adjacent pixel is included in a respective flat area containing the respective pixel.

3. The method according to claim 2, wherein adjusting the current value for the predefined flatness threshold for detecting a flat area within the respective frame includes:

increasing the current value for the predefined flatness threshold in accordance with a determination that the change in the current network status of the network connection is a decrease in the bandwidth of the network connection below a first bandwidth threshold.

4. The method according to claim 3, wherein adjusting the current value for the predefined flatness threshold for detecting a flat area within the respective frame includes:

decreasing the current value for the predefined flatness threshold in accordance with a determination that the change in the current network status of the network connection is an increase in the bandwidth of the network connection above a second bandwidth threshold.

5. The method according to claim 1, wherein performing the multi-stage denoising processing on the respective frame of the sequence of images frames in the video stream further includes:

after performing the denoising processing on the one or more flat areas that have been identified in the respective frame, performing denoising processing during compression encoding of the respective frame, including;

determining whether the respective image frame is a forward predicted frame (P-frame); and

in accordance with a determination that the respective frame is a P-frame;

reading a respective sub-macroblock of a plurality of sub-macroblocks of the respective frame;

determining whether the respective sub-macroblock is an inter sub-macroblock;

in accordance with a determination that the respective sub-macroblock is an inter sub macroblock, performing integer discrete cosine transform on the respective sub-macroblock; and

in accordance with a determination that the respective sub-macroblock is not an inter sub-macroblock, forgoing performing the integer discrete cosine transform on the respective sub-macroblock.

6. The method according to claim 5, wherein performing denoising processing during compression encoding of the respective frame further includes:

in accordance with the determination that the respective frame is a P-frame and in accordance with the determination that the respective sub-macroblock is an inter sub macroblock, performing denoising processing during encoding of the respective sub-macroblock on which the integer discrete cosine transform has been performed.

7. The method according to claim 5, wherein performing denoising processing during compression encoding of the respective frame further includes:

in accordance with the determination that the respective frame is a P-frame and in accordance with the determination that the respective sub-macroblock is an intra sub macroblock, forgoing performing the integer discrete cosine transform and the denoising processing on the respective sub-macroblock.

8. An electronic device for video denoising, comprising:

one or more processors and memory including instructions, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising;

receiving an image stream comprising a sequence of image frames from a video source; and

while continuing to receive the video stream from the video source, performing multi-stage denoising processing on a respective frame of the sequence of images frames in the video stream, including;

detecting a change in a current network status of a network connection between the electronic device and a remote destination device for the image stream, the change in the current network status includes a change in a bandwidth of the network connection; and

in response to detecting the change in the current network status of the network connection between the electronic device and the remote destination device for the image stream;

adjusting a current value for a predefine flatness threshold for detecting a flat area within the respective frame of the image stream in accordance with the change in the current network status of the network connection;

identifying one or more flat areas within the respective image frame in accordance with the predefined flatness threshold that has been adjusted in accordance with the change in the current network status of the network connection; and

performing, using a predefined filter, denoising processing on the one or more flat areas that have been identified in accordance with the predefined flatness threshold that has been adjusted in accordance with the change in the network status of the network connection.

9. The electronic device according to claim 8, wherein, when a different between pixel values of a respective pixel and an adjacent pixel thereof is below the predefined flatness threshold, the adjacent pixel is included in a respective flat area containing the respective pixel.

10. The electronic device according to claim 9, wherein adjusting the current value for the predefined flatness threshold for detecting a flat area within the respective frame includes:

increasing the current value for the predefined flatness threshold in accordance with a determination that the change in the current network status of the network connection is a decrease in the bandwidth of the network connection below a first bandwidth threshold.

11. The electronic device according to claim 10, wherein adjusting the current value for the predefined flatness threshold for detecting a flat area within the respective frame includes:

decreasing the current value for the predefined flatness threshold in accordance with a determination that the change in the current network status of the network connection is an increase in the bandwidth of the network connection above a second bandwidth threshold.

12. The electronic device according to claim 8, wherein performing the multi-stage denoising processing on the respective frame of the sequence of images frames in the video stream further includes:

after performing the denoising processing on the one or more flat areas that have been identified in the respective frame, performing denoising processing during compression encoding of the respective frame, including;

determining whether the respective image frame is a forward predicted frame (P-frame); and

in accordance with a determination that the respective frame is a P-frame;

reading a respective sub-macroblock of a plurality of sub-macroblocks of the respective frame;

determining whether the respective sub-macroblock is an inter sub-macroblock;

in accordance with a determination that the respective sub-macroblock is an inter sub macroblock, performing integer discrete cosine transform on the respective sub-macroblock; and

in accordance with a determination that the respective sub-macroblock is not an inter sub-macroblock, forgoing performing the integer discrete cosine transform on the respective sub-macroblock.

13. The electronic device according to claim 12, wherein performing denoising processing during compression encoding of the respective frame further includes:

in accordance with the determination that the respective frame is a P-frame and in accordance with the determination that the respective sub-macroblock is an inter sub macroblock, performing denoising processing during encoding of the respective sub-macroblock on which the integer discrete cosine transform has been performed.

14. The electronic device according to claim 13, wherein performing denoising processing during compression encoding of the respective frame further includes:

in accordance with the determination that the respective frame is a P-frame and in accordance with the determination that the respective sub-macroblock is an intra sub macroblock, forgoing performing the integer discrete cosine transform and the denoising processing on the respective sub-macroblock.

15. A non-transitory computer-readable storage medium storing instructions, the instructions, when executed by one or more processors, cause the processors to perform operations comprising:

receiving an image stream comprising a sequence of image frames from a video source; and

while continuing to receive the video stream from the video source, performing multi-stage denoising processing on a respective frame of the sequence of images frames in the video stream, including;

detecting a change in a current network status of a network connection between the electronic device and a remote destination device for the image stream, the change in the current network status includes a change in a bandwidth of the network connection; and

in response to detecting the change in the current network status of the network connection between the electronic device and the remote destination device for the image stream;

adjusting a current value for a predefine flatness threshold for detecting a flat area within the respective frame of the image stream in accordance with the change in the current network status of the network connection;

identifying one or more flat areas within the respective image frame in accordance with the predefined flatness threshold that has been adjusted in accordance with the change in the current network status of the network connection; and

performing, using a predefined filter, denoising processing on the one or more flat areas that have been identified in accordance with the predefined flatness threshold that has been adjusted in accordance with the change in the network status of the network connection.

16. The computer-readable storage medium according to claim 15, wherein, when a different between pixel values of a respective pixel and an adjacent pixel thereof is below the predefined flatness threshold, the adjacent pixel is included in a respective flat area containing the respective pixel.

17. The computer-readable storage medium according to claim 16, wherein adjusting the current value for the predefined flatness threshold for detecting a flat area within the respective frame includes:

increasing the current value for the predefined flatness threshold in accordance with a determination that the change in the current network status of the network connection is a decrease in the bandwidth of the network connection below a first bandwidth threshold.

18. The computer-readable storage medium according to claim 17, wherein adjusting the current value for the predefined flatness threshold for detecting a flat area within the respective frame includes:

decreasing the current value for the predefined flatness threshold in accordance with a determination that the change in the current network status of the network connection is an increase in the bandwidth of the network connection above a second bandwidth threshold.

19. The computer-readable storage medium according to claim 15, wherein performing the multi-stage denoising processing on the respective frame of the sequence of images frames in the video stream further includes:

after performing the denoising processing on the one or more flat areas that have been identified in the respective frame, performing denoising processing during compression encoding of the respective frame, including;

determining whether the respective image frame is a forward predicted frame (P-frame); and

in accordance with a determination that the respective frame is a P-frame;

reading a respective sub-macroblock of a plurality of sub-macroblocks of the respective frame;

determining whether the respective sub-macroblock is an inter sub-macroblock;

in accordance with a determination that the respective sub-macroblock is an inter sub macroblock, performing integer discrete cosine transform on the respective sub-macroblock; and

in accordance with a determination that the respective sub-macroblock is not an inter sub-macroblock, forgoing performing the integer discrete cosine transform on the respective sub-macroblock.

20. The computer-readable storage medium according to claim 19, wherein performing denoising processing during compression encoding of the respective frame further includes:

in accordance with the determination that the respective frame is a P-frame and in accordance with the determination that the respective sub-macroblock is an inter sub macroblock, performing denoising processing during encoding of the respective sub-macroblock on which the integer discrete cosine transform has been performed; and

in accordance with the determination that the respective frame is a P-frame and in accordance with the determination that the respective sub-macroblock is an intra sub macroblock, forgoing performing the integer discrete cosine transform and the denoising processing on the respective sub-macroblock.