METHOD AND SYSTEM FOR EFFICIENT VIDEO PROCESSING

• US 20110091127A1
• Filed: 06/20/2008
• Published: 04/21/2011
• Est. Priority Date: 06/20/2008
• Status: Abandoned Application

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Schein, Segi, Kisilev, Pavel
• Application Number:
  US12/999,306
• Publication Number:
  US 20110091127A1
• Time in Patent Office: Days
• US Class Current: 382/274
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20004 Adaptive image processing
• CPC Class Codes: G06T 2207/20182 Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 5/002 Denoising; Smoothing noise processing or correction adapted to be used in an image pickup device containing and electronic image sensor H04N23/81, H04N25/60 - H04N25/67
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04N 5/21 Circuitry for suppressing or minimising disturbance, e.g. moiré or halo
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Embodiments of the present invention are directed to efficient video processing methods and systems for computationally efficient denoising, sharpening, contrast enhancement, deblurring, and other spatial and temporal processing of a stream of video frames. Embodiments of the present invention separate statistics-related calculations, including estimation of pixel-value-associated variances, standard deviations, noise thresholds, and signal-contrast thresholds, carried out on only a small percentage of video frames selected at a fixed or variable interval from the video stream, from various spatial and temporal processing steps carried out on each frame of the video stream. In certain embodiments of the present invention, the statistics-related calculations are carried out by the general processor or processors of a computer system, while the frame-by-frame spatial and temporal processing is carried out by one or more specialized graphics processors within the computer system.

Claims

1. A video-processing system that receives an input stream of video frames and outputs a stream of enhanced frames for storage, transmission, and/or rendering for display, the real-time video-processing system comprising:

an measurement module, executed on one or more central processing units of a computer system, that generates, from selected frames of the input stream of video frames, one or more functions of statistics with respect to pixel value; and

a processing module, executing on one or more graphical processor units of a computer system, that employ the one or more functions of statistics with respect to pixel value, generated by the measurement module, to process each frame of the input stream of video frames, including denoising, sharpening, contrast enhancing, and deblurring each frame, in order to output the stream of enhanced frames.

2. The video-processing system of claim 1 wherein the measurement module selects frames from which to generate the one or more functions of statistics with respect to pixel value by one of:

selecting each frame that occurs within the input stream of video frames at a fixed frame interval;

selecting each frame that occurs within the input stream of video frames at a fixed time interval;

selecting each frame that occurs within the input stream of video frames at a variable frame interval; and

selecting each frame that occurs within the input stream of video frames at a variable time interval.

3. The video-processing system of claim 1 wherein the measurement module generates, from selected frames of the input stream of video frames, one or more functions of statistics with respect to pixel value by:

generating blocks of pixels from the frame;

computing an average pixel value and estimated sample variance for each block;

partitioning the blocks into partitions, each partition associated with a different average pixel value;

for each partition,

removing outlier blocks based on estimated sample variances to produce a block partition without outlier blocks.

computing a noise-related estimated variance and/or standard deviation from a number of blocks of the block partition without outlier blocks, the number of blocks having lowest estimated sample variances among the blocks of the block partition without outlier blocks, and

computing a contrast-related variance from the blocks of the block partition without outlier blocks;

computing a function of noise-related variance with respect to pixel value and/or a function of noise-related standard deviation with respect to pixel value; and

computing a function of contrast-related variance with respect to pixel value.

4. The video-processing system of claim 1 wherein the processing module includes a multi-scale denoising, sharpening, and contrast-enhancement module that denoises, sharpens, and enhances the contrast of each input frame to generate a corresponding spatially-processed frame by:

at each of one or more currently considered scales of resolution greater than a lowest resolution scale,

downscaling either the input frame or an intermediate frame derived from the input flame to produce an intermediate frame for input to a next-lower resolution scale,

upscaling an intermediate frame received from a lower-resolution scale to produce an intermediate frame for output to a next-higher-resolution scale, and

robustly filtering at least one of the input frame or intermediate frames, using the function of noise-related variance with respect to pixel value or the function of noise-related standard deviation with respect to pixel value to compute noise-related thresholds.

5. The video-processing system of claim 4 wherein the processing module includes a motion-detection module and an adaptive-temporal-processing module.

6. The video-processing system of claim 5 wherein the motion-detection module generates a factor ω

_i,j for each pixel in each spatially-processed frame generated by the multi-scale denoising, sharpening, and contrast-enhancement module by;

in a neighborhood-based operation, considering each pixel (i,j) in the spatially-processed frame occurring at time t in the stream of video frames within neighborhood n(i,j,t) by

computing a magnitude of the difference between each pixel k in the neighborhood n(i,j,t), n(i,j,t)_k, and a corresponding pixel k in the neighborhood n(i,j,t-1)_k of the spatially-processed frame occurring at time t-1 in the stream of video frames and immediately preceding the spatially-processed frame occurring at time t in the stream of video frames,

computing, from the computed magnitudes between the corresponding pixels in neighborhood n(i,j,t) and neighborhood n(i,j,t-1), a probability that the value of pixel (i,j) is influenced by noise, P_noise, and a probability that the value of pixel (i,j) is influenced by motion, P_motion, and

computing ω

_i,j as proportional to the ratio

7. The video-processing system of claim 6 wherein the motion-detection module computes P_noise and P_motion as:

8. The video-processing system of claim 5 wherein the adaptive-temporal-processing module maintains a history frame H that represents a recursively generated history of previous enhanced frames output by the adaptive-temporal-processing module, the history frame defined as:

9. The video-processing system of claim 8 wherein the adaptive-temporal-processing module carries out temporal processing on each spatially-processed frame to produce a corresponding enhanced frame by:

for each pixel (i,j) in the spatially processed frame,

computing the value of the corresponding enhanced-frame pixel E_i,j as

E_i,j=ω

_i,jI(i,j)+(1−

ω

_i,j)H(i,j)

10. The video-processing system of claim 9 wherein, following computing a next enhanced frame E_t corresponding to a spatially-processed frame occurring at time t in the input stream of video frames, the video-processing system replaces the current history frame H by the next enhanced frame E_t.

11. The video-processing system of claim 5 wherein the motion-detection module generates a factor ω

_i,j for each pixel in each spatially-processed frame generated by the multi-scale denoising, sharpening, and contrast-enhancement module by;

in a neighborhood-based operation, considering each pixel (i,j) in the spatially-processed frame occurring at time t in the stream of video frames within neighborhood n(i,j,t) by

computing a magnitude of the difference between each pixel k in the neighborhood n(i,j,t), n(i,j,t)_k, and a corresponding pixel k in the history H(i,j,t-1)_k calculated at time t-1 in the stream of video frames and immediately preceding the spatially-processed frame occurring at time t in the stream of video frames,

computing, from the computed magnitudes between the corresponding pixels in neighborhood n(i,j,t) and history H(i,j,t-1), a probability that the value of pixel (i,j) is influenced by noise, P_noise, and a probability that the value of pixel (i,j) is influenced by motion, P_motion, and

computing ω

_i,j as proportional to the ratio

12. The video-processing system of claim 11 wherein the motion-detection module computes P_noise and P_motion as:

13. The video-processing system of claim 10 wherein the motion-detection module and the adaptive-temporal-processing module are one of:

separate modules; and

functionality within a single temporal-processing module.

14. The video-processing system of claim 10 wherein the multi-scale denoising, sharpening, and contrast-enhancement module is one of:

a separate module;

functionality within a single processing module.

15. 1. A method for enhancing an input stream of video frames, the method comprising:

on one or more central-processing units of a computer system, generating, from selected frames of the input stream of video frames, one or more functions of statistics with respect to pixel value; and

on one or more graphical processor units of the computer system, employing the one or more functions of statistics with respect to pixel value to process each frame of the input stream of video frames, including denoising, sharpening, contrast enhancing, and deblurring each frame, in order to output a stream of enhanced frames for storage, transmission, and/or rendering for display.