Perceptual preprocessing filter for viewing-conditions-aware video coding

• US 10,045,050 B2
• Filed: 04/17/2015
• Issued: 08/07/2018
• Est. Priority Date: 04/25/2014
• Status: Active Grant

• Current Assignee: VID SCALE Inc. (InterDigital, Inc.)
• Original Assignee: VID SCALE Inc. (InterDigital, Inc.)
• Inventors: Vanam, Rahul, Kerofsky, Louis, Reznik, Yuriy
• Primary Examiner(s): Fuentes, Luis Perez
• Application Number:
  US14/690,089
• Publication Number:
  US 20150312595A1
• Time in Patent Office: 1,208 Days
• Field of Search: 37524029, 375240
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/182 the unit being a pixel
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression

Abstract

A perceptual filter may be implemented to filter one or more spatial frequencies from a video signal that are below a contrast sensitivity limit of a viewer of the video signal. The perceptual filter may be configured to adapt one or more perceptual filter parameters on a pixel-basis based on, for example, content, viewing distance, display density, contrast ratio, display luminance, background luminance, and/or age of the viewer. A spatial cutoff frequency of the perceptual filter may be mapped to a contrast sensitivity. The perceptual filter may be used as a preprocessing step for a video encoder so as to lower an encoded bitrate. Temporal filtering of the video frames may be used to maintain continuity of a spatial cutoff frequency to ensure the perceptual filtering effects are not identified as motion by a video encoder, and the temporal filtering may be restricted to static areas of a frame.

Claims

1. A method comprising:

determining a frame of luminance data from an input image having a plurality of pixels, wherein the input image corresponds to a current video frame of a video stream;

generating black-level adjusted luminance pixel data based on the luminance data and a contrast ratio;

performing temporal stabilization of the black-level adjusted luminance pixel data, wherein performing the temporal stabilization comprises;

performing spatial de-noising of the black-level adjusted luminance pixel data at least in part by applying a median filter to the black-level adjusted luminance pixel data;

generating a difference image between the median-filtered current video frame and a past median-filtered video frame of the video stream; and

applying a Gaussian filter to the difference image, wherein a length of the Gaussian filter is determined based on at least one of a viewing distance and a display pixel density;

determining a respective localized contrast sensitivity for each pixel based on the corresponding temporally-stabilized black-level adjusted luminance pixel data;

selecting a respective filter bandwidth for each pixel based at least in part on the respective localized contrast sensitivity of the pixel;

generating a filtered video frame by filtering each pixel according to the respective selected filter bandwidth of the pixel; and

providing the filtered video frame to a video encoder.

2. The method of claim 1, wherein the median filter has a size, in pixels×

pixels, selected from the group consisting of 7×

7, 5×

5, and 3×

3.

3. The method of claim 1, wherein generating the difference image comprises determining a difference square between respective pixel values of the current and past median-filtered video frames.

4. The method of claim 1, wherein generating the difference image comprises determining a difference between respective pixel values of the current and past median-filtered video frames.

5. The method of claim 1, wherein the past median-filtered video frame comprises a video frame immediately preceding the current video frame.

6. The method of claim 1, further comprising comparing the respective pixel values of the Gaussian-filtered difference image with a threshold value T.

7. The method of claim 6, wherein the threshold value T=10⁻

6.

8. The method of claim 6, wherein performing the temporal stabilization further comprises applying an IIR filter to pixels having respective pixel values that exceed the threshold value T.

9. The method of claim 1, wherein performing the temporal stabilization further comprises determining respective IIR-filtered values for pixels P_i,j of a video frame of the video stream.

10. The method of claim 1, wherein performing the temporal stabilization further comprises:

selecting a filter parameter based on a monotonically-decreasing function of Gaussian-filtered pixels; and

applying a first-order IIR filter based on the selected filter parameter.

11. The method of claim 1, wherein performing temporal stabilization of a given pixel is conditioned on determining that the given pixel is associated with a static region of the input image.

12. The method of claim 11, wherein the static region of the input image is determined in part based on pixel differences between successive median-filtered image frames.

13. A video filtering device comprising:

a communication interface;

a processor; and

data storage containing instructions executable by the processor for causing the video filtering device to carry out a set of functions, the set of functions comprising;

determining a frame of luminance data from an input image having a plurality of pixels, wherein the input image corresponds to a current video frame of a video stream;

generating black-level adjusted luminance pixel data based on the luminance data and a contrast ratio;

performing temporal stabilization of the black-level adjusted luminance pixel data, wherein performing the temporal stabilization comprises;

performing spatial de-noising of the black-level adjusted luminance pixel data, wherein performing the spatial de-noising comprises applying a median filter to the black-level adjusted luminance pixel data;

generating a difference image between the median-filtered current video frame and a past median-filtered video frame of the video stream; and

applying a Gaussian filter to the difference image, wherein a length of the Gaussian filter is determined based on at least one of a viewing distance and a display pixel density;

determining a respective localized contrast sensitivity for each pixel based on the corresponding temporally-stabilized black-level adjusted luminance pixel data;

selecting a respective filter bandwidth for each pixel based at least in part on the respective localized contrast sensitivity of the pixel;

generating a filtered video frame by filtering each pixel according to the respective selected filter bandwidth of the pixel; and

providing the filtered video frame to a video encoder.

14. A method comprising:

determining a frame of luminance data from an input image having a plurality of pixels, wherein the input image corresponds to a current video frame of a video stream;

generating black-level adjusted luminance pixel data based on the luminance data and a contrast ratio;

performing temporal stabilization of the black-level adjusted luminance pixel data, wherein performing the temporal stabilization comprises;

performing spatial de-noising of the black-level adjusted luminance pixel data, wherein performing the spatial de-noising comprises applying a median filter to the black-level adjusted luminance pixel data;

generating a difference image between the median-filtered current video frame and a past median-filtered video frame of the video stream; and

applying a Gaussian filter to the difference image, wherein a length N of the Gaussian filter is determined based on a ratio of pixels to cycles according to;