Video postfiltering with motion-compensated temporal filtering and/or spatial-adaptive filtering

• US 6,178,205 B1
• Filed: 12/12/1997
• Issued: 01/23/2001
• Est. Priority Date: 12/12/1997
• Status: Expired due to Term

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Vtel Corporation
• Inventors: Cheung, Sen-ching S., Drizen, David, Haskell, Paul E.
• Primary Examiner(s): Britton, Howard
• Assistant Examiner(s): PHILIPPE, GIMS S
• Application Number:
  US08/989,839
• Time in Patent Office: 1,138 Days
• Field of Search: A04/B.166, 348/845, 348/606, 348/607, 348/620, 348/699, 348/701, 348/403, 348/405, 348/413, 348/416, 348/667, 348/420, 348/448, 382/261, 382/263, 382/264, 382/272, 370/290, 375/240, 375/240.29
• US Class Current: 375/240.29
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 19/86 involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/21 Circuitry for suppressing or minimising disturbance, e.g. moiré or halo

Abstract

A postfiltering process for improving the appearance of a video image includes motion compensated temporal filtering and spatial adaptive filtering. For each target pixel being filtered, the temporal filtering uses multiple motion vectors and one or more pixel values for a prior frame to determine one of more reference values for the target filter. In one embodiment, a weighted average of multiple motion vectors for blocks near or containing the target pixel value provides a filter vector that points to a pixel value in the prior frame. This pixel value is a reference value for the target pixel value and is combined with the target pixel value in a filter operation. Alternatively, multiple motion vectors for blocks near or containing the target pixel value point to pixel values in the prior frame that are averaged to determine a reference value for the target pixel value. In each alternative, the weighting for the average is selected according to the position of the target pixel value. The spatial filtering determines a dynamic range of pixel values in a smaller block containing the target pixel value and a dynamic range of pixel values in a larger block containing the target pixel value. The two dynamic ranges suggest the image context of the target pixel, and an appropriate spatial filter for the target pixel is selected according to the suggested context.

Claims

1. A method for improving appearance of a video image, comprising:

representing a first frame in the video image by a first array of pixel values and a second frame in the video image by a second array of pixel values;

selecting a plurality of motion vectors for a target pixel value in the first array, wherein each motion vector corresponds to a block of pixel values in the first array and identifies a block of pixel values in the second array;

determining a reference value for the target pixel value, wherein the reference value depends on the motion vectors selected for the target pixel value and one or more pixel values from the second array; and

combining the target pixel value with the reference value in a filter operation that generates an output pixel value for a third array, the third array representing improved version of the first frame, wherein the output pixel value is equal to the target pixel value if a difference between the target value and the reference value is greater than a threshold value and is equal to a linear combination of the target pixel value and the reference value if the difference is not greater than the threshold value.

2. The method of claim 1, wherein selecting the motion vectors comprises:

selecting a first motion vector that corresponds to a first block containing the target pixel value; and

selecting a second motion vector that corresponds to a second block neighboring the first block.

3. The method of claim 2, wherein the second block abuts the first block, and of blocks that abut the first block, the second block has a boundary closest to the target pixel value.

4. The method of claim 3, wherein determining the reference value for the target pixel value comprises:

combining the motion vectors selected for the target pixel value to generate a filter vector; and

selecting as the reference value a pixel value in the second array, at a position offset from a position of the target pixel value by an amount indicated by the filter vector.

5. The method of claim 4, wherein combining the motion vectors comprises:

selecting weighting factors that depend on the position of the target pixel value in the first array; and

determining a weighted average of the motion vectors using the selected weighting factors.

6. The method of claim 3, wherein determining the reference value for the target pixel value comprises:

for each of the motion vectors selected for the target pixel value, identifying a pixel value that is in the second array, at a position that is offset from a position corresponding to the target pixel value by an amount indicated by the motion vector; and

combining the pixel values identified to determine the reference value for the target pixel value.

7. The method of claim 6, wherein combining the pixel values comprises:

selecting weighting factors that depend on the position of the target pixel value in the first array; and

determining a weighted average of the identified pixel values using the selected weighting factors.

8. The method of claim 1, wherein determining the reference value for the target pixel value comprises:

combining the motion vectors selected for the target pixel value to generate a filter vector; and

selecting as the reference value a pixel value from the second array, wherein the pixel value selected is in the second array, at a position that is offset from a position corresponding to the target pixel value by an amount indicated by the filter vector.

9. The method of claim 8, wherein combining the motion vectors comprises:

selecting weighting factors that depend on the position of the target pixel value in the first array; and

determining a weighted average of the motion vectors using the selected weighting factors.

10. The method of claim 1, wherein determining the reference value for the target pixel value comprises:

for each of the motion vectors selected for the target pixel value, identifying a pixel value that is in the second array, at a position offset from a position corresponding to the target pixel value by an amount indicated by the motion vector; and

combining the pixel values identified to determine the reference value for the target pixel value.

11. The method of claim 10, wherein combining the pixel values comprises:

selecting weighting factors that depend on the position of the target pixel value in the first array; and

determining a weighted average of the identified pixel values using the selected weighting factors.

12. The method of claim 1, further comprising decoding a bit stream representing the video image, wherein:

the decoding exacts from the bit stream motion vectors that are required for further decoding of the bit stream; and

selecting the plurality of motion vectors for the target pixel value comprises selecting a motion vector extracted from the bit stream.

13. The method of claim 12, wherein the bit stream is encoded according to a video standard selected from a group consisting of the MPEG-1 standard, the MPEG-2 standard, the H.261 standard, and the H.263 standard.

14. The method of claim 12, wherein decoding further includes determining a quantization factor from the bit stream.

15. The method of claim 1, further comprising for each pixel value in the first array, repeating the selecting, determining, and combining steps with the pixel value as the target pixel value.

16. A method for improving appearance of a video image, comprising:

determining motion vectors for first areas in a first frame of the video image that is represented by a first array of pixel values, each motion vector corresponding to a first area in the first frame and a second area in a second frame, wherein image content of the second area in the second frame is similar to the image content of the first area in the first frame;

determining for each pixel in the first frame a reference vector that is a combination of a motion vector for a first area containing the pixel and one or more of the motion vectors for adjacent first areas;

generating a reference array containing reference values, wherein each reference value in the reference array is equal to the pixel value at a relative position in the second array that is offset from a position of the reference value by an amount indicated by the reference vector; and

generating a filtered array representing an improved version of the first frame, wherein the filtered array contains pixel values that are combinations of pixel values from the first array and the reference values, and wherein each pixel value in the filtered array is equal to a corresponding pixel value in the first array if a difference between the corresponding value and a corresponding reference value in the reference array is greater than a threshold value and is equal to a linear combination of the corresponding pixel value and the corresponding reference value if the difference is not greater than the threshold value.

17. The method of claim 16, wherein determining reference vectors comprises combining of the motion vector for the first area containing the pixel and motion vectors for first areas that are nearest to the first area containing the pixel.

18. A method for improving appearance of an image, comprising:

representing the image using a first array of pixel values;

determining a first range for pixel values in a first block that is in the first array and includes a target pixel value;

determining a second range for pixel values in a second block that is in the first array and includes the target pixel value, wherein the second block is smaller than the first block;

selecting a spatial filter from a plurality of spatial filters, wherein the spatial filter is selected according to the first and second ranges; and

applying the selected spatial filter to the target pixel value, wherein applying the selected spatial filter combines the target pixel value with surrounding pixel values in the first array to generate a corresponding pixel value in a second array representing the image.

19. The method of claim 18, wherein the second block is a 3×

3 block of pixel values centered on the target pixel value.

20. The method of claim 19, further comprising performing an inverse frequency transformation on a block of transform coefficients to determine the pixel values in the first block.

21. The method of claim 18, wherein selecting the spatial filter comprises:

selecting a first spatial filter in response to the second range being greater than a first threshold value; and

selecting a second spatial filter in response to the first range being greater than a second threshold and the second range being less than a third threshold, wherein the second spatial filter is stronger than first spatial filter.

22. The method of claim 18, for each pixel value in the first array, using that pixel value as the target pixel in a repetition of the steps of determining the first range, determining the second range, selecting a spatial filter, and applying the selected spatial filter.

23. The method of claim 18, wherein applying the selected spatial filter comprises:

identifying a likeness threshold that corresponds to the second range; and

excluding from the combination that generates the corresponding pixel value any pixel values that differ from the target pixel value by more than the likeness threshold.

24. The method of claim 18, wherein selecting a spatial filter comprises selecting a filter strength parameter β

corresponding to the first and second ranges.

25. The method of claim 24, wherein:

the target pixel value is Pij;

the corresponding value is Oij and is determined from pixel values of the first array according to an equation Oij=(1=β

)*Pij+β

*F(Pij), where F(Pij) is a linear combination of one or more pixel values near the target pixel value in the first array.

26. The method of claim 25, wherein applying the selected spatial filter comprises identifying a likeness threshold that corresponds to the second range, and linear combination F(Pij) excludes pixel values that differ from the target pixel value by more than the likeness threshold.

27. A method for improving appearance of an image, comprising:

representing the image using a first array of pixel values;

determining a range of pixel values in a block that is in the first array and includes a target pixel value;

identifying a likeness threshold that corresponds to the range determined; and

generating an output pixel value for a second array representing an improved-appearance version of the image, the output pixel value being a linear combination of the target pixel value and one or more pixel values of the first array, the linear combination excluding pixel values that differ from the target pixel by more than the likeness threshold.

28. The method of claim 27, wherein the likeness threshold is linearly related to the range.

29. A method for improving appearance of a video image, comprising:

decoding a signal to generate a first series of arrays of pixel values, wherein each array of pixel values represents a frame in the video image and comprises a set of blocks;

applying a block boundary filter to pixel values at boundaries of the blocks in the frames to generate a second series of arrays of pixel values, wherein applying the block boundary filter leaves unchanged pixel values that are not at a boundary of any of the blocks;

performing a temporal filtering operation that combines pixel values from different arrays in the second series to generate a third series of arrays of pixel values; and

applying a spatial filter to the arrays in the third series to generate a fourth series of arrays representing the video image with improved appearance.

30. The method of claim 29, wherein the signal comprises a plurality of sets of transformation coefficients with each set corresponding to a different one of the blocks in the arrays of the first series, and decoding comprises for each set of transformation coefficients, performing an inverse transformation on the set of transformation coefficients to generate pixel values in the block corresponding to the set of transformation coefficients.

31. The method of claim 29, wherein applying the spatial filter comprises:

filtering each pixel value in an array using a filter that has an adjustable parameter; and

altering the parameter according to content of an area in a frame that includes a pixel represented by a pixel value being filtered.