Enhancement of decompressed video

• US 20060188168A1
• Filed: 02/22/2005
• Published: 08/24/2006
• Est. Priority Date: 02/22/2005
• Status: Active Grant

• Current Assignee: Rateze Remote Mgmt. L.L.C. (Intellectual Ventures LLC)
• Original Assignee: Somle Development LLC (Intellectual Ventures LLC)
• Inventors: Sheraizin, Vitaly S., Sheraizin, Semion M.
• Granted Patent: US 7,526,142 B2
• Time in Patent Office: Days
• US Class Current: 382/254
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20182 Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/86 involving reduction of coding artifacts, e.g. of blockiness

Abstract

A method and apparatus for enhancing the video quality of compressed video signals adaptively removes distortions and ringing effects embedded in the decompressed images. The apparatus operates in conjunction with decoder devices installed in set-top boxes, satellite receivers, TV broadcast channel servers, digital still cameras, DVD players and recorders, large screen TV sets, media players, and the like.

Claims

1. An apparatus comprising:

an estimation unit to generate visual quality parameters that indicate the visual quality of decompressed video frames; and

a decompressed video enhancer to substantially improve the visual quality of said decompressed video frames using their associated parameters.

2. The apparatus of claim 1 wherein said visual quality is defined at least as the extent of distortions and ringing effects.

3. The apparatus of claim 1, wherein said apparatus is installed in at least one of:

a set-top box, a satellite receiver, a television (TV) broadcast channel server, and a large screen TV set.

4. The apparatus of claim 1, wherein said apparatus is installed in at least one of:

a digital still camera and a mobile video.

5. The apparatus of claim 1, wherein said apparatus is installed in at least one of:

a digital videodisc (DVD) player and a DVD recorder.

6. The apparatus of claim 1, wherein said apparatus is installed in a media player.

7. The apparatus of claim 1, wherein said decompressed video frames are decompressed according to a decoding technique including at least one of:

a joint photographic expert group (JPEG), a moving picture expert group (MPEG), an H-compression, and a Windows Media (WM).

8. The apparatus of claim 1, wherein said estimation unit comprises:

a distortion estimator to estimate a distortion level DR in a frame;

a parameter estimator to estimate an image complexity value NC and a per pixel intensity change h_ij; and

a high contrast details analyzer to estimate high contrast small details levels CH_i,j in a frame.

9. The apparatus of claim 8, wherein said distortion estimator comprises a blockness level determiner.

10. The apparatus of claim 9, wherein said blockness level determiner comprises:

an into line integrator to sum high frequency components in every k pixels along a line of said frame; and

a line-to-line periodicity detector to sum the output of said line integrator every k^th line, starting at different lines and to sum those blocks which are distorted.

11. The apparatus of claim 8, wherein said parameter estimator comprises:

an intensity change generator to generate per-pixel intensity change h_{i,j; and}

an image complexity generator to determine per-frame image complexity level NC.

12. The apparatus according to claim 11 and wherein said intensity change generator comprises:

a first frame memory to generate a difference frame Δ

_F of said decompressed video frame and a previous decompressed video frame;

an absolute value unit to take an absolute value of said difference frame Δ

_F; and

a horizontal low pass filter (LPF) to reduce noise in the output of said absolute value unit.

13. The apparatus according to claim 11 and wherein said image complexity generator comprises:

a histogram difference generator to determine how different a histogram of the intensities of said frame is from that of a previous frame;

a frame intensity change generator to determine an overall change value from said per-pixel intensity changes h_i,j, said overall change value indicating the extent of significant change in said frame; and

an image complexity generator to generate said per-frame image complexity level NC from a weighted sum of the outputs of said generators.

14. The apparatus of claim 8, wherein said high contrast details analyzer comprises:

a shifter to shift said frequency components of said decompressed video frame downward by a first threshold, wherein said first threshold THD1 is defined by expected intensity levels of high contrast details;

a first limiter to limit high frequency components of said decompressed video frame to an intensity level below a second threshold;

a two-dimensional low pass filter operating on the output of said limiter to detect dots in said frame; and

a second limiter to limit the output of said two-dimensional low pass filter to an intensity level below a third threshold.

15. The apparatus of claim 8 wherein said enhancer comprises:

an adaptive temporal processor to adapt temporal processing of each said decompressed video frame with at least one of said associated parameters; and

an adaptive spatial processor to adapt spatial processing of the output of said adaptive temporal processor with at least one of said associated parameters.

16. The apparatus of claim 15 and further comprising a controller to generate at least temporal control parameters for said adaptive temporal processor and spatial-temporal control parameters for said adaptive spatial processor from at least one of said visual quality parameters.

17. The apparatus of claim 16, wherein said temporal control parameters comprise at least one of:

a temporal threshold THD_TP determined by;

THD_TP=2└

(1+DR*)+(1+NC)┘

;

a temporal texture coefficient K^TP_text determined by;

K_text^TP=1−

DR*;

a temporal contrast coefficient K^TP_cont determined by;

K_cont^TP=1−

DR*; and

a low frequency coefficient K^TP determined by;

K_F=1-0.5 DR*,

wherein said DR* is a normalized value of said distortion level DR.

18. The apparatus of claim 16, wherein said spatial-temporal control parameters comprise at least one of:

a spatial-temporal threshold THD_SP determined by;

THD_SP=3(1+DR*),

a per-pixel, recursion coefficient K_rec(i,j) determined by;

K_rec(i,j)=K_rec.o└

1+0.25(DR*+CH_i,j)┘

;

a per-pixel ringing coefficient K_ring(i,j) determined by;

K_ring(i,j)=1−

0.5 h_ij,

a spatial-temporal texture coefficient K^SP_text determined by;

K_text^SP=K_text.o(1−

DR*); and

a per-pixel, spatial-temporal contrast coefficient K^SP_cont(i,j) K_cont^SP(i,j)=K_cont.o└

1−

0.5(DR*−

CH_i,j*)┘

,

wherein said DR* is a normalized value of said distortion level DR, said CH_i,j* are normalized values of said CH_i,j values and said h_ij* are normalized values of said h_ij, and

where said K_cont.o is a maximum contrast coefficient,

said K_rec.o is a maximum recursion coefficient, and

said K_text.o is a maximum texture coefficient.

19. The apparatus of claim 18, wherein said per-pixel, ringing coefficient K_ring(i,j) is used to reduce ringing effects in said decompressed frame.

20. The apparatus of claim 17, wherein said adaptive temporal processor comprises:

a temporal processor to temporally process said decompressed video frame;

a texture improver to receive vertical and horizontal high pass filtered signals from said temporal processor and to attempt to sharpen textual elements in said frame therefrom;

a small details sharpener to adjust the sharpness of small details in said frame; and

a distortion reducer to reduce blockness in said frame by attenuating low frequency components of said frame received from said temporal processor.

21. The apparatus of claim 20 and wherein said texture improver comprises:

a first limiter to limit an input signal level below said temporal threshold THD_TP, wherein the input signal includes vertical high frequency components of said decompressed frame;

a second limiter to limit an input signal level below said temporal threshold THD_TP, wherein the input signal includes horizontal high frequency components of said decompressed frame;

a horizontal low pass filter (LPF) to pass low frequency components of a signal produced by said first limiter;

a vertical LPF to pass low frequency components of a signal produced by said second limiter; and

a texture sharpener to adjust texture sharpness of the sum of the outputs of said LPFs according to said temporal texture coefficient K^TP_text.

22. The apparatus of claim 21 and wherein said small details sharpener comprises:

a small details unit to generate a small details signal from said vertical and horizontal high frequency components and the outputs of said limiters; and

a small details sharpener to adjust a sharpness of said small details signal according to said temporal contrast coefficient K^TP_cont.

23. The apparatus of claim 21 and wherein said distortion reducer comprises a blockness sharpener to reduce the level of low frequency components of said decompressed video frame according to said blockness coefficient K_F.

24. The apparatus of claim 15, wherein said adaptive temporal processor comprises an infinite impulse response (IIR) filter using an adaptive recursion coefficient determined at least from said estimated distortion level DR.

25. The apparatus of claim 18, wherein said adaptive spatial processor comprises:

a spatial-temporal processor to perform spatial processing on said decompressed video frame;

a texture improver to adjust textual elements in said frame;

a small details sharpener to adjust the sharpness of small details in said frame; and

a ringing reducer to receive vertical and horizontal high pass filtered signals from said spatial-temporal processor and to reduce ringing effects in said frame.

26. The apparatus of claim 25, wherein said ringing reducer comprises:

a first limiter to limit an input signal level below said spatial threshold THD_SP, wherein the input signal includes vertical high frequency components of said decompressed frame;

a second limiter to limit an input signal level below said spatial threshold THD_sp, wherein the input signal includes horizontal high frequency components of said decompressed frame;

a vertical anti-ringing unit to reduce ringing effects in said vertical high pass filtered signals according to said ringing coefficient K_ring(i,j); and

a horizontal anti-ringing to reduce ringing effects in said vertical high pass filtered signals according to said ringing coefficient K_{ring (i,j)}.

27. The apparatus of claim 25 wherein said small details sharpener comprises a high contrast sharpener to adjust a sharpness of said small details signal according to said spatial-temporal contrast coefficient K^SP_cont(i,j).

28. The apparatus of claim 25 and wherein said texture improver comprises:

a temporal low pass filter (LPF) to reduce noise according to said recursion coefficient K_rec; and

a texture sharpener to adjust texture sharpness of an output of said temporal LPF according to a value of said spatial-temporal texture coefficient K^SP_text.

29. A method comprising:

estimating visual quality parameters that indicate the visual quality of decompressed video frames; and

substantially improving the visual quality of said decompressed video frames using their associated parameters.

30. The method of claim 29 wherein said visual quality is defined at least as the extent of distortions and ringing effects.

31. The method of claim 29, wherein said decompressed video frames are decompressed according to a decoding technique including at least one of:

a joint photographic expert group (JPEG), a moving picture expert group (MPEG), a H-compression, and a Windows Media (WM).

32. The method of claim 29, wherein said estimating comprises:

estimating a distortion level DR in a frame, an image complexity value NC, a per pixel intensity change h_ij and a high contrast small details level CH in a frame.

33. The method of claim 32, wherein said estimating comprises determining a blockness level.

34. The method of claim 33, wherein said determining comprises:

summing high frequency components in every k pixels along a line of said frame; and

detecting line-to-line periodicity by summing the output of said high frequency summing every k^th line, starting at different lines and by summing those blocks which are distorted.

35. The method of claim 32, wherein said estimating comprises:

generating a per-pixel intensity change h_i,j; and

determining a per-frame image complexity level NC.

36. The method according to claim 35 and wherein said generating comprises:

subtracting said decompressed video frame and a previous decompressed video frame to generate a difference frame Δ

_F;

taking an absolute value of said difference frame Δ

_F; and

horizontally low pass filtering to reduce noise in the output of said absolute value unit.

37. The method according to claim 35 and wherein said determining comprises:

subtracting a histogram of the intensities of said frame from that of a previous frame;

generating an overall change value from said per-pixel intensity changes h_i,j, said overall change value indicating the extent of significant change in said frame; and

weighted summing the outputs of said subtracting and generating to generate said per-frame image complexity level NC.

38. The method of claim 32, wherein said estimating comprises:

shifting said frequency components of said decompressed video frame downward by a first threshold, wherein said first threshold THD1 is defined by expected intensity levels of high contrast details;

limiting high frequency components of said decompressed video frame to an intensity level below a second threshold;

two-dimensional low pass filtering the output of said limiting to detect dots in said frame; and

limiting the output of said two-dimensional low pass filter to an intensity level below a third threshold.

39. The method of claim 32 wherein said substantially improving comprises:

temporally adapting processing of each said decompressed video frame with at least one of said associated parameters; and

spatially adapting processing of the output of said temporal adapting with at least one of said associated parameters.

40. The method of claim 39 and further comprising generating at least temporal control parameters and spatial-temporal control parameters from at least one of said visual quality parameters.

41. The method of claim 40, wherein said temporal control parameters comprise at least one of:

a temporal threshold THD_TP determined by;

THD_TP=2└

(1+DR*)+(1+NC)┘

;

a temporal texture coefficient K^TP_text determined by;

K_text^TP, 1−

DR*;

a temporal contrast coefficient K^TP_cont determined by;

K_cont^TP=1−

DR*; and

a low frequency coefficient K_F determined by;

K_F=1−

0.5 DR*,

wherein said DR* is a normalized value of said distortion level DR.

42. The method of claim 40, wherein said spatial-temporal control parameters comprise at least one of:

a spatial-temporal threshold THD_SP determined by;

THD_SP=3(1+DR*),

a per-pixel, recursion coefficient K_rec(i,j) determined by;

K_rec(i,j)=K_rec.o└

1+0.25(DR*+CH_i,j*)┘

;

a per-pixel ringing coefficient K_ring(i,j) determined by;

K_ring(i,j)=1−

0.5 h_ij,

a spatial-temporal texture coefficient K^SP_text determined by;

K_text^SP=K_text.o(1−

DR*); and

a per-pixel, spatial-temporal contrast coefficient K^SP_cont(i,j) K_cont^SP(i,j)=K_cont.o└

1−

0.5(DR*−

CH_i,j*)┘

,

wherein said DR* is a normalized value of said distortion level DR, said CH_i,j* are normalized values of said CH_i,j values and said h_ij* are normalized values of said h_ij, and

where said K_cont.o is a maximum contrast coefficient,

said K_rec.o is a maximum recursion coefficient, and

said K_text.o is a maximum texture coefficient.

43. The method of claim 42, and comprising using said per-pixel, ringing coefficient K_ring(i,j) to reduce ringing effects in said decompressed frame.

44. The method of claim 41, wherein said temporally adapting comprises:

temporally processing said decompressed video frame;

first adjusting the sharpness of textual elements in said frame;

second adjusting the sharpness of small details in said frame; and

reducing blockness in said frame by attenuating low frequency components of said frame received from said temporal processing.

45. The method of claim 44 and wherein said first adjusting comprises:

first limiting an input signal level below said temporal threshold THD_TP, wherein the input signal includes vertical high frequency components of said decompressed frame;

second limiting an input signal level below said temporal threshold THD_TP, wherein the input signal includes horizontal high frequency components of said decompressed frame;

horizontal low pass filtering of a signal produced by said first limiting;

vertical low pass filtering of a signal produced by said second limiting; and

adjusting texture sharpness of the sum of the outputs of said filtering steps according to said temporal texture coefficient K^TP_text.

46. The method of claim 45 and wherein said adjusting comprises:

generating a small details signal from said vertical and horizontal high frequency components and the outputs of said steps of limiting; and

adjusting a sharpness of said small details signal according to said temporal contrast coefficient K^TP_cont.

47. The method of claim 45 and wherein said reducing blockness comprises reducing the level of low frequency components of said decompressed video frame according to said blockness coefficient K_F.

48. The method of claim 42, wherein said spatially adapting comprises:

performing spatial processing on said decompressed video frame;

adjusting textual elements in said frame;

sharpening small details in said frame; and

reducing ringing effects in said frame.

49. The method of claim 48, wherein said reducing comprises:

first limiting an input signal level below said spatial threshold THD_SP, wherein the input signal includes vertical high frequency components of said decompressed frame;

second limiting an input signal level below said spatial threshold THD_SP, wherein the input signal includes horizontal high frequency components of said decompressed frame; and

reducing ringing effects in high pass filtered signals according to said ringing coefficient K_ring(i,j).

50. The method of claim 48 wherein said sharpening comprises adjusting a sharpness of said small details signal according to said spatial-temporal contrast coefficient K^SP_cont(i,j).

51. The method of claim 48 and wherein said adjusting comprises:

a temporally low pass filtering to reduce noise according to said recursion coefficient K_rec; and

adjusting texture sharpness of an output of said filtering according to a value of said spatial-temporal texture coefficient K^SP_text.

52. The method of claim 39, wherein said temporally adapting comprises an infinite impulse response (IIR) filtering using an adaptive recursion coefficient determined at least from said estimated distortion level DR.

53. A computer product readable by a machine, tangibly embodying a program of instructions executable by the machine to implement a process, said process comprising:

estimating visual quality parameters that indicate the visual quality of decompressed video frames; and

substantially improving the visual quality of said decompressed video frames using their associated parameters.

54. The computer software product of claim 53, wherein said estimating comprises:

estimating a distortion level DR in a frame, an image complexity value NC, a per pixel intensity change h_ij and a high contrast small details level CH in a frame.

55. The computer software product of claim 54, wherein said estimating comprises determining a blockness level.

56. The computer software product of claim 54, wherein said estimating comprises:

generating a per-pixel intensity change h_i,j; and

determining a per-frame image complexity level NC.

57. The computer software product of claim 54 wherein said substantially improving comprises:

temporally adapting processing of each said decompressed video frame with at least one of said associated parameters; and

spatially adapting processing of the output of said temporal adapting with at least one of said associated parameters.

58. The computer software product of claim 57, wherein said temporally adapting comprises:

temporally processing said decompressed video frame;

first adjusting the sharpness of textual elements in said frame;

second adjusting the sharpness of small details in said frame; and

reducing blockness in said frame by attenuating low frequency components of said frame received from said temporal processing.

59. The computer software product of claim 57, wherein said spatially adapting comprises:

performing spatial processing on said decompressed video frame;

adjusting textual elements in said frame;

sharpening small details in said frame; and

reducing ringing effects in said frame.

60. The computer product of claim 57, wherein said temporally adapting comprises an infinite impulse response (IIR) filtering using an adaptive recursion coefficient determined at least from said estimated distortion level DR.