Pre-processing method and system for data reduction of video sequences and bit rate reduction of compressed video sequences using spatial filtering

• US 20050036704A1
• Filed: 08/13/2003
• Published: 02/17/2005
• Est. Priority Date: 08/13/2003
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Dumitras, Adriana, Salsbury, Ryan R., Normile, James Oliver
• Granted Patent: US 7,430,335 B2
• Time in Patent Office: Days
• US Class Current: 382/260
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/136 Incoming video signal characteristics or properties
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/192 the adaptation method, adaptation tool or adaptation type being iterative or recursive
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence
• CPC Class Codes: H04N 5/21 Circuitry for suppressing or minimising disturbance, e.g. moiré or halo
• CPC Class Codes: H04N 9/8042 involving data reduction

Abstract

Methods for pre-processing video sequences prior to compression to provide data reduction of the video sequence. Also, after compression of the pre-processed video sequence, the bit rate of the pre-processed and compressed video sequence will be lower than the bit rate of the video sequence after compression but without pre-processing. Pre-processing may include spatial anisotropic diffusion filtering such as Perona-Malik filtering, Fallah-Ford filtering, or omni-directional filtering that extends Perona-Malik filtering to perform filtering in at least one diagonal direction. Pre-processing may also include performing filtering differently on a foreground region than on a background region of a video frame. This method includes identifying pixel locations having pixel values matching characteristics of human skin and determining a bounding shape for each contiguous grouping of matching pixel locations. The foreground region is comprised of pixel locations contained in a bounding shape and the background region is comprised of all other pixel locations.

Claims

1. A method of pre-filtering an original video sequence, the method comprising:

a) receiving the original video sequence;

b) pre-filtering the original video sequence using anisotropic diffusion filtering;

c) outputting a pre-filtered video sequence; and

d) compressing the pre-filtered video sequence using a first compression method to produce a pre-filtered and compressed video sequence.

2. The method of claim 1 wherein the anisotropic diffusion filtering is Fallah-Ford diffusion filtering.

3. The method of claim 1 wherein the anisotropic diffusion filtering is Perona-Malik diffusion filtering.

4. The method of claim 1 wherein the anisotropic diffusion filtering performs diffusion filtering in at least one diagonal direction with respect to a pixel location of a video frame of the original video sequence.

5. A method of pre-processing a video frame having a plurality of pixels at pixel locations where each pixel location contains one or more pixel values, the method comprising:

performing anisotropic diffusion filtering on a pixel in the plurality of pixels, the pixel having a pixel location wherein the diffusion filtering is performed in at least one diagonal direction with respect to the pixel location.

6. The method of claim 5 wherein the diffusion filtering is performed in four horizontal or vertical directions (north, south, east and west) and four diagonal directions (north-east, north-west, south-east, and south-west) with respect to the pixel location.

7. The method of claim 6 wherein the diffusion filtering is expressed by the equation:

$I<br><br>\left(x,y,t+1\right)=I<br><br>\left(x,y,t\right)+\mathrm{\lambda <br><br>}<br><br>\left[\underset{N,S,E,W}{\overset{\text{\hspace{1em}<br><br>}}{\sum <br><br>}}<br><br>c_m<br><br>\left(x,y,t\right)<br><br>\nabla <br><br>\text{\hspace{1em}<br><br>}<br><br>I_m<br><br>\left(x,y,t\right)+\mathrm{\beta <br><br>}<br><br>\underset{\mathrm{NE},\mathrm{SE},\mathrm{SW},\mathrm{NW}}{\overset{\text{\hspace{1em}<br><br>}}{\sum <br><br>}}<br><br>c_n<br><br>\left(x,y,t\right)<br><br>\nabla <br><br>\text{\hspace{1em}<br><br>}<br><br>I_n<br><br>\left(x,y,t\right)\right]$

where;

I(x, y, t) is a discrete image;

∇

I(x, y, t) is the image gradient;

(x, y) specifies the pixel location in a discrete, two dimensional grid covering the video frame;

t denotes discrete time steps;

scalar constant λ

determines a rate of diffusion;

subscripts m and n correspond to eight directions of diffusion with respect to the pixel location (x, y), where m is a horizontal or vertical direction (north, south, east and west) and n is a diagonal direction (north-east, north-west, south-east, and south-west); and

notations c_m, and c_n are diffusion coefficients where horizontal or vertical directions (north, south, east and west) are indexed by subscript m and diagonal directions (north-east, north-west, south-east, and south-west) are indexed by subscript n, each diffusion coefficient being referred to as an edge stopping function g(x) of ∇

I(x,y,t) in the corresponding direction as expressed in the following equations;

c_m(x, y, t)=g(∇

I_m(x, y, t))
c_n(x, y, t)=g(∇

I_n(x, y, t))

where g(x) satisfies the condition g(x)→

0 when x→

∞

such that the diffusion filtering is stopped across edges that are present in the video frame.

8. A method of pre-filtering an original video sequence, the original video sequence being comprised of a plurality of frames, each frame having a plurality of pixel locations where each pixel location contains a pixel value, the method comprising:

a) setting a current frame of the original video sequence;

b) identifying a region-of-interest in the current frame;

c) determining a bounding shape that encloses all or a portion of the region-of-interest; and

d) filtering pixel locations in the bounding shape differently than other pixel locations in the current frame.

9. The method of claim 8 wherein the bounding shape has a four-sided, three-sided, or circular form.

10. The method of claim 8 wherein each pixel location in the region-of-interest has a chrominance value within a predetermined low chrominance threshold value and a predetermined high chrominance threshold value.

11. The method of claim 8 wherein the bounding shape encloses over ½

or ¾

of the pixel locations in the region-of-interest.

12. The method of claim 8 wherein:

the region of the current frame within the bounding shape is referred to as a foreground region; and

the region of the current frame not within the bounding shape is referred to as a background region.

13. The method of claim 12 wherein the filtering comprises applying anisotropic diffusion to the pixel locations in the foreground and background regions where different parameter values are used for the foreground and background regions.

14. The method of claim 12 wherein the filtering comprises applying anisotropic diffusion to the pixel locations in the foreground and background regions where different edge stopping functions are used for the foreground and background regions.

15. The method of claim 12 wherein the filtering comprises applying Fallah-Ford diffusion filtering to the pixel locations in the foreground region differently than to the pixel locations in the background region.

16. The method of claim 12 wherein the filtering comprises applying Perona-Malik diffusion filtering to the pixel locations in the foreground region differently than to the pixel locations in the background region.

17. The method of claim 12 wherein the filtering comprises applying anisotropic diffusion to the pixel locations in the foreground region differently than to the pixel locations in the background region, the anisotropic diffusion performing filtering on a pixel location in at least one diagonal direction with respect to the pixel location.

18. The method of claim 8 further comprising:

e) setting a next frame in the original video sequence as the current frame;

f) repeating steps b), c), and d) for each frame of the plurality of frames in the original video sequence;

g) outputting a pre-filtered video sequence comprised of a plurality of pre-filtered video frames; and

h) compressing the pre-filtered video sequence using a first compression method to produce a pre-filtered and compressed video sequence, wherein a bit rate associated with the pre-filtered and compressed video sequence is lower than a bit rate that would result from compressing the original video sequence using the first compression method without performing steps a) through f).

19. A method of pre-filtering a video frame having a plurality of pixel locations where each pixel location contains a pixel value and is identifiable by pixel location coordinates, the method comprising:

identifying matching pixel locations in the video frame containing a chrominance (U) value within a predetermined low chrominance (U) threshold value and a predetermined high chrominance (U) threshold value and a chrominance (V) value within a predetermined low chrominance (V) threshold value and a predetermined high chrominance (V) threshold value, the matching pixel locations comprising a set of matching pixel locations;

identifying at least one contiguous grouping of matching pixel locations in the set of matching pixel locations, the contiguous grouping of matching pixel locations being referred to as a region-of-interest;

determining a bounding shape for each region-of-interest, the bounding shape enclosing all or a portion of the region-of-interest;

determining a foreground region of the video frame, the foreground region being comprised of a totality of pixel locations in the video frame enclosed within a bounding shape;

determining a background region of the video frame, the background region being comprised of a totality of pixel locations in the video frame not enclosed within a bounding shape; and

filtering the pixel locations in the foreground region differently than the pixel locations in the background region.

20. The method of claim 19 wherein the low and high chrominance (U) threshold values and the low and high chrominance (V) threshold values reflect the chrominance (U, V) ranges of human skin.

21. The method of claim 19 wherein the filtering comprises applying anisotropic diffusion to the pixel locations in the foreground region differently than to the pixel locations in the background region.

22. A computer program product having a computer readable medium having computer program instructions recorded thereon, the computer program product comprising:

instructions for receiving the original video sequence;

instructions for pre-filtering the original video sequence using anisotropic diffusion filtering;

instructions for outputting a pre-filtered video sequence; and

instructions for compressing the pre-filtered video sequence.

23. The computer program product of claim 22 wherein the anisotropic diffusion filtering is Fallah-Ford diffusion filtering.

24. The computer program product of claim 22 wherein the anisotropic diffusion filtering is Perona-Malik diffusion filtering.

25. The computer program product of claim 22 wherein the anisotropic diffusion filtering performs diffusion filtering in at least one diagonal direction with respect to a pixel location of a video frame of the original video sequence.

26. A computer program product having a computer readable medium having computer program instructions recorded thereon for pre-processing a video frame having a plurality of pixels at pixel locations where each pixel location contains one or more pixel values, the computer program product comprising:

instructions for performing anisotropic diffusion filtering on a pixel in the plurality of pixels, the pixel having a pixel location wherein the diffusion filtering is performed in at least one diagonal direction with respect to the pixel location.

27. The computer program product of claim 26 wherein the diffusion filtering is performed in four horizontal or vertical directions (north, south, east and west) and four diagonal directions (north-east, north-west, south-east, and south-west) with respect to the pixel location.

28. A computer program product having a computer readable medium having computer program instructions recorded thereon for pre-filtering an original video sequence, the original video sequence being comprised of a plurality of frames, each frame having a plurality of pixel locations where each pixel location contains a pixel value, the computer program product comprising:

instructions for setting a current frame of the original video sequence;

instructions for identifying a region-of-interest in the current frame;

instructions for determining a bounding shape that encloses all or a portion of the region-of-interest; and

instructions for filtering pixel locations in the bounding shape differently than other pixel locations in the current frame.

29. The computer program product of claim 28 wherein each pixel location in the region-of-interest has a chrominance value within a predetermined low chrominance threshold value and a predetermined high chrominance threshold value.

30. The computer program product of claim 28 wherein the filtering comprises applying anisotropic diffusion to the pixel locations in the bounding shape differently than other pixel locations in the current frame.

31. A system for processing an original video sequence, the original video sequence being comprised of a plurality of frames, each frame having a plurality of pixel locations where each pixel location contains a pixel value, the system comprising:

a pre-processing component that;

receives the original video sequence;

pre-filters the original video sequence using anisotropic diffusion filtering; and

outputs a pre-filtered video sequence; and

an encoder component coupled to pre-processing component, wherein the encoder component compresses the pre-filtered video sequence.

32. The system of claim 31 wherein the anisotropic diffusion filtering is Fallah-Ford diffusion filtering.

33. The system of claim 31 wherein the anisotropic diffusion filtering is Perona-Malik diffusion filtering.

34. The system of claim 31 wherein the anisotropic diffusion filtering performs diffusion filtering in at least one diagonal direction with respect to a pixel location of a video frame of the original video sequence.

35. A system for pre-processing a video frame having a plurality of pixels at pixel locations where each pixel location contains one or more pixel values, the system comprising:

a pre-processing component that performs anisotropic diffusion filtering on a pixel in the plurality of pixels, the pixel having a pixel location wherein the diffusion filtering is performed in at least one diagonal direction with respect to the pixel location.

36. The system of claim 35 wherein the diffusion filtering is performed in four horizontal or vertical directions (north, south, east and west) and four diagonal directions (north-east, north-west, south-east, and south-west) with respect to the pixel location.

37. A system for pre-filtering an original video sequence, the original video sequence being comprised of a plurality of frames, each frame having a plurality of pixel locations where each pixel location contains a pixel value, the system comprising:

a pre-processing component that;

sets a current frame of the original video sequence;

identifies a region-of-interest in the current frame;

determines a bounding shape that encloses all or a portion of the region-of-interest; and

filters pixel locations in the bounding shape differently than other pixel locations in the current frame.

38. The system of claim 37 wherein each pixel location in the region-of-interest has a chrominance value within a predetermined low chrominance threshold value and a predetermined high chrominance threshold value.

39. The system of claim 37 wherein the filtering comprises applying anisotropic diffusion to the pixel locations in the bounding shape differently than other pixel locations in the current frame.

40. A system for pre-processing a video frame having a plurality of pixels at pixel locations where each pixel location contains one or more pixel values, the system comprising:

means for performing anisotropic diffusion filtering on a pixel in the plurality of pixels, the pixel having a pixel location wherein the diffusion filtering is performed in at least one diagonal direction with respect to the pixel location.

41. The system of claim 40 wherein the diffusion filtering is performed in four horizontal or vertical directions (north, south, east and west) and four diagonal directions (north-east, north-west, south-east, and south-west) with respect to the pixel location.

42. A system for pre-filtering an original video sequence, the original video sequence being comprised of a plurality of frames, each frame having a plurality of pixel locations where each pixel location contains a pixel value, the system comprising:

means for setting a current frame of the original video sequence;

means for identifying a region-of-interest in the current frame;

means for determining a bounding shape that encloses all or a portion of the region-of-interest; and

means for filtering pixel locations in the bounding shape differently than other pixel locations in the current frame.

43. The system of claim 42 wherein the filtering comprises applying anisotropic diffusion to the pixel locations in the bounding shape differently than other pixel locations in the current frame.