Method and apparatus for use in transmitting video information over a communication network

• US 6,754,266 B2
• Filed: 10/09/1998
• Issued: 06/22/2004
• Est. Priority Date: 10/09/1998
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hsu, Wei-Lien, Bahl, Paramvir
• Primary Examiner(s): VO, TUNG T
• Application Number:
  US09/169,724
• Publication Number:
  US 20010016008A1
• Time in Patent Office: 2,083 Days
• Field of Search: H04/B.166, 348/384.1, 348/394.1, 348/395.1, 348/400--42, 382/232, 382/233, 382/234, 382/238, 382/240, 375/240, 375/240.2, 375/240.01, 375/240.012, 375/240.25
• US Class Current: 375/240.01
• CPC Class Codes: H04N 19/00 Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets
• CPC Class Codes: H04N 19/649 the transform being applied to non rectangular image segments
• CPC Class Codes: H04N 19/65 using error resilience
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 21/23655 Statistical multiplexing, e.g. by controlling the encoder to alter its bitrate to optimize the bandwidth utilization

Abstract

An adaptive region-based, multi-scale, motion compensated video compression algorithm design for transmission over hostile communication channels. The algorithm is embodied in a video encoder that extracts spatial information from video frames to create video regions that are then decomposed into sub-bands of different perceptual importance before being compressed and transmitted independently. The system further uses unequal error protection, prioritized transmission and reconstruction to guarantee a minimum spatial and temporal resolution at the receiver. In particular, the region segmented frames bound both spatial and temporal error propagation within frames. A connection-level inter-region statistical multiplexing scheme is also employed to ensure optimal utilization of reserved transmission bandwidth.

Claims

1. A method of generating information indicative of a video frame for subsequent transmission over a communication network, the method comprising the steps of:

segmenting a video frame into a plurality of discrete spatial components;

transforming each of the plurality of discrete spatial components into a plurality of discrete frequency components; and

assigning a transmission priority to each of the plurality of discrete frequency components that determines the order in which the plurality of discrete frequency components are to be transmitted over the communications network;

whereby information indicative of the video frame is generated for subsequent transmission over the communications network.

2. The method as recited in claim 1 further comprising the step of compressing the information indicative of the video frame before it is transmitted over the communication network.

3. The method as recited in claim 2 wherein the step of compressing the information indicative of the video frame further comprises the step of using an adaptive quantization algorithm.

4. The method as recited in claim 3 further comprising the step of bit stream packing the information indicative of the video frame after the step of using the quantization algorithm.

5. The method as recited in claim 4 wherein the step of bit stream packing the information indicative of the video frame further comprises the step of using a run length encoding algorithm.

6. The method as recited in claim 1 wherein the step of transforming each of the plurality of discrete spatial components further comprises the step of decomposing each of the plurality of discrete spatial components into a plurality of subbands each having a different level of resolution.

7. The method as recited in claim 6 wherein the step of transforming each of the plurality of discrete spatial components further comprises the step of decorrelating each of the plurality of subbands.

8. The method as recited in claim 7 wherein the step of decorrelating each of the plurality of subbands further comprises the step of using a Discrete Cosine Transform algorithm.

9. The method as recited in claim 6 wherein the step of decomposing each of the plurality of discrete spatial components further comprises the step of using a Harr filter.

10. The method as recited in claim 9 wherein the Harr filter comprises a two-tap Harr filter for decomposing each of the plurality of discrete spatial components into four frequency bands including a DC subband and three AC subbands.

11. The method as recited in claim 10 further comprising the step of assigning a transmission priority to each of the decomposed plurality of discrete spatial components that determines the order in which the decomposed plurality of discrete spatial components are to be transmitted over the communications network.

12. The method as recited in claim 10 wherein the step of assigning a transmission priority further comprises the step of assigning the highest transmission priority to the DC subbands.

13. The method as recited in claim 1 wherein the step of segmenting a video frame into a plurality of discrete spatial components is performed using a static segmentation map.

14. The method as recited in claim 1 wherein the step of segmenting a video frame into a plurality of discrete spatial components further comprises the steps of partitioning the video frame into blocks of a predetermined size, determining the variance of pixel intensity for each block and merging adjacent blocks that have a pixel intensity within a predetermined threshold.

15. A computer-readable medium having computer executable instructions for generating information indicative of a video frame, the instructions performing steps comprising:

segmenting the video frame into a plurality of discrete spatial components;

transforming each of the plurality of discrete spatial components into a plurality of discrete frequency components; and

assigning a transmission priority to each component of the decomposed plurality of discrete spatial components that determines the order in which the components of the decomposed plurality of discrete spatial components are to be transmitted over a communications network.

16. The computer-readable medium as recited in claim 15 further comprising instructions for performing the step of compressing the discrete frequency components.

17. The computer-readable medium as recited in claim 16 wherein the instructions for performing the step of compressing the discrete frequency components perform the step using an adaptive quantization algorithm.

18. The computer-readable medium as recited in claim 17 wherein the instructions for performing the step of compressing the discrete frequency components further perform the step of bit stream packing the compressed discrete frequency components.

19. The computer-readable medium as recited in claim 18 wherein the instructions for performing the step of bit stream packing the compressed discrete frequency components perform the step using a run length encoding algorithm.

20. The computer-readable medium as recited in claim 15 wherein the instructions for performing the step of transforming each of the plurality of discrete spatial components further comprises instructions for performing the step of decomposing each of the plurality of discrete spatial components into a plurality of subbands each having a different level of resolution.

21. The computer-readable medium as recited in claim 20 wherein the instructions for performing the step of transforming each of the plurality of discrete spatial components further comprises instructions for performing the step of decorrelating each of the plurality of subbands.

22. The computer-readable medium as recited in claim 21 wherein the instructions for performing the step of decorrelating each of the plurality of subbands perform the step using a Discrete Cosine Transform algorithm.

23. The computer-readable medium as recited in claim 20 wherein the instructions for performing the step of decomposing each of the plurality of discrete spatial components perform the step using a Harr filter.

24. The computer-readable medium as recited in claim 23 wherein the Harr filter comprises a two-tap Harr filter for decomposing each of the plurality of discrete spatial components into four frequency bands including a DC subband and three AC subbands.

25. The computer-readable medium as recited in claim 24 wherein the instructions for performing the step of assigning a transmission priority further comprises instruction for performing the step of assigning the highest transmission priority to the DC subbands.

26. The computer-readable medium as recited in claim 15 wherein the instructions for performing the step of segmenting the video frame into a plurality of discrete spatial components is performed using a static segmentation map.

27. The computer-readable medium as recited in claim 15 wherein the instructions for performing the step of segmenting the video frame into a plurality of discrete spatial components further comprises instructions for performing the steps of partitioning the video frame into blocks of a predetermined size, determining the variance of pixel intensity for each block and merging adjacent blocks that have a pixel intensity within a predetermined threshold.

28. In a communications network comprising a communication source and a communication receiver, a method for transmitting video information comprising:

performing at the communication source the steps of;

segmenting a video frame into a plurality of discrete spatial components;

transforming each of the plurality of discrete spatial components into a plurality of discrete frequency components;

assigning a transmission priority to each of the plurality of discrete frequency components that determines the order in which the plurality of discrete frequency components are to be transmitted over the communications network; and

transmitting at least one of the plurality of discrete frequency components over the communications network; and

performing at the communication receiver the steps of;

receiving the transmitted at least one of the plurality of discrete frequency components; and

decoding the received at least one of the discrete frequency components whereby the decoded frequency components are available to be viewed on a video display.

29. The method as recited in claim 28 further comprising the step of compressing at the communication source the information indicative of the video frame before it is transmitted over the communication network.

30. The method as recited in claim 29 wherein the step of compressing the information indicative of the video frame further comprises the step of using an adaptive quantization algorithm.

31. The method as recited in claim 30 further comprising the step of bit stream packing at the communication source the information indicative of the video frame after the step of using the quantization algorithm.

32. The method as recited in claim 31 wherein the step of bit stream packing the information indicative of the video frame further comprises the step of using a run length encoding algorithm.

33. The method as recited in claim 32 wherein the step of transforming each of the plurality of discrete spatial components further comprises the step of decomposing each of the plurality of discrete spatial components into a plurality of subbands each having a different level of resolution.

34. The method as recited in claim 33 wherein the step of transforming each of the plurality of discrete spatial components further comprises the step of decorrelating each of the plurality of subbands.

35. The method as recited in claim 34 wherein the step of decorrelating each of the plurality of subbands further comprises the step of using a Discrete Cosine Transform algorithm.

36. The method as recited in claim 33 wherein the step of decomposing each of the plurality of discrete spatial components further comprises the step of using a Harr filter.

37. The method as recited in claim 36 wherein the Harr filter comprises a two-tap Harr filter for decomposing each of the plurality of discrete spatial components into four frequency bands including a DC subband and three AC subbands.

38. The method as recited in claim 37 further comprising the step of assigning a transmission priority to each of the decomposed plurality of discrete spatial components that determines the order in which the decomposed plurality of discrete spatial components are to be transmitted over the communications network.

39. The method as recited in claim 37 wherein the step of assigning a transmission priority further comprises the step of assigning the highest transmission priority to the DC subbands.

40. The method as recited in claim 28 wherein the step of segmenting a video frame into a plurality of discrete spatial components is performed using a static segmentation map.

41. The method as recited in claim 28 wherein the step of segmenting a video frame into a plurality of discrete spatial components further comprises the steps of partitioning the video frame into blocks of a predetermined size, determining the variance of pixel intensity for each block and merging adjacent blocks that have a pixel intensity within a predetermined threshold.

42. The method as recited in claim 28 further comprising the step of reconstructing at the communication receiver the video frame received from the communication source.

43. The method as recited in claim 42, wherein the step of reconstructing the video frame further comprises the steps of receiving from the communication source a plurality of discrete frequency components representative of a first video frame, receiving from the communication source a plurality of discrete frequency components representative of a second video frame, determining if a transmission error is associated with any of the plurality of discrete frequency components representative of the second video frame and substituting for each of the plurality of discrete frequency components representative of the second video frame determined to be in error a corresponding discrete frequency component representative of the first video frame.

44. The method as recited in claim 28, further comprising the step of using at the communication source a statistical multiplexing algorithm to improve the efficiency of transmission.

45. A video encoder for use in generating information representative of a video frame for use in providing visual communications over a communication network, the video encoder comprising:

a spatial segmenter for segmenting a digitally represented video frame into a plurality of discrete digitally represented spatial components; and

a frequency segmenter in communication with the spatial segmenter for transforming each of the plurality of discrete digitally represented spatial components into a plurality of discrete digitally represented frequency components, and for assigning a transmission priority to each of the discrete digitally represented frequency components

whereby the plurality of discrete digitally represented frequency components comprise the information representative of the video frame for use in providing the visual communications over the communication network.

46. The video encoder as recited in claim 45 further comprising a compressor in communication with the frequency segmenter for compressing each of the discrete digitally represented frequency components whereby the information representative of the video frame may be more efficiently transmitted over the communication network.

47. The video encoder as recited in claim 46 wherein the compressor comprises a quantizer.

48. The video encoder as recited in claim 47 further comprising a bit stream packer in communication with the quantizer for reducing the amount of bits in each of the compressed, discrete digitally represented frequency components whereby the information representative of the video frame may be still more efficiently transmitted over the communication network.

49. The video encoder as recited in claim 48 wherein the bit stream packer comprises a run length encoder.

50. The video encoder as recited in claim 45 wherein the frequency segmenter comprises a decomposer for decomposing each of the plurality of discrete spatial components into a plurality of subbands each having a different level of resolution.

51. The video encoder as recited in claim 50 wherein the decomposer comprises a Harr filter.

52. The video encoder as recited in claim 51 wherein the Harr filter comprises a two-tap Harr filter for decomposing each of the plurality of discrete spatial components into four frequency bands including a DC subband and three AC subbands.

53. The video encoder as recited in claim 50 wherein the frequency segmenter further comprises a Discrete Cosine Transformer in communication with the decomposer.

54. The video encoder as recited in claim 45 wherein the spatial segmenter utilizes a static segmentation map.