System and method for layered video coding enhancement

• US 6,510,177 B1
• Filed: 03/24/2000
• Issued: 01/21/2003
• Est. Priority Date: 03/24/2000
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: De Bonet, Jeremy, Sullivan, Gary J.
• Primary Examiner(s): RAO, ANAND SHASHIKANT
• Application Number:
  US09/535,159
• Time in Patent Office: 1,033 Days
• Field of Search: 375/240-8-2
• US Class Current: 375/240.16
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/52 by predictive encoding
• CPC Class Codes: H04N 19/86 involving reduction of coding artifacts, e.g. of blockiness

Abstract

A system and a method for encoding, transmitting, decoding and storing a high-resolution video sequence using a low-resolution base layer and a higher-resolution enhancement layer. The base layer contains low-resolution video information and is compressed in a manner such that it can be decoded independently of the enhancement layer. The base layer may be encoded in an existing standard video format (such as MPEG-2). The enhancement layer, which contains high-resolution information, provides a variable amount of enhancement to the base layer. The compressed video sequence is transmitted in two layers and uses a minimum amount of bit rate by using information transmitted in the base layer in the decoding of both the base layer and the enhancement layer. In an enhanced decoder, both the base layer and the enhancement layer use this information that has been transmitted in the base layer.

Claims

1. A method for encoding a high-resolution video sequence containing low-resolution and high-resolution information, comprising:

encoding the low-resolution information to create a base layer;

encoding the high-resolution information to create an enhancement layer capable of enhancing the low-resolution information of the base layer;

encoding base layer data within the base layer; and

making the encoded base layer data available to both a base layer decoder and an enhancement layer decoder.

2. The method of claim 1, wherein the base layer is capable of being decoded independently of the enhancement layer.

3. The method of claim 2, wherein the base layer is capable of being decoded using a standard decoding format.

4. The method of claim 1, wherein the base layer data includes at least one of:

(a) motion vector data;

(b) macroblock prediction mode data;

(c) quantized residual difference data including quantized discrete transform coefficient data;

(d) quantized residual difference data including quantized wavelet coefficient data.

5. The method of claim 1, wherein the encoding of the low-resolution information uses an existing standard encoding format.

6. The method of claim 5, wherein the existing standard encoding format is at least one of:

(a) H.261;

(b) H.263;

(c) MPEG-1;

(d) MPEG-2;

(e) MPEG-4.

7. The method of claim 1, further comprising generating base layer data by performing high-resolution motion estimation.

8. The method of claim 7, wherein the high-resolution motion estimation is performed at approximately the same resolution as the enhancement layer.

9. The method of claim 7, wherein the high-resolution motion estimation includes a high-resolution block matching search.

10. The method of claim 9, wherein motion vector estimates obtained from the block matching search are scaled down to apply at the lower resolution and rounded to an accuracy supported by an encoding format of the base layer.

11. The method of claim 10, wherein at least one motion vector is selected from the rounded motion vector estimates based on a fidelity criterion to adjust a first block size used for the high-resolution motion estimation to a second block size used for low-resolution motion representation.

12. The method of claim 9, wherein motion vector estimates obtained from the block matching search are scaled down to apply at the lower resolution and further processed by a local block matching search to an accuracy supported by an encoding format of the base layer.

13. The method of claim 1, wherein encoding the high-resolution information includes performing motion compensation using motion vector values obtained by scaling up base layer motion vectors that are used to predict high-resolution motion vector values.

14. The method of claim 13, wherein no motion vector values are contained in the enhancement layer.

15. The method of claim 13, wherein any prediction differences are contained in the enhancement layer.

16. The method of claim 13, further comprising selecting at least one of the scaled-up base layer motion vectors to adjust a first block size used for motion compensation in the base layer to a second block size used for motion compensation in the enhancement layer.

17. The method of claim 15, further comprising:

selecting at least one of the high-resolution motion vectors to adjust a first block size used for motion compensation in the base layer to a second block size used for motion compensation in the enhancement layer; and

predicting the high-resolution motion vectors using at least one of;

(a) the scaled-up base layer motion vectors;

(b) previously-encoded motion vectors contained within the enhancement layer.

18. The method of claim 1, wherein the base layer is created using overlapped block motion compensation.

19. The method of claim 1, wherein the enhancement layer is created using overlapped block motion compensation.

20. The method of claim 1, wherein encoding of the high-resolution information uses at least one of:

(a) discrete cosine transform;

(b) scalar inverse quantization;

(c) wavelet coding;

(d) progressive wavelet coding.

21. A computer-readable medium having instructions stored thereon that causes a computer to perform the method of claim 1.

22. A method of compressing high-resolution video frames containing low-resolution frames and high-resolution frames, comprising:

generating a base layer containing the low-resolution frames that is capable of being decompressed into a low-resolution video representation;

generating an enhancement layer containing the high-resolution frames that is capable of being decompressed into a high-resolution video representation;

placing compressed base layer data into the base layer; and

providing the compressed base layer data to a base layer decompressor and an enhancement layer decompressor.

23. The method of claim 22, wherein the base layer data includes at least one of:

(a) motion vector data;

(b) macroblock prediction mode data;

(c) quantized residual difference data.

24. The method of claim 22, wherein the base layer data is also capable of being used in the decompression of the base layer.

25. A method of decoding an encoded high-resolution video sequence containing a base layer and an enhancement layer, comprising:

decoding the base layer to generate a decoded low-resolution representation of the high-resolution video sequence;

extracting encoded base layer data from the base layer; and

decoding the enhancement layer using the encoded base layer data and the decoded low-resolution representation to provide enhancement to the decoded low-resolution representation.

26. The method of claim 25, wherein the base layer data includes motion vector data that was encoded within the base layer.

27. The method of claim 26, wherein the base layer data includes at least one of:

(a) macroblock prediction mode data;

(b) quantized residual difference data;

(c) quantized discrete cosine transform coefficient data.

28. The method of claim 26, wherein the base layer data includes quantized residual difference data and quantized wavelet coefficient data.

29. The method of claim 28, wherein a residual encoding format for the enhancement layer uses at least one of:

(a) discrete cosine transform;

(b) scalar inverse quantization;

(c) wavelet coding;

(d) progressive wavelet coding.

30. The method of claim 25, wherein the base layer is decoded independently of the enhancement layer.

31. The method of claim 30, wherein a motion compensation process for the enhancement layer uses motion vectors obtained by scaling up base layer motion vectors and rounding them to an accuracy appropriate for use in the enhancement layer.

32. The method of claim 25, wherein the enhancement layer is decoded using base layer motion vectors to predict enhancement layer motion vectors.

33. The method of claim 32, wherein the enhancement layer motion vectors are obtained by scaling up the base layer motion vectors and rounding them to an accuracy suitable for use with the enhancement layer.

34. The method of claim 33, wherein the base layer data includes at least one of the following:

(a) information used to select from a plurality of predicted enhancement layer motion vectors so as to adjust a first block size in a motion compensation process;

(b) information used to determine whether to perform a prediction of the enhancement layer motion vectors using scaled-up base layer motion vectors;

(c) information used to determine whether to perform a prediction of the enhancement layer motion vectors using prior-encoded motion vectors within the enhancement layer.

35. The method of claim 25, wherein the base layer was encoded using overlapped block motion compensation.

36. The method of claim 25, wherein the enhancement layer was encoded using overlapped block motion compensation.

37. The method of claim 25, wherein the base layer was previously encoded using a standard compression format.

38. The method of claim 37, wherein the standard compression format is at least one of:

(a) H.261;

(b) H.263;

(c) MPEG-1;

(d) MPEG-2;

(e) MPEG-4.

39. A computer-readable medium having instructions stored thereon that causes a computer to perform the method of claim 25.

40. A layered video encoder for encoding a high-resolution video sequence, comprising:

a base layer module that encodes at least a portion of a low-resolution video sequence to create a base layer, the low-resolution video sequence obtained from the high-resolution video sequence;

an enhancement layer module that encodes at least a portion of the high-resolution video sequence to create an enhancement layer that enhances the base layer; and

encoded base layer data that is used by an enhancement layer decoder to decode the enhancement layer.

41. The layered video encoder of claim 40, wherein the base layer is capable of being decoded independently of the enhancement layer.

42. The layered video encoder of claim 41, wherein the encoded base layer data can be used to decode both the base layer and the enhancement layer.

43. The layered video encoder of claim 40, wherein the base layer module further comprises a downsample module that generates downsampled low-resolution motion vectors from high-resolution motion vectors, the high-resolution motion vectors being calculated from the high-resolution video sequence.

44. A computer-readable medium having computer-executable modules, comprising:

a base layer module that creates a low-resolution video sequence from a high-resolution video sequence and generates a base layer from the low-resolution video sequence;

an enhancement layer module that encodes at least a portion of the high-resolution video sequence to generate an enhancement layer; and

encoded base layer data that is contained in the base layer and is used by a decoder to decode the enhancement layer.

45. A layered video decoder that decodes an encoded high-resolution video sequence containing a base layer and an enhancement layer, comprising:

a base layer decoder module that decodes the base layer to produce a decoded low-resolution representation of the video sequence;

an enhancement layer decoder module that decodes the enhancement layer to produce a high-resolution enhancement to the decoded low-resolution representation; and

motion vector data that is used along with the decoded low-resolution representation to decode the enhancement layer.

46. The layered video decoder of claim 45, wherein the high-resolution enhancement can be varied to produce a desired enhancement to the low-resolution representation.

47. The layered video decoder of claim 45, wherein the base layer data is encoded in the base layer.

48. The layered video decoder of claim 45, wherein the enhancement layer decoder module includes a P-frame decoder module capable of determining at least one of:

(a) whether to predict enhancement layer motion vectors using base layer motion vectors;

(b) whether to predict enhancement layer motion vectors using prior-encoded enhancement layer motion vectors.

49. The method of claim 45, wherein the base layer decoder module decodes the base layer independently of the enhancement layer.

50. The method of claim 45, wherein the enhancement layer decoder module uses base layer motion vectors to predict enhancement layer motion vectors.

51. The method of claim 45, wherein the base layer was encoded using overlapped block motion compensation.

52. The method of claim 45, wherein the enhancement layer was encoded using overlapped block motion compensation.

53. A method for encoding a high-resolution video sequence, comprising:

computing a set of high-resolution motion vectors corresponding to the high-resolution video sequence;

downsampling the set of high-resolution motion vectors to generate a set of downsampled motion vectors;

downsampling the high-resolution video sequence to create a low-resolution video sequence; and

generating a base layer by encoding the low-resolution video sequence using the downsampled motion vectors.

54. A method for encoding a high-resolution video sequence having low-resolution information and high-resolution information, comprising:

performing high-resolution motion estimation on the high-resolution video sequence to generate motion estimation results;

creating a base layer containing the low-resolution information using the motion estimation results;

creating an enhancement layer containing the high-resolution information; and

incorporating motion vector data within the base layer for use in the decoding of the enhancement layer.