Advanced bi-directional predictive coding of video frames

• US 7,609,763 B2
• Filed: 07/18/2003
• Issued: 10/27/2009
• Est. Priority Date: 07/18/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Srinivasan, Sridhar, Mukerjee, Kunal, Lin, Bruce Chih-Lung
• Primary Examiner(s): DIEP, NHON THANH
• Application Number:
  US10/622,378
• Publication Number:
  US 20050013365A1
• Time in Patent Office: 2,293 Days
• Field of Search: None
• US Class Current: 375/240.16
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/56 Motion estimation with initialisation of the vector search, e.g. estimating a good candidate to initiate a search
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence

Abstract

Techniques and tools for coding/decoding of video images, and in particular, B-frames, are described. In one aspect, a video encoder/decoder determines a fraction for a current image in a sequence. The fraction represents an estimated temporal distance position for the current image relative to an interval between a reference images for the current image. The video encoder/decoder processes the fraction along with a motion vector for a first reference image, resulting in a representation of motion (e.g., constant or variable velocity motion) in the current image. Other aspects are also described, including intra B-frames, forward and backward buffers for motion vector prediction, bitplane encoding of direct mode prediction information, multiple motion vector resolutions/interpolation filters for B-frames, proactive dropping of B-frames, and signaling of dropped predicted frames.

Claims

1. In a computing device that implements a video decoder, the computing device including a processor and memory, a method of decoding images in a sequence of video images, the method comprising:

receiving and decoding, with the computing device that implements the video decoder, a code in a bit stream to determine a fraction for a current image in the sequence, wherein the fraction represents an estimated temporal distance position for the current image relative to an interval between a first reference image for the current image and a second reference image for the current image, and wherein the determination of the fraction is independent of actual temporal distance positions of the respective reference images; and

for motion compensation for a direct mode macroblock in the current image, with the computing device that implements the video decoder, processing the fraction along with a motion vector for a co-located macroblock in the first reference image, wherein the motion vector represents motion in the first reference image relative to the second reference image, and wherein the processing the fraction along with the motion vector results in a representation of motion for the direct mode macroblock in the current image relative to the first reference image and relative to the second reference image.

2. The method of claim 1 wherein the fraction is represented by the code, and wherein the code comprises a variable length code in the bit stream.

3. The method of claim 1 wherein the fraction is selected from a set of discrete values, wherein each of the values is greater than zero and less than one so as to indicate the estimated temporal distance position within the interval.

4. The method of claim 1 wherein the fraction is selected from the group consisting of:

½

, ⅓

, ⅔

, ¼

, ¾

, ⅕

, ⅖

, ⅗

, ⅘

, ⅙

, ⅚

, 1/7, 2/7, and 3/7.

5. The method of claim 1 wherein the estimated temporal distance position for the current image relative to the interval between the first reference image for the current image and the second reference image for the current image is not the true temporal distance position of the current image.

6. The method of claim 1 wherein the fraction is based on motion information for the sequence of video images.

7. The method of claim 1 wherein the fraction is based on a proximity of the current image to an end of the sequence of video images.

8. The method of claim 1 further comprising, with the computing device that implements the video decoder, repeating the acts of claim 1 for each of plural bi-directionally predicted images in the sequence of video images.

9. The method of claim 2 wherein the determination of the fraction comprises looking up the variable length code in a variable length code table to obtain a value for the fraction.

10. The method of claim 9 wherein at least one entry in the variable length code table represents a frame type, and wherein the frame type is B/I-frame.

11. The method of claim 1 wherein the processing the fraction along with the motion vector comprises scaling the motion vector for the co-located macroblock using the fraction.

12. The method of claim 11 wherein the scaling the motion vector for the co-located macroblock comprises scaling a vertical component and a horizontal component of the motion vector for the co-located macroblock.

13. The method of claim 11 wherein the scaling the motion vector for the co-located macroblock comprises:

scaling the motion vector for the co-located macroblock by a factor of the fraction, to obtain an implied forward motion vector for the direct mode macroblock; and

scaling the motion vector for the co-located macroblock by a factor of the fraction minus one, to obtain an implied backward motion vector for the direct mode macroblock.

14. The method of claim 13 wherein the motion compensation comprises:

addressing a macroblock in the future reference image using the implied forward motion vector;

addressing a macroblock in the previous reference image using the implied backward motion vector; and

predicting the current macroblock using an average of the macroblock in the future reference image and the macroblock in the previous reference image.

15. In a computing device that implements a video encoder, the computing device including a processor and memory, a method of encoding images in a sequence of video images, the method comprising:

with the computing device that implements the video encoder, determining a fraction for a current image in the sequence, wherein the current image has a previous reference image and a future reference image, and wherein the fraction represents a temporal position for the current image relative to the respective reference images;

with the computing device that implements the video encoder, selecting direct mode prediction for a current direct mode macroblock in the current image;

with the computing device that implements the video encoder, finding a motion vector for a co-located macroblock in the future reference image;

with the computing device that implements the video encoder, scaling the motion vector for the co-located macroblock using the fraction;

with the computing device that implements the video encoder, using results of the scaling in motion compensation for the current direct mode macroblock in the current image; and

with the computing device that implements the video encoder, outputting a code in a bit stream, wherein the code represents the fraction, and wherein the outputting the code facilitates determination of the fraction independent of actual temporal positions of the respective reference images during decoding.

16. The method of claim 15 wherein the fraction facilitates representation of variable velocity motion in the direct mode prediction.

17. The method of claim 15 wherein the scaling the motion vector for the co-located macroblock comprises scaling a vertical component and a horizontal component of the motion vector for the co-located macroblock.

18. The method of claim 15 wherein the scaling the motion vector for the co-located macroblock comprises:

scaling the motion vector for the co-located macroblock by a factor of the fraction, to obtain an implied forward motion vector for the current direct mode macroblock; and

scaling the motion vector for the co-located macroblock by a factor of the fraction minus one, to obtain an implied backward motion vector for the current direct mode macroblock, wherein for the direct mode macroblock the motion compensation uses the implied forward motion vector and the implied backward motion vector.

19. A system comprising:

one or more processors;

memory;

at least one input device, output device or communication connection; and

one or more storage media having stored thereon computer-executable instructions for causing one or more computers to perform a method of decoding images in a sequence of video images, the method comprising;

receiving and decoding a code in a bit stream to determine a fraction for a current image in the sequence, wherein the fraction represents an estimated temporal distance position for the current image relative to an interval between a first reference image for the current image and a second reference image for the current image, and wherein the determination of the fraction is independent of actual temporal distance positions of the respective reference images; and

for motion compensation for a direct mode macroblock in the current image, processing the fraction along with a motion vector for a co-located macroblock in the first reference image, wherein the motion vector represents motion in the first reference image relative to the second reference image, and wherein the processing the fraction along with the motion vector results in a representation of motion for the direct mode macroblock in the current image relative to the first reference image and relative to the second reference image.

20. The system of claim 19 wherein the fraction is represented by the code, and wherein the code comprises a variable length code in the bit stream.

21. The system of claim 19 wherein the fraction is selected from a set of discrete values, wherein each of the values is greater than zero and less than one so as to indicate the estimated temporal distance position within the interval.

22. The system of claim 19 wherein the fraction is selected from the group consisting of:

½

, ⅓

, ⅔

, ¼

, ¾

, ⅕

, ⅖

, ⅗

, ⅘

, ⅙

, ⅚

, 1/7, 2/7, and 3/7.

23. The system of claim 19 wherein the estimated temporal distance position for the current image relative to the interval between the first reference image for the current image and the second reference image for the current image is not the true temporal distance position of the current image.

24. The system of claim 19 wherein the fraction is based on motion information for the sequence of video images.

25. The system of claim 19 wherein the fraction is based on a proximity of the current image to an end of the sequence of video images.

26. The system of claim 19 wherein the determination of the fraction comprises looking up the code in a code table to obtain a value for the fraction.

27. The system of claim 26 wherein at least one entry in the code table represents a frame type, and wherein the frame type is B/I-frame.

28. In a computing device that implements a video encoder, the computing device including a processor and memory, a method of encoding images in a sequence of video images, the method comprising:

with the computing device that implements the video encoder, determining a fraction for a current image in the sequence, wherein the fraction represents an estimated temporal distance position for the current image relative to an interval between a first reference image for the current image and a second reference image for the current image, and wherein the determination of the fraction is independent of actual temporal distance positions of the respective reference images;

for motion compensation for a direct mode macroblock in the current image, with the computing device that implements the video encoder, processing the fraction along with a motion vector for a co-located macroblock in the first reference image, wherein the motion vector represents motion in the first reference image relative to the second reference image, and wherein the processing the fraction along with the motion vector results in a representation of motion for the direct mode macroblock in the current image relative to the first reference image and relative to the second reference image; and

with the computing device that implements the video encoder, outputting a code in a bit stream, wherein the code represents the fraction.

29. The method of claim 28 wherein the code comprises a variable length code in the bit stream.

30. The method of claim 28 wherein the fraction is selected from a set of discrete values, wherein each of the values is greater than zero and less than one so as to indicate the estimated temporal distance position within the interval.

31. The method of claim 28 wherein the fraction is selected from the group consisting of:

½

, ⅓

, ⅔

, ¼

, ¾

, ⅕

, ⅖

, ⅗

, ⅘

, ⅙

, ⅚

, 1/7, 2/7, and 3/7.

32. The method of claim 28 wherein the estimated temporal distance position for the current image relative to the interval between the first reference image for the current image and the second reference image for the current image is not the true temporal distance position of the current image.

33. The method of claim 28 wherein the fraction is based on motion information for the sequence of video images.

34. The method of claim 28 wherein the fraction is based on a proximity of the current image to an end of the sequence of video images.

35. The method of claim 28 wherein the determination of the fraction comprises looking up the code in a code table to obtain a value for the fraction.

36. The method of claim 35 wherein at least one entry in the code table represents a frame type, and wherein the frame type is B/I-frame.

37. The method of claim 28 wherein the determining the fraction comprises:

evaluating each of plural fractions to determine bit costs for encoding the current image using the respective fractions; and

selecting the fraction based on the evaluating.

38. A system comprising:

one or more processors;

memory;

at least one input device, output device or communication connection; and

one or more storage media having stored thereon computer-executable instructions for causing one or more computers to perform a method of encoding images in a sequence of video images, the method comprising;

determining a fraction for a current image in the sequence, wherein the fraction represents an estimated temporal distance position for the current image relative to an interval between a first reference image for the current image and a second reference image for the current image, and wherein the determination of the fraction is independent of actual temporal distance positions of the respective reference images;

for motion compensation for a direct mode macroblock in the current image, processing the fraction along with a motion vector for a co-located macroblock in the first reference image, wherein the motion vector represents motion in the first reference image relative to the second reference image, and wherein the processing the fraction along with the motion vector results in a representation of motion for the direct mode macroblock in the current image relative to the first reference image and relative to the second reference image; and

outputting a code in a bit stream, wherein the code represents the fraction.

39. The system of claim 38 wherein the code comprises a variable length code in the bit stream.

40. The system of claim 38 wherein the fraction is selected from a set of discrete values, wherein each of the values is greater than zero and less than one so as to indicate the estimated temporal distance position within the interval.

41. The system of claim 38 wherein the fraction is selected from the group consisting of:

½

, ⅓

, ⅔

, ¼

, ¾

, ⅕

, ⅖

, ⅗

, ⅘

, ⅙

, ⅚

, 1/7, 2/7, and 3/7.

42. The system of claim 38 wherein the estimated temporal distance position for the current image relative to the interval between the first reference image for the current image and the second reference image for the current image is not the true temporal distance position of the current image.

43. The system of claim 38 wherein the fraction is based on motion information for the sequence of video images.

44. The system of claim 38 wherein the fraction is based on a proximity of the current image to an end of the sequence of video images.

45. The system of claim 38 wherein the determination of the fraction comprises looking up the code in a code table to obtain a value for the fraction.

46. The system of claim 45 wherein at least one entry in the code table represents a frame type, and wherein the frame type is B/I-frame.

47. The system of claim 38 wherein the determining the fraction comprises:

evaluating each of plural fractions to determine bit costs for encoding the current image using the respective fractions; and

selecting the fraction based on the evaluating.

48. A system comprising:

one or more processors;

memory;

at least one input device, output device or communication connection; and

one or more storage media having stored thereon computer-executable instructions for causing one or more computers to perform a method of encoding images in a sequence of video images, the method comprising;

determining a fraction for a current image in the sequence, wherein the current image has a previous reference image and a future reference image, and wherein the fraction represents a temporal position for the current image relative to the respective reference images;

selecting direct mode prediction for a current direct mode macroblock in the current image;

finding a motion vector for a co-located macroblock in the future reference image;

scaling the motion vector for the co-located macroblock using the fraction;

using results of the scaling in motion compensation for the current direct mode macroblock in the current image; and

outputting a code in a bit stream, wherein the code represents the fraction, and wherein the outputting the code facilitates determination of the fraction independent of actual temporal positions of the respective reference images during decoding.

49. The system of claim 48 wherein the fraction facilitates representation of variable velocity motion in the direct mode prediction.

50. The system of claim 48 wherein the scaling the motion vector for the co-located macroblock comprises scaling a vertical component and a horizontal component of the motion vector for the co-located macroblock.

51. The system of claim 48 wherein the scaling the motion vector for the co-located macroblock comprises:

scaling the motion vector for the co-located macroblock by a factor of the fraction, to obtain an implied forward motion vector for the current direct mode macroblock; and

scaling the motion vector for the co-located macroblock by a factor of the fraction minus one, to obtain an implied backward motion vector for the current direct mode macroblock, wherein for the current direct mode macroblock the motion compensation uses the implied forward motion vector and the implied backward motion vector.