Media coding for loss recovery with remotely predicted data units

• US 8,548,051 B2
• Filed: 02/03/2010
• Issued: 10/01/2013
• Est. Priority Date: 03/12/1999
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wang, Albert Szu-Chi, Lee, Ming-Chieh
• Primary Examiner(s): Czekaj, David
• Assistant Examiner(s): LI, TRACY Y
• Application Number:
  US12/699,354
• Publication Number:
  US 20100135412A1
• Time in Patent Office: 1,336 Days
• Field of Search: 375/240.01, 375/240.12
• US Class Current: 375/240.12
• CPC Class Codes: G10L 19/005 Correction of errors induced by the transmission channel, if related to the coding algorithm
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/114 Adapting the group of pictures [GOP] structure, e.g. number of B-frames between two anchor frames H04N19/107 takes precedence
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/166 concerning the amount of transmission errors, e.g. bit error rate [BER]
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/177 the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/50 using predictive coding H04N19/61 takes precedence
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/573 Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/58 Motion compensation with long-term prediction, i.e. the reference frame for a current frame not being the temporally closest one H04N19/23 takes precedence
• CPC Class Codes: H04N 19/593 involving spatial prediction techniques
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/89 involving methods or arrangements for detection of transmission errors at the decoder
• CPC Class Codes: H04N 19/91 : Entropy coding, e.g. variable length coding [VLC] or arithmetic coding
• CPC Class Codes: H04N 21/6377 : directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server

Abstract

An improved loss recovery method for coding streaming media classifies each data unit in the media stream as an independent data unit (I unit), a remotely predicted unit (R unit) or a predicted data unit (P unit). Each of these units is organized into independent segments having an I unit, multiple P units and R units interspersed among the P units. The beginning of each segment is the start of a random access point, while each R unit provides a loss recovery point that can be placed independently of the I unit. This approach separates the random access point from the loss recovery points provided by the R units, and makes the stream more impervious to data losses without substantially impacting coding efficiency. The most important data units are transmitted with the most reliability to ensure that the majority of the data received by the client is usable. The I units are the least sensitive to transmission losses because they are coded using only their own data. While they provide the best coding efficiency, the P units are the most sensitive to data loss because the loss of one P unit renders useless all of the P units that depend on it. The remotely predicted units are dependent on the I unit, or in an alternative implementation, on another R unit.

Claims

1. A computer system comprising a processor and memory that stores instructions for causing the computer system to operate as a video decoder adapted to:

receive encoded video in a data stream, wherein the encoded video includes an entropy coded macroblock type for a current macroblock of a current frame;

decode video using the encoded video, wherein the video decoder is further adapted to;

decode the entropy coded macroblock type of the current macroblock to determine its macroblock type from among plural macroblock types possible for the current macroblock, the plural macroblock types including;

an intra type that indicates the current macroblock is to be decoded using intra decoding;

a first motion-compensated prediction type that indicates motion compensation for the current macroblock is to be performed relative to a first reference frame in a first buffer, wherein the first buffer is adapted to store as the first reference frame a reconstructed version of a previous frame that immediately precedes the current frame; and

a second motion-compensated prediction type that indicates motion compensation for the current macroblock is to be performed relative to a second reference frame in a second buffer, wherein the second buffer is adapted to store as the second reference frame a reconstructed version of a previous frame that precedes the first reference frame; and

decode the current macroblock according to the macroblock type of the current macroblock, wherein the video decoder is further adapted to;

when the macroblock type for the current macroblock is the intra type, decode the current macroblock in an intra decoding mode;

when the macroblock type for the current macroblock is the first motion-compensated prediction type, decode the current macroblock using motion compensation relative to the first reference frame in the first buffer, wherein the motion compensation for the first motion-compensated prediction type uses at most one motion vector and one reference frame per block; and

when the macroblock type for the current macroblock is the second motion-compensated prediction type, decode the current macroblock using motion compensation relative to the second reference frame in the second buffer, wherein the motion compensation for the second motion-compensated prediction type uses at most one motion vector and one reference frame per block.

2. The computer system of claim 1 wherein the video decoder is further adapted to, during decoding of plural frames that include the current frame and the previous frame, update the first buffer and the second buffer depending on classification of the plural frames, respectively, including:

if a given frame of the plural frames is an intra frame, store a reconstructed version of the given frame in the first buffer and in the second buffer;

if the given frame is classified as a type of frame from which subsequent macroblocks with the second motion-compensated prediction type are predicted, store the reconstructed version of the given frame in the first buffer and in the second buffer; and

otherwise, store the reconstructed version of the given frame in the first buffer but not in the second buffer.

3. The computer system of claim 1 wherein the video decoder is further adapted to:

store a reconstructed version of the current frame in the first buffer for use in motion-compensated prediction for subsequent macroblocks with the first motion-compensated prediction type; and

determine if the current frame is to be used as a reference frame for subsequent macroblocks with the second motion-compensated prediction type and, if so, store the reconstructed version of the current frame in the second buffer.

4. The computer system of claim 1 wherein, when the macroblock type for the current macroblock is the first motion-compensated prediction type or the second motion-compensated prediction type, the video decoder is further adapted to:

decode error signal values for the current macroblock; and

combine the error signal values for the current macroblock with results of the motion compensation performed for the current macroblock.

5. The computer system of claim 4 wherein the video decoder is further adapted to:

decode the entropy coded macroblock type for the current macroblock using arithmetic decoding;

decode the current macroblock by determining a motion vector for the current macroblock based at least in part upon arithmetic decoding of motion vector information for the current macroblock; and

decode the error signal values for the current macroblock by;

arithmetic decoding of encoded transform coefficient values;

inverse quantization of the decoded transform coefficient values; and

performing an inverse DCT of the inverse quantized transform coefficient values.

6. The computer system of claim 1 wherein use of the first motion-compensated prediction type promotes coding efficiency at the expense of loss recovery, and wherein use of the second motion-compensated prediction type promotes loss recovery at the expense of coding efficiency.

7. The computer system of claim 1 wherein the video decoder is adapted to use the first motion-compensated prediction type and the second motion-compensated prediction type for decoding of different objects in the first reference frame and second reference frame, respectively.

8. The computer system of claim 1 wherein the current frame has a frame type that permits only intra decoding or inter decoding with forward motion-compensated prediction.

9. The computer system of claim 1 wherein the video decoder is implemented using any one or more of:

software that executes the instructions on the processor of the computer system, wherein the processor is a general-purpose processor, the software being stored in the memory of the computer system; and

software that executes the instructions on the processor of the computer system, wherein the processor is special-purpose digital logic in the computer system.

10. The computer system of claim 1 wherein the computer system is any one of a desktop computer, hand-held device, other consumer electronic device, or distributed computing system.

11. A computer system comprising a processor and memory that stores instructions for causing the computer system to operate as a video encoder adapted to:

encode video to produce encoded video for output to a data stream, wherein the video encoder is further adapted to;

select a macroblock type of a current macroblock of a current frame from among plural macroblock types possible for the current macroblock, the plural macroblock types including;

an intra type that indicates the current macroblock is encoded using intra coding;

a first motion-compensated prediction type that indicates motion compensation for the current macroblock is performed relative to a first reference frame in a first buffer, wherein the first buffer is adapted to store as the first reference frame a reconstructed version of a previous frame that immediately precedes the current frame; and

a second motion-compensated prediction type that indicates motion compensation for the current macroblock is performed relative to a second reference frame in a second buffer, wherein the second buffer is adapted to store as the second reference frame a reconstructed version of a previous frame that precedes the first reference frame;

entropy code the macroblock type of the current macroblock; and

encode the current macroblock according to the macroblock type of the current macroblock, wherein the video encoder is further adapted to;

when the macroblock type for the current macroblock is the intra type, encode the current macroblock in an intra coding mode;

when the macroblock type for the current macroblock is the first motion-compensated prediction type, encode the current macroblock using motion compensation relative to the first reference frame in the first buffer, wherein the motion compensation for the first motion-compensated prediction type uses at most one motion vector and one reference frame per block; and

when the macroblock type for the current macroblock is the second motion-compensated prediction type, encode the current macroblock using motion compensation relative to the second reference frame in the second buffer, wherein the motion compensation for the second motion-compensated prediction type uses at most one motion vector and one reference frame per block; and

output the encoded video in the data stream, wherein the encoded video includes the entropy coded macroblock type for the current macroblock of a current frame.

12. The computer system of claim 11 wherein the video encoder is further adapted to, during encoding of plural frames that include the current frame and the previous frame, update the first buffer and the second buffer depending on classification of the plural frames, respectively, including:

if a given frame of the plural frames is an intra frame, store a reconstructed version of the given frame in the first buffer and in the second buffer;

if the given frame is classified as a type of frame from which subsequent macroblocks with the second motion-compensated prediction type are predicted, store the reconstructed version of the given frame in the first buffer and in the second buffer; and

otherwise, store the reconstructed version of the given frame in the first buffer but not in the second buffer.

13. The computer system of claim 11 wherein, when the macroblock type for the current macroblock is the first motion-compensated prediction type or the second motion-compensated prediction type, the video encoder is further adapted to:

determine error signal values for the current macroblock; and

encode the error signal values for the current macroblock.

14. The computer system of claim 13 wherein the video encoder is further adapted to:

encode the macroblock type for the current macroblock using arithmetic coding;

encode the current macroblock by encoding motion vector information for a motion vector for the current macroblock using arithmetic coding; and

encode the error signal values for the current macroblock by arithmetic coding of quantized transform coefficient values.

15. The computer system of claim 11 wherein the video encoder is adapted to select between the first and second motion-compensated prediction types for the current macroblock depending at least in part on a motion-compensated prediction error threshold.

16. A computer-readable medium storing computer-executable instructions for causing a video decoder programmed thereby to perform a method, the computer-readable medium including one or more of magnetic storage and optical storage, the method comprising:

receiving encoded video in a data stream, wherein the encoded video includes an entropy coded macroblock type for a current macroblock of a current frame;

decoding video using the encoded video, including;

decoding the entropy coded macroblock type of the current macroblock to determine its macroblock type from among plural macroblock types possible for the current macroblock, the plural macroblock types including;

an intra type that indicates the current macroblock is to be decoded using intra decoding;

a first motion-compensated prediction type that indicates motion compensation for the current macroblock is to be performed relative to a first reference frame in a first buffer, wherein the first buffer is adapted to store as the first reference frame a reconstructed version of a previous frame that immediately precedes the current frame; and

a second motion-compensated prediction type that indicates motion compensation for the current macroblock is to be performed relative to a second reference frame in a second buffer, wherein the second buffer is adapted to store as the second reference frame a reconstructed version of a previous frame that precedes the first reference frame; and

decoding the current macroblock according to the macroblock type of the current macroblock, including;

when the macroblock type for the current macroblock is the intra type, decoding the current macroblock in an intra decoding mode;

when the macroblock type for the current macroblock is the first motion-compensated prediction type, decoding the current macroblock using motion compensation relative to the first reference frame in the first buffer, wherein the motion compensation for the first motion-compensated prediction type uses at most one motion vector and one reference frame per block; and

when the macroblock type for the current macroblock is the second motion-compensated prediction type, decoding the current macroblock using motion compensation relative to the second reference frame in the second buffer, wherein the motion compensation for the second motion-compensated prediction type uses at most one motion vector and one reference frame per block.

17. The computer-readable medium of claim 16 wherein the method further comprises:

storing a reconstructed version of the current frame in the first buffer used in motion-compensated prediction for subsequent macroblocks with the first motion-compensated prediction type; and

determining if the current frame is to be used as a reference frame for subsequent macroblocks with the second motion-compensated prediction type and, if so, storing the reconstructed version of the current frame in the second buffer.

18. The computer-readable medium of claim 16 wherein:

the decoding of the entropy coded macroblock type for the current macroblock uses arithmetic decoding; and

the decoding the current macroblock further comprises determining a motion vector for the current macroblock based at least in part upon arithmetic decoding of motion vector information for the current macroblock.

19. A method of decoding video using a video decoder implemented in a computer system, the method comprising:

at the video decoder, receiving encoded video in a data stream, wherein the encoded video includes an entropy coded macroblock type for a current macroblock of a current frame;

with the video decoder, decoding video using the encoded video, including;

decoding the entropy coded macroblock type of the current macroblock to determine its macroblock type from among plural macroblock types possible for the current macroblock, the plural macroblock types including;

an intra type that indicates the current macroblock is to be decoded using intra decoding;

a first motion-compensated prediction type that indicates motion compensation for the current macroblock is to be performed relative to a first reference frame in a first buffer, wherein the first buffer stores as the first reference frame a reconstructed version of a previous frame that immediately precedes the current frame; and

a second motion-compensated prediction type that indicates motion compensation for the current macroblock is to be performed relative to a second reference frame in a second buffer, wherein the second buffer stores as the second reference frame a reconstructed version of a previous frame that precedes the first reference frame; and

decoding the current macroblock according to the macroblock type of the current macroblock, including;

when the macroblock type for the current macroblock is the intra type, decoding the current macroblock in an intra decoding mode;

when the macroblock type for the current macroblock is the first motion-compensated prediction type, decoding the current macroblock using motion compensation relative to the first reference frame in the first buffer, wherein the motion compensation for the first motion-compensated prediction type uses at most one motion vector and one reference frame per block; and

when the macroblock type for the current macroblock is the second motion-compensated prediction type, decoding the current macroblock using motion compensation relative to the second reference frame in the second buffer, wherein the motion compensation for the second motion-compensated prediction type uses at most one motion vector and one reference frame per block.

20. The method of claim 19 wherein, during decoding of plural frames that include the current frame and the previous frame, the video decoder updates the first buffer and the second buffer depending on classification of the plural frames, respectively, including:

if a given frame of the plural frames is an intra frame, storing a reconstructed version of the given frame in the first buffer and in the second buffer;

if the given frame is classified as a type of frame from which subsequent macroblocks with the second motion-compensated prediction type are predicted, storing the reconstructed version of the given frame in the first buffer and in the second buffer; and

otherwise, storing the reconstructed version of the given frame in the first buffer but not in the second buffer.

21. The method of claim 19 further comprising:

storing a reconstructed version of the current frame in the first buffer for use in motion-compensated prediction for subsequent macroblocks with the first motion-compensated prediction type; and

determining if the current frame is to be used as a reference frame for subsequent macroblocks with the second motion-compensated prediction type and, if so, storing the reconstructed version of the current frame in the second buffer.