MEDIA CODING FOR LOSS RECOVERY WITH REMOTELY PREDICTED DATA UNITS

• US 20100135412A1
• Filed: 02/03/2010
• Published: 06/03/2010
• Est. Priority Date: 03/12/1999
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wang, Albert Szu-chi, Lee, Ming-Chieh
• Granted Patent: US 8,548,051 B2
• Time in Patent Office: Days
• US Class Current: 375/240.160
• CPC Class Codes: G10L 19/005 Correction of errors induced by the transmission channel, if related to the coding algorithm
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/114 Adapting the group of pictures [GOP] structure, e.g. number of B-frames between two anchor frames H04N19/107 takes precedence
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/166 concerning the amount of transmission errors, e.g. bit error rate [BER]
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/177 the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/50 using predictive coding H04N19/61 takes precedence
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/573 Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/58 Motion compensation with long-term prediction, i.e. the reference frame for a current frame not being the temporally closest one H04N19/23 takes precedence
• CPC Class Codes: H04N 19/593 involving spatial prediction techniques
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/89 involving methods or arrangements for detection of transmission errors at the decoder
• CPC Class Codes: H04N 19/91 : Entropy coding, e.g. variable length coding [VLC] or arithmetic coding
• CPC Class Codes: H04N 21/6377 : directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server

Abstract

An improved loss recovery method for coding streaming media classifies each data unit in the media stream as an independent data unit (I unit), a remotely predicted unit (R unit) or a predicted data unit (P unit). Each of these units is organized into independent segments having an I unit, multiple P units and R units interspersed among the P units. The beginning of each segment is the start of a random access point, while each R unit provides a loss recovery point that can be placed independently of the I unit. This approach separates the random access point from the loss recovery points provided by the R units, and makes the stream more impervious to data losses without substantially impacting coding efficiency. The most important data units are transmitted with the most reliability to ensure that the majority of the data received by the client is usable. The I units are the least sensitive to transmission losses because they are coded using only their own data. While they provide the best coding efficiency, the P units are the most sensitive to data loss because the loss of one P unit renders useless all of the P units that depend on it. The remotely predicted units are dependent on the I unit, or in an alternative implementation, on another R unit.

Claims

1-26. -26. (canceled)

27. A method of decoding video using a video decoder implemented in a computer system, the method comprising:

at the video decoder, receiving encoded video in a data stream, wherein the encoded video includes an entropy coded macroblock type for a current macroblock of a current frame;

with the video decoder, decoding video using the encoded video, including;

decoding the entropy coded macroblock type of the current macroblock to determine its macroblock type from among plural macroblock types possible for the current macroblock, the plural macroblock types including;

an intra type that indicates the current macroblock is to be decoded using intra decoding;

a first motion-compensated prediction type that indicates motion compensation for the current macroblock is to be performed relative to a first reference frame that immediately precedes the current frame; and

a second motion-compensated prediction type that indicates motion compensation for the current macroblock is to be performed relative to a second reference frame different than the first reference frame; and

decoding the current macroblock according to the macroblock type of the current macroblock, including;

when the macroblock type for the current macroblock is the intra type, decoding the current macroblock in an intra decoding mode;

when the macroblock type for the current macroblock is the first motion-compensated prediction type, decoding the current macroblock using motion compensation relative to the first reference frame, wherein the motion compensation for the first motion-compensated prediction type uses at most one motion vector and one reference frame per block; and

when the macroblock type for the current macroblock is the second motion-compensated prediction type, decoding the current macroblock using motion compensation relative to the second reference frame, wherein the motion compensation for the second motion-compensated prediction type uses at most one motion vector and one reference frame per block.

28. The method of claim 27 wherein:

a first buffer stores the first reference frame;

a second buffer stores the second reference frame; and

the second reference frame is a reconstructed version of a previous frame having at least some macroblocks decoded using motion compensation.

29. The method of claim 28 wherein, during decoding of plural frames that include the current frame and the previous frame, the video decoder updates the first buffer and the second buffer depending on classification of the plural frames, respectively, including:

if a given frame of the plural frames is an intra frame, storing a reconstructed version of the given frame in the first buffer and in the second buffer;

if the given frame is classified as a type of frame from which subsequent macroblocks with the second motion-compensated prediction type are predicted, storing the reconstructed version of the given frame in the first buffer and in the second buffer; and

otherwise, storing the reconstructed version of the given frame in the first buffer but not in the second buffer.

30. The method of claim 27 further comprising:

storing a reconstructed version of the current frame in a first buffer used in motion-compensated prediction for subsequent macroblocks with the first motion-compensated prediction type; and

determining if the current frame is to be used as a reference frame for subsequent macroblocks with the second motion-compensated prediction type and, if so, storing the reconstructed version of the current frame in a second buffer.

31. The method of claim 27 wherein the decoding the current macroblock further comprises, when the macroblock type for the current macroblock is the first motion-compensated prediction type or the second motion-compensated prediction type:

decoding error signal values for the current macroblock; and

combining the error signal values for the current macroblock with results of the motion compensation performed for the current macroblock.

32. The method of claim 31 wherein:

the decoding of the entropy coded macroblock type for the current macroblock uses arithmetic decoding;

the decoding the current macroblock further comprises determining a motion vector for the current macroblock based at least in part upon arithmetic decoding of motion vector information for the current macroblock; and

the decoding the error signal values for the current macroblock includes;

arithmetic decoding of encoded transform coefficient values;

inverse quantization of the decoded transform coefficient values; and

performing an inverse DCT of the inverse quantized transform coefficient values.

33. The method of claim 27 wherein use of the first motion-compensated prediction type promotes coding efficiency at the expense of loss recovery, and wherein use of the second motion-compensated prediction type promotes loss recovery at the expense of coding efficiency.

34. The method of claim 27 wherein the first motion-compensated prediction type and the second motion-compensated prediction type are used for decoding of different objects in the first reference frame and second reference frame, respectively.

35. The method of claim 27 wherein the current frame has a frame type that permits only intra decoding or inter decoding with forward motion-compensated prediction.

36. The method of claim 27 wherein the video decoder is implemented using any one or more of:

software that executes on a general-purpose processor of the computer system, the software being stored in memory of the computer system;

special-purpose digital logic in the computer system; and

software the controls special-purpose digital logic in the computer system.

37. The method of claim 27 wherein the computer system is any one of a desktop computer, hand-held device, other consumer electronic device, or distributed computing system.

38. A method of encoding video using a video encoder implemented in a computer system, the method comprising:

with the video encoder, encoding video to produce encoded video for output to a bit stream, including;

selecting a macroblock type of a current macroblock of a current frame from among plural macroblock types possible for the current macroblock, the plural macroblock types including;

an intra type that indicates the current macroblock is encoded using intra coding;

a first motion-compensated prediction type that indicates motion compensation for the current macroblock is performed relative to a first reference frame that immediately precedes the current frame; and

a second motion-compensated prediction type that indicates motion compensation for the current macroblock is performed relative to a second reference frame different than the first reference frame;

entropy coding the macroblock type of the current macroblock; and

encoding the current macroblock according to the macroblock type of the current macroblock, including;

when the macroblock type for the current macroblock is the intra type, encoding the current macroblock in an intra coding mode;

when the macroblock type for the current macroblock is the first motion-compensated prediction type, encoding the current macroblock using motion compensation relative to the first reference frame, wherein the motion compensation for the first motion-compensated prediction type uses at most one motion vector and one reference frame per block; and

when the macroblock type for the current macroblock is the second motion-compensated prediction type, encoding the current macroblock using motion compensation relative to the second reference frame, wherein the motion compensation for the second motion-compensated prediction type uses at most one motion vector and one reference frame per block.

with the video encoder, outputting the encoded video in a data stream, wherein the encoded video includes the entropy coded macroblock type for the current macroblock of a current frame.

39. The method of claim 38 wherein:

a first buffer stores the first reference frame;

a second buffer stores the second reference frame; and

the second reference frame is a reconstructed version of a previous frame having at least some macroblocks encoded and decoded using motion compensation.

40. The method of claim 39 wherein, during encoding of plural frames that include the current frame and the previous frame, the video encoder updates the first buffer and the second buffer depending on classification of the plural frames, respectively, including:

if a given frame of the plural frames is an intra frame, storing a reconstructed version of the given frame in the first buffer and in the second buffer;

if the given frame is classified as a type of frame from which subsequent macroblocks with the second motion-compensated prediction type are predicted, storing the reconstructed version of the given frame in the first buffer and in the second buffer; and

otherwise, storing the reconstructed version of the given frame in the first buffer but not in the second buffer.

41. The method of claim 38 wherein the encoding the current macroblock further comprises, when the macroblock type for the current macroblock is the first motion-compensated prediction type or the second motion-compensated prediction type:

determining error signal values for the current macroblock; and

encoding the error signal values for the current macroblock.

42. The method of claim 41 wherein:

the entropy coding of the macroblock type for the current macroblock uses arithmetic coding;

the encoding the current macroblock further comprises encoding motion vector information for a motion vector for the current macroblock using arithmetic coding; and

the encoding the error signal values for the current macroblock includes arithmetic coding of quantized transform coefficient values;

43. The method of claim 38 wherein the video encoder selects between the first and second motion-compensated prediction types for the current macroblock depending at least in part on a motion-compensated prediction error threshold.

44. A computer-readable medium storing in non-volatile storage computer-executable instructions for causing a video decoder programmed thereby to perform a method comprising:

receiving encoded video in a data stream, wherein the encoded video includes an entropy coded macroblock type for a current macroblock of a current frame;

decoding video using the encoded video, including;

decoding the entropy coded macroblock type of the current macroblock to determine its macroblock type from among plural macroblock types possible for the current macroblock, the plural macroblock types including;

an intra type that indicates the current macroblock is to be decoded using intra decoding;

a first motion-compensated prediction type that indicates motion compensation for the current macroblock is to be performed relative to a first reference frame that immediately precedes the current frame; and

a second motion-compensated prediction type that indicates motion compensation for the current macroblock is to be performed relative to a second reference frame different than the first reference frame; and

decoding the current macroblock according to the macroblock type of the current macroblock, including;

when the macroblock type for the current macroblock is the intra type, decoding the current macroblock in an intra decoding mode;

when the macroblock type for the current macroblock is the first motion-compensated prediction type, decoding the current macroblock using motion compensation relative to the first reference frame, wherein the motion compensation for the first motion-compensated prediction type uses at most one motion vector and one reference frame per block; and

when the macroblock type for the current macroblock is the second motion-compensated prediction type, decoding the current macroblock using motion compensation relative to the second reference frame, wherein the motion compensation for the second motion-compensated prediction type uses at most one motion vector and one reference frame per block.

45. The computer-readable medium of claim 44 wherein the method further comprises:

storing a reconstructed version of the current frame in a first buffer used in motion-compensated prediction for subsequent macroblocks with the first motion-compensated prediction type; and

determining if the current frame is to be used as a reference frame for subsequent macroblocks with the second motion-compensated prediction type and, if so, storing the reconstructed version of the current frame in a second buffer.

46. The computer-readable medium of claim 44 wherein:

the decoding of the entropy coded macroblock type for the current macroblock uses arithmetic decoding;

the decoding the current macroblock further comprises determining a motion vector for the current macroblock based at least in part upon arithmetic decoding of motion vector information for the current macroblock.