Media coding for loss recovery with remotely predicted data units

• US 7,685,305 B2
• Filed: 06/28/2005
• Issued: 03/23/2010
• Est. Priority Date: 03/12/1999
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lee, Ming-Chieh, Wang, Albert Szu-Chi
• Primary Examiner(s): Bruckart; Benjamin R
• Assistant Examiner(s): Baturay; Alicia
• Application Number:
  US11/170,281
• Publication Number:
  US 20050237987A1
• Time in Patent Office: 1,729 Days
• Field of Search: 709230-237
• US Class Current: 709/231
• CPC Class Codes: G10L 19/005 Correction of errors induced by the transmission channel, if related to the coding algorithm
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04N 19/107 between spatial and temporal predictive coding, e.g. picture refresh
• CPC Class Codes: H04N 19/114 Adapting the group of pictures [GOP] structure, e.g. number of B-frames between two anchor frames H04N19/107 takes precedence
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/166 concerning the amount of transmission errors, e.g. bit error rate [BER]
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/177 the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/50 using predictive coding H04N19/61 takes precedence
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/573 Motion compensation with multiple frame prediction using two or more reference frames in a given prediction direction
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/58 Motion compensation with long-term prediction, i.e. the reference frame for a current frame not being the temporally closest one H04N19/23 takes precedence
• CPC Class Codes: H04N 19/593 involving spatial prediction techniques
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/89 involving methods or arrangements for detection of transmission errors at the decoder
• CPC Class Codes: H04N 19/91 : Entropy coding, e.g. variable length coding [VLC] or arithmetic coding
• CPC Class Codes: H04N 21/6377 : directed to server one-way streaming services wherein the source is controlled by the destination H04L65/613
• CPC Class Codes: H04N 21/658 : Transmission by the client directed to the server

Abstract

An improved loss recovery method for coding streaming media classifies each data unit in the media stream as an independent data unit (I unit), a remotely predicted unit (R unit) or a predicted data unit (P unit). Each of these units is organized into independent segments having an I unit, multiple P units and R units interspersed among the P units. The beginning of each segment is the start of a random access point, while each R unit provides a loss recovery point that can be placed independently of the I unit. This approach separates the random access point from the loss recovery points provided by the R units, and makes the stream more impervious to data losses without substantially impacting coding efficiency. The most important data units are transmitted with the most reliability to ensure that the majority of the data received by the client is usable. The I units are the least sensitive to transmission losses because they are coded using only their own data. While they provide the best coding efficiency, the P units are the most sensitive to data loss because the loss of one P unit renders useless all of the P units that depend on it. The remotely predicted units are dependent on the I unit, or in an alternative implementation, on another R unit.

Claims

1. In a computing device that implements a video encoder, the computing device including a processor and memory, a method of encoding video data streams in encoded form, the method comprising:

with the computing device that implements the video encoder, determining a type value for a unit of video data in a video image, wherein the type value is one of plural possible type values including an intra type value, a first single-unit single-reference inter-prediction type value, and a second single-unit single-reference inter-prediction type value;

when the type value for the unit is the intra type value, with the computing device that implements the video encoder, encoding at least some of the video data for the unit using an intra encoding mode including intra encoding;

when the type value for the unit is the first single-unit single-reference inter-prediction type value, with the computing device that implements the video encoder, encoding at least some of the video data for the unit using a first inter encoding mode that favors quality at the expense of decoding flexibility for a given bitrate, wherein the encoder uses at most one motion vector and one reference identifier per block in the first inter encoding mode;

when the type value for the unit is the second single-unit single-reference inter-prediction type value, with the computing device that implements the video encoder, encoding at least some of the video data for the unit using a second inter encoding mode different than the first inter encoding mode, wherein the second inter encoding mode favors decoding flexibility at the expense of quality for the given bitrate, and wherein the encoder also uses at most one motion vector and one reference identifier per block in the second inter encoding mode;

with the computing device that implements the video encoder, entropy encoding the type value for the unit; and

with the computing device that implements the video encoder, outputting part of a bitstream, the outputted part indicating the type value for the unit, wherein the type value for the unit is in encoded form in the bitstream.

2. The method of claim 1 wherein the unit is less than the entire video image.

3. The method of claim 1 further comprising, for the first and second inter encoding modes, with the computing device that implements the video encoder:

selecting from among plural reference images; and

outputting one or more additional parts of the bitstream, the outputted one or more additional parts indicating one or more reference identifiers for motion compensation.

4. The method of claim 3 wherein for at least one block of the unit the encoder uses one motion vector and one reference identifier and for at least one block of the unit the encoder uses one motion vector and no reference identifier.

5. The method of claim 1 wherein the determining the type value includes choosing the type value based at least in part upon a user selectable parameter for balancing quality and loss recovery.

6. The method of claim 1 wherein the determining the type value includes choosing the type value based at least in part upon packet loss information from a communications channel.

7. The method of claim 1 wherein the determining the type value includes choosing the type value based at least in part upon media content type.

8. The method of claim 1 wherein, according to the first inter encoding mode and according to the second inter encoding mode, the encoder performs motion compensation in a single temporal direction.

9. In a computing device that implements a video encoder, the computing device including a processor and memory, a method of encoding video data including plural units, wherein each of the plural units is for video data in one video image, the method comprising:

for each of the plural units;

with the computing device that implements the video encoder, determining a type value for the unit, wherein the type value is one of plural possible type values, the plural possible type values including an intra type value, a first single-unit single-reference inter-prediction type value, and a second single-unit single-reference inter-prediction type value;

when the type value for the unit is the intra type value, with the computing device that implements the video encoder, encoding at least some of the video data for the unit using an intra encoding mode;

when the type value for the unit is the first single-unit single-reference inter-prediction type value, with the computing device that implements the video encoder, encoding at least some of the video data for the unit using a first inter encoding mode that favors quality at the expense of decoding flexibility for a given bitrate, wherein the encoder uses at most one motion vector and one reference identifier per block in the first inter encoding mode;

when the type value for the unit is the second single-unit single-reference inter-prediction type value, with the computing device that implements the video encoder, encoding at least some of the video data for the unit using a second inter encoding mode, wherein the encoder also uses at most one motion vector and one reference identifier per block in the second inter encoding mode, wherein the second inter encoding mode differs from the first inter encoding mode, and wherein the second inter encoding mode favors decoding flexibility at the expense of quality for the given bitrate;

with the computing device that implements the video encoder, entropy encoding the type value for the unit; and

with the computing device that implements the video encoder, outputting part of a bitstream, the outputted part indicating the type value for the unit, wherein the type value for the unit is in encoded form in the bitstream;

wherein, for the first and second inter encoding modes, the encoder selects from among plural reference images and outputs one or more additional parts of the bitstream, the outputted one or more additional parts indicating one or more reference identifiers for motion compensation.

10. The method of claim 9 wherein at least one of the plural units is less than an entire video image.

11. The method of claim 9 further comprising, with the computing device that implements the video encoder, updating the plural reference images, including adding a current image to the plural reference images and removing a previous image from the plural reference images.

12. The method of claim 9 wherein for at least one block of the unit the encoder uses one motion vector and one reference identifier and for at least one block of the unit the encoder uses one motion vector and no reference identifier.

13. The method of claim 9 wherein the determining the type value includes choosing the type value based at least in part upon media content type.

14. The method of claim 9 wherein the determining the type value includes choosing the type value based at least in part upon packet loss information from a communications channel.

15. The method of claim 9 wherein, according to the first inter encoding mode and according to the second inter encoding mode, the encoder performs motion compensation in a single temporal direction.

16. In a computing device that implements a video encoder, the computing device including a processor and memory, a method of encoding video data, the method comprising:

with the computing device that implements the video encoder, determining a type value for a unit of the video data, wherein the type value is one of plural possible type values, the plural possible type values including an intra type value, a first single-unit single-reference inter-prediction type value, and a second single-unit single-reference inter-prediction type value;

when the type value for the unit is the intra type value, with the computing device that implements the video encoder, encoding at least some of the video data for the unit using intra encoding;

when the type value for the unit is the first single-unit single-reference inter-prediction type value, with the computing device that implements the video encoder, encoding at least some of the video data for the unit using a first inter encoding mode that favors quality at the expense of decoding flexibility for a given bitrate, wherein the encoder uses at most one motion vector and one reference identifier per block in the first inter encoding mode;

when the type value for the unit is the second single-unit single-reference inter-prediction type value, with the computing device that implements the video encoder, encoding at least some of the video data for the unit using a second inter encoding mode different than the first inter encoding mode, wherein the encoder also uses at most one motion vector and one reference identifier per block in the second inter encoding mode, and wherein the second inter encoding mode favors decoding flexibility at the expense of quality for the given bitrate;

with the computing device that implements the video encoder, entropy encoding the type value for the unit; and

with the computing device that implements the video encoder, outputting part of a bitstream, the outputted part indicating the entropy coded type value for the unit, wherein the type value for the unit is in encoded form in the bitstream.

17. The method of claim 16 wherein the unit is less than an entire video image.

18. The method of claim 16 wherein, for the first and second inter encoding modes, the encoder selects from among plural reference images and outputs one or more reference identifiers for motion compensation.

19. The method of claim 16 wherein for at least one block of the unit the encoder uses one motion vector and one reference identifier and for at least one block of the unit the encoder uses one motion vector and no reference identifier.

20. The method of claim 16 wherein the determining the type value includes choosing the type value based at least in part upon packet loss information from a communications channel.

21. The method of claim 16 wherein the determining the type value includes choosing the type value based at least in part upon magnitude of prediction error.

22. The method of claim 16 wherein, according to the first inter encoding mode and according to the second inter encoding mode, the encoder performs motion compensation in a single temporal direction.