Layered video encoding and decoding

• US 9,609,338 B2
• Filed: 10/25/2013
• Issued: 03/28/2017
• Est. Priority Date: 10/25/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Fu, Jingjing, Lu, Yan, Li, Shipeng
• Primary Examiner(s): Chen, Cai
• Application Number:
  US14/063,476
• Publication Number:
  US 20150117545A1
• Time in Patent Office: 1,250 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: H04N 19/103 Selection of coding mode or of prediction mode
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/174 the region being a slice, e.g. a line of blocks or a group of blocks
• CPC Class Codes: H04N 19/187 the unit being a scalable video layer
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/60 using transform coding

Abstract

In some examples, a layered encoding component and a layered decoding component provide for different ways to encode and decode, respectively, video streams transmitted between devices. For instance, in encoding a video stream, a layered encoding component may analyze the content of successive video frames and determine different types of encoding techniques to use for different ones of the video frames. Further, in some cases, some of the encoding techniques may be used on less than an entire video frame. In another example, in decoding a video stream, a layered decoding component may receive video frames encoded with different types of encoding. The layered decoding component may decode the differently encoded video frames and combine them to reconstruct a video stream.

Claims

1. A system comprising:

one or more computing nodes, each comprising at least one processor and memory, wherein the memory includes computer storage media storing computer readable instructions that when executed by the at least one processor perform operations comprising;

generating a first encoding of a first frame of a video stream according to a first layer encoding, wherein the first frame references a second frame previous to the first frame in the video stream and the second frame is encoded according to the first layer encoding;

determining that an amount of change between one or more regions of a third frame of a video stream corresponding with one or more regions of a fourth frame previous to the third frame in the video stream is greater than a threshold amount of change;

determining, based at least partly on the amount of change, that the one or more regions of the third frame are suitable for a second layer encoding;

generating a second encoding of the third frame according to the second layer encoding, wherein the third frame references the fourth frame and the fourth frame is encoded according to the second layer encoding;

detecting, after generating the first encoding and the second encoding, changes to network conditions during transmission of additional frames of the video stream;

generating, based at least partly on the changes to the network conditions, a first additional encoding for a first additional frame encoded according to the first layer encoding and referencing a second additional frame previous to the first additional frame in the video stream with the second additional frame being encoded according to the first layer encoding; and

generating, based at least partly on the changes to the network conditions, a second additional encoding for a third additional frame encoded according to the second layer encoding and referencing the second additional frame, wherein the third additional frame is subsequent to the first additional frame and the second additional frame in the video stream.

2. The system as recited in claim 1, wherein remaining regions of the third frame other than the one or more regions of the third frame are not encoded according to the first encoding or the second encoding.

3. The system as recited in claim 1, wherein a first region of the third frame is encoded with a pixel-domain coding technique and a second region of the third frame is encoded with a transform-based coding technique.

4. The system as recited in claim 1, wherein the operations further comprise:

generating a reconstructed frame of a fifth frame of an additional video stream, the fifth frame including one or more regions encoded according to the first layer encoding and one or more additional regions encoded according to the second layer encoding.

5. A method comprising:

under control of one or more computing devices configured with executable instructions;

generating a first encoding of a first frame of a video stream according to a first layer encoding, wherein the first frame references a second frame previous to the first frame in the video stream and encoded according to the first layer encoding;

determining that one or more regions of a third frame are suitable for a second layer encoding;

generating a second encoding of the one or more regions of the third frame according to the second layer encoding, wherein the third frame references a fourth frame previous to the third frame in the video stream and encoded according to the second layer encoding;

detecting changes to conditions of a network during transmission of additional frames of the video stream;

generating, based at least partly on the changes to the network conditions, a first additional encoding of a first additional frame encoded according to the first layer encoding, wherein the first additional frame references a second additional frame previous to the first additional frame in the video stream and encoded according to the first layer encoding; and

generating, based at least partly on the changes to the network conditions, a second additional encoding of a third additional frame encoded according to the second layer encoding, wherein the third additional frame references the second additional frame.

6. The method as recited in claim 5, wherein detecting changes to the conditions of the network includes determining that the network is unstable.

7. The method as recited in claim 5, wherein the first encoding and the second encoding are generated based at least partly on expected conditions of the network before the first encoding and the second encoding are generated.

8. The method as recited in claim 5, further comprising transmitting the first encoding separately from the second encoding.

9. The method as recited in claim 5, further comprising transmitting the first encoding and the second encoding in an arbitrary order, wherein the first encoding includes first metadata specifying a position of the first frame within the video stream, and wherein the second encoding includes second metadata specifying a position of the third frame within the video stream.

10. The method as recited in claim 5, wherein generating the first encoding and generating the second encoding are performed in parallel.

11. The method as recited in claim 5, wherein generating the second encoding comprises generating metadata specifying a size and location for each of the one or more regions of the third frame.

12. The method as recited in claim 11, wherein the metadata further specifies one or more encoding techniques used in generating the second encoding.

13. The method as recited in claim 12, wherein the metadata further specifies that the fourth frame is a reference video frame upon which to at least partly base a reconstruction of the third frame.

14. The method as recited in claim 13, further comprising:

detecting additional changes to the conditions of the network during transmission of the additional frames of the video stream; and

determining that the network is stable based at least partly on the additional changes to the conditions of the network.

15. The method as recited in claim 5, further comprising determining that an amount of change between the one or more regions of the third frame and one or more additional regions of the fourth frame is at least in part based on a luminance histogram of characteristics of image values of the third frame.

16. The method as recited in claim 15, wherein determining that the one or more regions of the third frame are suitable for the second layer encoding is at least in part based on a sparsity of a distribution of the luminance histogram.

17. A method comprising:

performing, by one or more computing devices;

receiving a first frame of a video stream, wherein the first frame is encoded with a first encoding technique;

decoding the first frame to generate a reconstructed first frame based at least partly on a second frame previous to the first frame in the video stream, wherein the second frame is encoded with the first encoding technique;

receiving a third frame of the video stream;

determining that a portion of regions of the third frame are encoded with a second encoding technique different from the first encoding technique;

decoding the portion of the regions of the third frame to generate a reconstructed third frame based at least partly on a fourth frame previous to the third frame in the video stream, wherein the fourth frame is encoded with the second encoding technique;

detecting changes to network conditions during transmission of additional frames of the video stream;

decoding, based at least partly on the changes to the network conditions, a first additional frame encoded according to the first layer encoding based at least partly on a second additional frame previous to the first additional frame, wherein the second additional frame is encoded according to the first layer encoding; and

decoding, based at least partly on the changes to the network conditions, a third additional frame encoded according to the second layer encoding based at least partly on the second additional frame, wherein the second additional frame is previous to the third additional frame in the video stream.

18. The method as recited in claim 17, wherein generating the reconstructed first frame comprises:

extracting metadata specifying a size and location for each of one or more regions of the first frame, wherein the metadata further specifies that the second frame is a reference video frame; and

generating, at least in part, the reconstructed first frame from the one or more regions combined with one or more additional regions from the reference video frame.

19. The method as recited in claim 17, wherein:

decoding the portion of the regions of the third frame produces a decoded portion of the third frame; and

generating the reconstructed third frame includes combining copies of remaining regions of the third frame that are not encoded with the decoded portion of the third frame.

20. The method as recited in claim 17, wherein decoding the first frame further comprises:

extracting metadata specifying an encoding technique used to encode the first frame; and

decoding the first frame according to the encoding technique.