Layered Video Encoding and Decoding

• US 20150117545A1
• Filed: 10/25/2013
• Published: 04/30/2015
• Est. Priority Date: 10/25/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Fu, Jingjing, Lu, Yan, Li, Shipeng
• Granted Patent: US 9,609,338 B2
• Time in Patent Office: Days
• US Class Current: 375/240.18
• CPC Class Codes: H04N 19/103 Selection of coding mode or of prediction mode
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/174 the region being a slice, e.g. a line of blocks or a group of blocks
• CPC Class Codes: H04N 19/187 the unit being a scalable video layer
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/60 using transform coding

Abstract

In some examples, a layered encoding component and a layered decoding component provide for different ways to encode and decode, respectively, video streams transmitted between devices. For instance, in encoding a video stream, a layered encoding component may analyze the content of successive video frames and determine different types of encoding techniques to use for different ones of the video frames. Further, in some cases, some of the encoding techniques may be used on less than an entire video frame. In another example, in decoding a video stream, a layered decoding component may receive video frames encoded with different types of encoding. The layered decoding component may decode the differently encoded video frames and combine them to reconstruct a video stream.

Claims

1. A system comprising:

one or more computing nodes, each comprising at least one processor and memory, wherein the one or more computing nodes are configured to implement an encoding component and a decoding component,

wherein the encoding component is configured to;

determine an amount of change between one or more regions of first image data with corresponding one or more regions of second image data;

determine, based at least partly on the amount of change, that the one or more regions of the first image data are suitable for a first layer encoding;

generate a first encoding of the one or more regions of the first image data; and

generate a second encoding of the second image data; and

wherein the decoding component is configured to;

decode the first encoding to generate a reconstructed first image data,

decode the second encoding to generate a reconstructed second image data, and

generate a video stream comprising the reconstructed first image data and the reconstructed second image data.

2. The system as recited in claim 1, wherein to generate the first encoding of the one or more regions of the first image data, the encoding component is further configured to not base the first encoding on a region or regions other than the determined one or more regions of the first image data.

3. The system as recited in claim 1, wherein to generate the first encoding of the one or more regions of the first image data, the encoding component is further configured to encode a first region of the determined one or more regions of the first image data with a pixel-domain coding technique and to encode a second region of the determined one or more regions of the first image data with a transform-based coding technique.

4. The system as recited in claim 1, wherein to generate the reconstructed first image data, the decoding component is further configured to:

reconstruct the one or more regions of the first image data based partly on a decoding of the first encoding;

determine one or more regions of the second image data to generate an entire video frame; and

generate an entire video frame based at least partly on the reconstructed one or more regions of the first image data combined with the determined one or more regions of the second image data.

5. A method comprising:

under control of one or more computing devices configured with executable instructions;

receiving a first frame and a second frame of a video stream;

generating a first encoding of the first frame;

determining an amount of change between one or more regions of the first frame and corresponding one or more regions of the second frame;

determining, based at least partly on the amount of change, that the one or more regions of the second frame are suitable for a second layer encoding; and

generating a second encoding of the determined one or more regions of the second frame suitable for second layer encoding.

6. The method as recited in claim 5, further comprising:

determining, based at least partly on the amount of change, that a second one or more regions of the second frame are suitable for a third layer encoding, wherein the one or more regions suitable for the second layer encoding are a first one or more regions suitable for the second layer encoding, and wherein the first one or more regions are distinct from the second one or more regions.

7. The method as recited in claim 6, wherein the second encoding further comprises a third layer encoding of the second one or more regions of the first frame.

8. The method as recited in claim 5, further comprising transmitting the first encoding separately from the second encoding.

9. The method as recited in claim 5, further comprising transmitting the first encoding and the second encoding in an arbitrary order, wherein the first encoding includes first metadata specifying a position of the first frame within the video stream, and wherein the second encoding includes second metadata specifying a position of the second frame within the video stream.

10. The method as recited in claim 5, wherein the generating the first encoding and the generating the second encoding are performed in parallel.

11. The method as recited in claim 5, wherein the generating the second encoding comprises generating metadata specifying a size and location for each of the one or more regions of the second frame determined to be suitable for a second layer encoding.

12. The method as recited in claim 11, wherein the metadata further specifies one or more encoding techniques used in generating the second encoding.

13. The method as recited in claim 12, wherein the metadata further specifies a reference video frame upon which to at least partly base a reconstruction of the second frame.

14. The method as recited in claim 13, further comprising:

determining one or more network conditions of a network over which the second encoding is to be transmitted; and

based at least in part on the one or more network conditions, determine whether the reference video frame is second layer encoded video frame or a first layer encoded video frame.

15. The method as recited in claim 5, wherein the determining the amount of change between the one or more regions of the first frame and corresponding one or more regions of the second frame is at least in part based on a luminance histogram of characteristics of image values of the second frame.

16. The method as recited in claim 15, wherein the determining that the one or more regions of the second frame are suitable for the second layer encoding is at least in part based on a sparsity of a distribution of the luminance histogram.

17. A method comprising:

performing, by one or more computing devices;

receiving a first encoding of a first frame of a video stream;

decoding the first encoding to generate a reconstructed first frame;

receiving a second encoding of a second frame of the video stream, wherein the second encoding is based at least partly on an amount of change between one or more corresponding regions of the first frame and the second frame;

decoding the second encoding to generate a reconstructed second frame; and

generating a reconstructed video stream comprising the reconstructed first frame and the reconstructed second frame.

18. The method as recited in claim 17, wherein generating the reconstructed first frame comprises:

extracting, from the first encoding, metadata specifying a size and location for each of one or more regions of the first frame, wherein the metadata further specifies a reference video frame; and

generating, at least in part, the reconstructed first frame from the one or more regions combined with one or more regions from the reference video frame.

19. The method as recited in claim 17, wherein:

the first encoding is encoded with a first encoding technique;

the second encoding is encoded with a second encoding technique; and

the first encoding technique is different from the second encoding technique.

20. The method as recited in claim 17, the decoding the first encoding further comprises:

extracting, from the first encoding, metadata specifying an encoding technique used to encode the first encoding; and

decoding the first encoding according to the encoding technique.