Systems and methods for encoding and playing back video at different frame rates using enhancement layers

• US 9,866,878 B2
• Filed: 04/06/2015
• Issued: 01/09/2018
• Est. Priority Date: 04/05/2014
• Status: Active Grant

• Current Assignee: Sonic IP Incorporated (Endeavor Group Holdings Inc.)
• Original Assignee: Sonic IP Incorporated (Endeavor Group Holdings Inc.)
• Inventors: van der Schaar, Auke Sjoerd, Funnell, John
• Primary Examiner(s): Schnurr, John
• Application Number:
  US14/679,469
• Publication Number:
  US 20150288996A1
• Time in Patent Office: 1,009 Days
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/234381 by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
• CPC Class Codes: H04N 21/25825 involving client display capabilities, e.g. screen resolution of a mobile phone optimising the visualisation of content during browsing in the Internet G06F16/9577; processing of terminal status or physical abilities in wireless networks H04W8/22; authentication in wireless network security H04W12/06
• CPC Class Codes: H04N 21/6587 Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/85406 involving a specific file format, e.g. MP4 format

Abstract

Systems and methods in accordance with embodiments of this invention provide for encoding and playing back video at different frame rates using enhancement layers. In a number of embodiments, video is encoded using temporal scalability involving the creation of a base layer at a first frame rate and an enhancement layer including additional frames enabling playback at a second higher frame rate. The second higher frame rate can also be referred to as an enhanced frame rate. In a number of embodiments, the base and enhancement layers are stored in one or more container files that contain metadata describing the enhancement layer. Based on the capabilities of a playback device, it can select the particular frame rate at which to playback encoded video.

Claims

1. A non-transitory machine readable medium containing processor instructions, where execution of the instructions by a processor causes the processor to perform a process comprising:

obtain video comprising a merged plurality of access units with temporal scalability, each access unit in the merged plurality of access units associated with a particular layer of a plurality of temporal layers based on a temporal identifier of the access unit that identifies the particular layer, wherein the temporal layers include a base layer and one or more enhancement layers;

obtain an ordered sequence of access units ordered based on a decoding order for decoding the video and generating metadata specifying an ordered sequence of temporal identifiers ordered based on, for each access unit in the ordered sequence of access units, identifying and retaining the temporal identifier associated with the access unit;

separate, based on the temporal identifier of each access unit in the ordered sequence of access units, the ordered sequence of access units into (1) a base layer of access units sequenced in decode order to provide a base frame rate and (2) at least one enhancement layer with additional access units ordered in decode order to provide an enhanced frame rate, wherein at least some of the access units in the enhancement layer are encoded based upon dependencies to access units in the base layer, and wherein access units in the base layer are encoded without dependencies upon access units in the enhancement layer; and

storing the base layer in a first container file; and

storing the enhancement layer, including the generated metadata specifying the ordered sequence of temporal identifiers, in a second container file separate from the first container file, wherein the metadata is used to recombine in decode order the access units stored in the second container file with access units stored in the first container file for playback at the enhanced frame rate.

2. The non-transitory machine readable medium of claim 1, wherein the temporal identifier of an access unit is stored in an initial set of bits of the access unit.

3. The non-transitory machine readable medium of claim 1, further comprising inserting a temporal identifier for a set of access units of the plurality of access units of the video.

4. The non-transitory machine readable medium of claim 1, further comprising inserting a metadata tag providing information regarding the layers available and the frame rate of each layer.

5. The non-transitory machine readable medium of claim 1, further comprising:

receiving a request for streaming of the video;

determining playback capabilities of a device requesting the video;

streaming different sets of layers of the video based on the playback capabilities of the device, wherein (i) the base layer is streamed to a playback device capable of playing back the base layer and (ii) the base layer and at least one enhancement layer is streamed to a playback device capable of streaming the plurality of layers.

6. The non-transitory machine readable medium of claim 5, wherein streaming the different sets of layers comprises merging the different layers using the metadata that specifies the sequence for combining access units from the base layer with access units from the enhancement layer.

7. The non-transitory machine readable medium of claim 1, further comprising:

receiving a request for downloading of the video from a playback device;

providing the encoded video to the playback device.

8. The non-transitory machine readable medium of claim 1, wherein the frame rate of the at least one enhancement layer is an integer multiple of the base layer.

9. The non-transitory machine readable medium of claim 1, further comprising storing each layer as a separate stream.

10. The non-transitory machine readable medium of claim 1, further comprising storing each layer in a separate container file.

11. A source encoder, comprising:

a processor configured by a source encoder application to;

obtain video comprising a merged plurality of access units with temporal scalability, each access unit in the merged plurality of access units associated with a particular layer of a plurality of temporal layers based on a temporal identifier of the access unit that identifies the particular layer, wherein the temporal layers include a base layer and one or more enhancement layer;

obtain an ordered sequence of access units ordered based on a decoding order for decoding the video and generating metadata specifying an ordered sequence of temporal identifiers ordered based on, for each access unit in the ordered sequence of access units, identifying and retaining the temporal identifier associated with the access unit;

separate, based on the temporal identifier of each access unit in the ordered sequence of access units, the ordered sequence of access units into (1) a base layer of access units sequenced in decode order to provide a base frame rate and (2) at least one enhancement layer with additional access units ordered in decode order to provide an enhanced frame rate, wherein at least some of the access units in the enhancement layer are encoded based upon dependencies to access units in the base layer, and wherein access units in the base layer are encoded without dependencies upon access units in the enhancement layer;

storing the base layer in a first container file; and

storing the enhancement layer, including the generated metadata specifying the ordered sequence of temporal identifiers, in a second container file separate from the first container file, wherein the metadata is used to recombine in decoding order the access units stored in the second container file with access units stored in the first container file for playback at the enhanced frame rate.

12. The source encoder of claim 11, wherein the temporal identifier of an access unit is stored in a header of the access unit.

13. The source encoder of claim 11, further comprising inserting a temporal identifier for a set of access units of the plurality of access units of the video.

14. The source encoder of claim 11, further comprising inserting a metadata tag providing information regarding the layers available and the frame rate of each layer.

15. A playback device configured to playback video comprising:

a processor configured to communicate with a memory, where the memory contains a client application;

wherein the client application configures the processor to;

obtain a first container file comprising a base layer of access units ordered in decode order for playback at a base frame rate;

obtain, for a device capable of playing back an enhancement layer, a second container file comprising (1) at least one enhancement layer with additional access units ordered in decode order for playback at an enhanced frame rate and (2) metadata specifying an ordered sequence of temporal identifiers for combining, in decode order, access units from the base layer with access units from the at least one enhancement layer for playback at the enhanced frame rate, wherein each access unit includes a temporal identifier identifying a particular layer associated with the access unit;

combine the base layer of access units with the at least one enhancement layer based on the metadata obtained from the second container file to obtain a combined sequence of access units ordered in a decoding order at the enhanced frame rate;

wherein at least some of the access units in the at least enhancement layer stored in the second container file are decoded based upon dependencies to access units in the base layer stored in the first container file, and wherein access units in the base layer that are stored in the first container file are decoded without dependencies upon access units in the enhancement layer.

16. The playback device of claim 15, wherein the client application further configures the processor to:

request streaming of the video to a playback server and provide information regarding playback capabilities of the playback device to the playback server;

receive the first container file comprising a first stream comprising the base layer of the video and the second container file comprising a second stream comprising the at least one enhancement layer of the video.

17. The playback device of claim 15, wherein the client application further configures the processor to:

request downloading of the video to a content distribution server and provide playback capabilities of the playback device to the content distribution server;

receive the video from the content distribution server, wherein a first device receives the first container file comprising the base layer when the first device is capable of playing back the video at the base frame rate and a second device receives the first container file and the second container file comprising the base layer and at least one enhancement layer when the second device is capable of playing back the video at the enhanced frame rate.

18. The playback device of claim 15, wherein the client application further configures the processor to:

retrieve a metadata tag from the video identifying the layers available and the frame rate of each layer;

determining a set of layers to playback based on the information in the metadata tag.

19. The playback device of claim 15, wherein the client application further configures the processor to:

identify the temporal identifier of an access unit by parsing an initial set of bytes of the access unit.