Systems and methods for encoding and playing back video at different frame rates using enhancement layers

• US 10,321,168 B2
• Filed: 01/05/2018
• Issued: 06/11/2019
• Est. Priority Date: 04/05/2014
• Status: Active Grant

• Current Assignee: DivX, LLC (f/k/a DivX CF Holdings LLC) (FIG LLC (d/b/a Fortress Investment Group LLC))
• Original Assignee: DivX, LLC (f/k/a DivX CF Holdings LLC) (FIG LLC (d/b/a Fortress Investment Group LLC))
• Inventors: van der Schaar, Auke Sjoerd, Funnell, John
• Primary Examiner(s): Schnurr, John R
• Application Number:
  US15/863,456
• Publication Number:
  US 20180131980A1
• Time in Patent Office: 522 Days
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/234381 by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
• CPC Class Codes: H04N 21/25825 involving client display capabilities, e.g. screen resolution of a mobile phone optimising the visualisation of content during browsing in the Internet G06F16/9577; processing of terminal status or physical abilities in wireless networks H04W8/22; authentication in wireless network security H04W12/06
• CPC Class Codes: H04N 21/6587 Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/85406 involving a specific file format, e.g. MP4 format

Abstract

Systems and methods in accordance with embodiments of this invention provide for encoding and playing back video at different frame rates using enhancement layers. In a number of embodiments, video is encoded using temporal scalability involving the creation of a base layer at a first frame rate and an enhancement layer including additional frames enabling playback at a second higher frame rate. The second higher frame rate can also be referred to as an enhanced frame rate. In a number of embodiments, the base and enhancement layers are stored in one or more container files that contain metadata describing the enhancement layer. Based on the capabilities of a playback device, it can select the particular frame rate at which to playback encoded video.

Claims

1. A content distribution server comprising:

a processor configured to communicate with a memory, where the memory contains a server application;

wherein the server application configures the processor to;

receive a request for video having a plurality of access units with temporal scalability, each access unit comprising a temporal identifier identifying a particular layer from a plurality of layers associated with the access unit;

wherein the video comprises a set of layers comprising a base layer of access units stored in a first container file and ordered in decode order to provide playback at a base frame rate and at least one enhancement layer with additional access units stored in a second container file separate from the first container file and ordered in decode order to provide playback at an enhanced frame rate;

send different sets of layers of the video, wherein (i) the base layer is sent to a playback device capable of playing back the base layer at the base frame rate and (ii) the base layer and at least one enhancement layer are sent to a playback device capable of playing back the plurality of layers at the enhanced frame rate, wherein the video includes metadata that indicates how to combine in decode order the access units stored in the second container file with access units stored in the first container file for playback at the enhanced frame rate.

2. The content distribution server of claim 1, wherein the temporal identifier of an access unit is stored in an initial set of bits of the access unit.

3. The content distribution server of claim 1, wherein the server application configures the processor to insert a temporal identifier for a set of access units of the plurality of access units of the video.

4. The content distribution server of claim 1, wherein the server application configures the processor to insert a metadata tag providing information regarding the layers available and the frame rate of each layer.

5. The content distribution server of claim 1, further comprising:

receiving a request for streaming of the video;

determining playback capabilities of a device requesting the video;

streaming different sets of layers of the video based on the playback capabilities of the device, wherein (i) the base layer is streamed to a playback device capable of playing back the base layer and (ii) the base layer and at least one enhancement layer is streamed to a playback device capable of streaming the plurality of layers.

6. The content distribution server of claim 5, wherein streaming the different sets of layers comprises merging the different layers using the metadata providing an order of the temporal identifiers of the plurality of access units of the video.

7. The content distribution server of claim 1, wherein the server application configures the processor to:

receive a request for downloading of the video from a playback device;

providing the encoded video to the playback device, wherein (i) the base layer is provided to a playback device capable of playing back the base layer and (ii) the base layer and at least one enhancement layer is provided to a playback device capable of playing back the plurality of layers.

8. The content distribution server of claim 1, wherein the frame rate of the at least one enhancement layer is an integer multiple of the base layer.

9. The content distribution server of claim 1, wherein the server application configures the processor to store each layer as a separate stream.

10. The content distribution server of claim 1, wherein the metadata specifies an ordered sequence of temporal identifiers for combining access units from the base layer with the additional access units from the at least one enhancement layer.

11. The content distribution server of claim 10, wherein the server application further configures the processor to:

determine that the playback device is capable of playing back video at the enhanced frame rate; and

combine, using the metadata, the access units from the base layer with the additional access units of the enhancement layer.

12. The content distribution server of claim 10, wherein the server application further configures the processor to:

determine that the playback device is capable of playing back video at the enhanced frame rate;

send the access units from the base layer; and

send the additional access units of the enhancement layer and the metadata to the playback device.

13. A source encoder, comprising:

a processor configured by a source encoder application to;

obtain video providing temporal scalability and having a plurality of access units ordered based on a decoding order for decoding the video; and

encode the video into a set of layers comprising (1) a base layer of access units sequenced in decode order to provide a base frame rate and (2) at least one enhancement layer with additional access units ordered in decode order to provide an enhanced frame rate that is greater than the base frame rate, wherein the encoding comprises;

(i) using a temporal identifier of each access unit to determine the particular layer associated with the access unit and (ii) retaining as metadata an order of the temporal identifiers of the plurality of access units of the video;

store the base layer in a first container file;

store the at least one enhancement layer in a second container file separate from the first container file;

receive a request for the video;

provide different sets of layers of the video, wherein (i) the base layer is provided to a playback device capable of playing back the base layer and (ii) the base layer and at least one enhancement layer is provided to a playback device capable of playing back the plurality of layers at the enhanced frame rate, wherein the video includes the metadata that indicates how to combine in decode order the access units stored in the second container file with access units stored in the first container file for playback at the enhanced frame rate.

14. The source encoder of claim 13, wherein the temporal identifier of an access unit is stored in a header of the access unit.

15. The source encoder of claim 13, further comprising inserting a temporal identifier for a set of access units of the plurality of access units of the video.

16. The source encoder of claim 13, further comprising inserting a metadata tag providing information regarding the layers available and the frame rate of each layer.

17. The source encoder of claim 13, further comprising:

determining that the playback device is capable of playing back video at the enhanced frame rate; and

combining, using the metadata, the access units from the base layer with the additional access units of the enhancement layer.

18. The source encoder of claim 13, further comprising:

determining that the playback device is capable of playing back video at the enhanced frame rate;

sending the access units from the base layer; and

sending the additional access units of the enhancement layer and the metadata to the playback device.

19. The source encoder of claim 13, further comprising:

receiving a request for streaming of the video;

streaming different sets of layers of the video based on the playback capabilities of the device, wherein (i) the base layer is streamed to a playback device capable of playing back the base layer and (ii) the base layer and at least one enhancement layer is streamed to a playback device capable of streaming the plurality of layers.

20. The source encoder of claim 19, wherein streaming the different sets of layers comprises merging the different layers using the metadata providing a retained order of the temporal identifiers of the plurality of access units of the video.

21. The source encoder of claim 13, further comprising:

receiving a request for downloading of the video from a playback device;

providing the encoded video to the playback device, wherein (i) the base layer is provided to a playback device capable of playing back the base layer and (ii) the base layer and at least one enhancement layer is provided to a playback device capable of playing back the plurality of layers.