Systems, methods, and computer program products for load adaptive streaming

• US 8,914,836 B2
• Filed: 09/28/2012
• Issued: 12/16/2014
• Est. Priority Date: 09/28/2012
• Status: Active Grant

• Current Assignee: Sonic IP Incorporated (Endeavor Group Holdings Inc.)
• Original Assignee: Sonic IP Incorporated (Endeavor Group Holdings Inc.)
• Inventors: Shivadas, Abhishek, Van Der Schaar, Auke S.
• Primary Examiner(s): Bruckart, Benjamin R
• Assistant Examiner(s): NGUYEN, TIEN M
• Application Number:
  US13/631,118
• Publication Number:
  US 20140096171A1
• Time in Patent Office: 809 Days
• Field of Search: 725/109, 725/110, 725/115, 725/145
• US Class Current: 725/109
• CPC Class Codes: H04N 21/23439 for generating different versions
• CPC Class Codes: H04N 21/27 Server based end-user applications
• CPC Class Codes: H04N 21/4143 embedded in a Personal Computer [PC]
• CPC Class Codes: H04N 21/44004 involving video buffer management, e.g. video decoder buffer or video display buffer
• CPC Class Codes: H04N 21/44209 Monitoring of downstream path of the transmission network originating from a server, e.g. bandwidth variations of a wireless network arrangements for maintenance or administration in data switching networks involving bandwidth and capacity management H04L41/0896
• CPC Class Codes: H04N 21/4621 Controlling the complexity of the content stream or additional data, e.g. lowering the resolution or bit-rate of the video stream for a mobile client with a small screen arrangements for using the results of monitoring on user's side in broadcast systems H04H60/65; flow control in packet networks H04L47/10

Abstract

Methods, systems, and computer program products that allow for the load adaptive downloading, processing, and rendering of video by a video processing client, where the downloading is adaptive to the processing load of a computing system on which a video processing client operates.

Claims

1. A playback device configured to perform load and rate adaptive streaming, comprising:

a processor configured via a load and rate adaptive streaming application to request segments of video from among a plurality of bitstreams, where the plurality of bitstreams are encoded at different decode levels in which resolution and bitrate vary between different decode levels;

wherein the load and rate adaptive streaming application further configures the processor to;

download and buffer a segment of video from a first bitstream at an initial decode level, where the segment of video comprises at least a plurality of frames of encoded video that are buffered in a coded picture buffer within the playback device;

perform video processing to decode the plurality of frames of encoded video in the coded picture buffer in order to generate a plurality of frames of decoded video;

store the plurality of frames of decoded video in a frame queue for display;

accumulate measurements in a sample collection buffer, wherein the accumulated measurements comprise (i) a frame delivery time for each frame of decoded video in the frame queue and (ii) a number of dropped frames over a predefined period;

when the number of dropped frames over the predefined period exceeds a frame drop threshold, flush a portion of the frames of encoded video buffered in the coded picture buffer; and

when the accumulated measurements of the frame delivery times for the frames of the decoded video in the frame queue exceed a first threshold, download another segment of video from a second bitstream having a lower decode level than the decode level of the first bitstream.

2. The playback device of claim 1, wherein the processor is configured to download another segment of video from the second bitstream at the lower decode level after a minimum observation period.

3. The playback device of claim 1, wherein the load and rate adaptive streaming application further-configures the processor to:

when the playback device flushes the portion of the frames of encoded video buffered in the coded picture buffer, download a segment of video from the second bitstream at the lower decode level including a particular frame of encoded video that has a same play time as an encoded frame of video flushed from the coded picture buffer.

4. The playback device of claim 1, wherein the load and rate adaptive streaming application further configures the processor to:

accumulate and maintain a sorted list of frame delivery times in the sample collection buffer; and

determine when the accumulated measurements of the frame delivery times for the frames of the decoded video in the frame queue exceed the first threshold by comparing a selected range of the sorted list to the first threshold.

5. The playback device of claim 1, wherein the number of frames flushed from the coded picture buffer is dependent on the number of dropped frames over the predefined period.

6. The playback device of claim 1, wherein the load and rate adaptive streaming application further configures the processor to:

when the accumulated measurements of the frame delivery times for the frames of the decoded video in the frame queue are below a second threshold, download another segment of video from a third bitstream at a higher decode level; and

automatically calculate the second threshold such that the higher decode level does not exceed performance capabilities of the playback device.

7. The playback device of claim 6, wherein the load and rate adaptive streaming application further configures the processor to download another segment of video from the third bitstream at the higher decode level after a minimum observation period.

8. The playback device of claim 6, wherein the another segment of video is downloaded at the higher decoding level only when an average of the accumulated measurements of the frame delivery times is less than the second threshold by a predetermined amount.

9. The playback device of claim 6, wherein the load and rate adaptive streaming application further configures the processor to dynamically derive one or more of the first and second thresholds.

10. The playback device of claim 6, wherein one or more of the first and second thresholds is determined statically by a remote server in communication with the playback device.

11. The playback device of claim 6, wherein the load and rate adaptive streaming application further configures the processor to store one or more of the first and second thresholds locally in a storage of at the playback device for retrieval upon start up of the playback device.

12. The playback device of claim 1, wherein the initial decode level is equal to a maximum previous decode level.

13. A non-transitory machine readable medium storing a load and rate adaptive streaming application, the load and rate adaptive streaming application comprising sets of instructions for:

requesting segments of video from among a plurality of bitstreams, where the plurality of bitstreams are encoded at different decode levels in which resolution and bitrate vary between different decode levels;

downloading and buffering a segment of video from a first bitstream at an initial decode level, where the segment of video comprises at least a plurality of frames of encoded video that are buffered in a coded picture buffer within the playback device;

performing video processing to decode the plurality of frames of encoded video in the coded picture buffer in order to generate a plurality of frames of decoded video;

storing the plurality of frames of decoded video in a frame queue for display;

accumulating measurements in a sample collection buffer, wherein the accumulated measurements comprise (i) a frame delivery time for each frame of decoded video in the frame queue and (ii) a number of dropped frames over a predefined period;

when the number of dropped frames over the predefined period exceeds a frame drop threshold, flushing a portion of the frames of encoded video buffered in the coded picture buffer; and

when the accumulated measurements of the frame delivery times for the frames of the decoded video in the frame queue exceed a first threshold, downloading another segment of video from a second bitstream at a lower decode level having a lower resolution and bitrate than the resolution and bitrate of the initial decode level of the first bitstream.

14. The non-transitory machine readable medium of claim 13, wherein the set of instructions for downloading another segment of video from the second bitstream at the lower decode level is executed after a minimum observation period.

15. The non-transitory machine readable medium of claim 13, wherein the load and rate adaptive streaming application further comprises sets of instructions for:

when the playback device flushes the portion of the frames of encoded video buffered in the coded picture buffer, downloading a segment of video from the second bitstream at the lower decode level including a particular frame of encoded video that has a same play time as an encoded frame of video flushed from the coded picture buffer.

16. The non-transitory machine readable medium of claim 13, wherein the load and rate adaptive streaming application further comprises sets of instructions for:

accumulating and maintaining a sorted list of frame delivery times in the sample collection buffer; and

determining when the accumulated measurements of the frame delivery times for the frames of the decoded video in the frame queue exceed the first threshold by comparing a selected range of the sorted list to the first threshold.

17. The non-transitory machine readable medium of claim 13, wherein the number of frames flushed from the coded picture buffer is dependent on the number of dropped frames over the predefined period.

18. The non-transitory machine readable medium of claim 13, wherein the load and rate adaptive streaming application further comprises sets of instructions for:

when the accumulated measurements of the frame delivery times for the frames of the decoded video in the frame queue are below a second threshold, downloading another segment of video from a third bitstream at a higher decode level; and

automatically calculating the second threshold such that the higher decode level does not exceed performance capabilities of the playback device.

19. The non-transitory machine readable medium of claim 18, wherein the downloading of the another segment of video at the higher decode level takes place after a minimum observation period.

20. The non-transitory machine readable medium of claim 18, wherein the another segment of video is downloaded at the higher decoding level only when an average of the accumulated measurements of the frame delivery times is less than the second threshold by a predetermined amount.

21. The non-transitory machine readable medium of claim 18, wherein the load and rate adaptive streaming application further comprises sets of instructions for dynamically deriving one or more of the first and second thresholds.

22. The non-transitory machine readable medium of claim 18, wherein one or more of the first and second thresholds is determined statically by a remote server in communication with the playback device.

23. The non-transitory machine readable medium of claim 18, wherein the load and rate adaptive streaming application further comprises sets of instructions for storing one or more of the first and second thresholds locally in a storage of the playback device for retrieval upon start up of the playback device.

24. The non-transitory machine readable medium of claim 13, wherein the initial decode level is equal to a maximum previous decode level.

25. A method for performing a load and rate adaptive streaming process, the method performed by a playback device, the method comprising:

requesting segments of video from among a plurality of bitstreams using a playback device, where the plurality of bitstreams are encoded at different decode levels in which resolution and bitrate vary between different decode levels;

downloading and buffering a segment of video from a first bitstream at an initial decode level using the playback device, where the segment of video comprises at least a plurality of frames of encoded video that are buffered in a coded picture buffer within the playback device;

performing video processing using the playback device to decode the plurality of frames of encoded video in the coded picture buffer in order to generate a plurality of frames of decoded video;

storing the plurality of frames of decoded video in a frame queue for display using the playback device;

accumulating measurements in a sample collection buffer using the playback device, wherein the accumulated measurements comprise (i) a frame delivery time for each frame of decoded video in the frame queue and (ii) a number of dropped frames over a predefined period;

when the number of dropped frames over the predefined period exceeds a frame drop threshold, flushing a portion of the frames of encoded video buffered in the coded picture buffer using the playback device; and

when the accumulated measurements of the frame delivery times for the frames of the decoded video in the frame queue exceed a first threshold, downloading another segment of video from a second bitstream at a lower decode level than the decode level of the first bitstream using the playback device.

26. The method of claim 25, wherein the downloading of the another segment of video at the lower decode level takes place after a minimum observation period.

27. The method of claim 25 further comprising:

when the playback device flushes the portion of the frames of encoded video buffered in the coded picture buffer, downloading a segment of video from the second bitstream at the lower decode level including a particular frame of encoded video that has a same play time as an encoded frame of video flushed from the coded picture buffer.

28. The method of claim 25 further comprising:

accumulating and maintaining a sorted list of frame delivery times in the sample collection buffer using the playback device; and

determining when the accumulated measurements of the frame delivery times for the frames of the decoded video in the frame queue exceed the first threshold using the playback device by comparing a selected range of the sorted list to the first threshold.

29. The method of claim 25, wherein the number of frames flushed from the coded picture buffer is dependent on the number of dropped frames over the predefined period.

30. The method of claim 25, further comprising:

when the accumulated measurements of the frame delivery times for the frames of the decoded video in the frame queue are below a second threshold, downloading another segment of video from a third bitstream at a higher decode level using the playback device; and

automatically calculating the second threshold using the playback device such that the higher decode level does not exceed performance capabilities of the playback device.

31. The method of claim 30, wherein the downloading of the another segment of video at the higher decode level takes place after a minimum observation period.

32. The method of claim 30, wherein the another segment of video is downloaded at the higher decoding level only when an average of the accumulated measurements of the frame delivery times is less than the second threshold by a predetermined amount.

33. The method of claim 30 further comprising dynamically deriving one or more of the first and second thresholds.

34. The method of claim 30, wherein one or more of the first and second thresholds is determined statically by a remote server in communication with the playback device.

35. The method of claim 30 further comprising storing one or more of the first and second thresholds locally in a storage of the playback device.

36. The method of claim 30, wherein the initial decode level is equal to a maximum previous decode level.