Signaling characteristics of segments for network streaming of media data

• US 9,445,136 B2
• Filed: 09/20/2012
• Issued: 09/13/2016
• Est. Priority Date: 09/21/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Gillies, Donald W., Mao, Yinian
• Primary Examiner(s): Hussain, Tauqir
• Assistant Examiner(s): Mohammadi, Kamran
• Application Number:
  US13/623,589
• Publication Number:
  US 20130103849A1
• Time in Patent Office: 1,454 Days
• Field of Search: 701/1
• US Class Current: 1/1
• CPC Class Codes: H04L 65/1083 In-session procedures
• CPC Class Codes: H04L 65/65 Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/23439 for generating different versions
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/44209 Monitoring of downstream path of the transmission network originating from a server, e.g. bandwidth variations of a wireless network arrangements for maintenance or administration in data switching networks involving bandwidth and capacity management H04L41/0896
• CPC Class Codes: H04N 21/4424 Monitoring of the internal components or processes of the client device, e.g. CPU or memory load, processing speed, timer, counter or percentage of the hard disk space used error monitoring in general G06F11/30; arrangements for monitoring conditions of receiving stations in broadcast systems H04H60/32; diagnosis, testing or measuring for television receivers H04N17/04
• CPC Class Codes: H04N 21/6332 directed to client
• CPC Class Codes: H04N 21/64322 IP
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments

Abstract

Video data is retrieved from a server. During retrieval of the video data, a client device receives information indicating bit rates of representations of multimedia content. In addition, the client device receives information indicating priority values for segments of the representations. The segments correspond to particular temporal sections of the representations. The client device requests selected ones of the segments based on the priority values for the segments and an estimated throughput. In some instances, the client device requests the segments in accordance with an adaptive streaming network protocol, such as Dynamic Adaptive Streaming over HTTP (DASH).

Claims

1. A method for processing video data, the method comprising:

determining an estimated throughput;

receiving, from a server device, information indicative of a respective priority value for each respective segment group of a plurality of segment groups, wherein;

for each respective segment group of the plurality of segment groups;

the respective segment group is a respective plurality of segments,

each of the segments of the respective segment group corresponds to a respective temporal section of media content different from temporal sections of the media content to which segments of other segment groups of the plurality of segment groups correspond,

each respective segment of the respective segment group represents the same respective temporal section of the media content, is retrievable individually, and is from a different representation of a plurality of representations of the media content, each of the representations being encoded at a different bit rate, and

the respective priority value for the respective segment group indicates an importance of selecting a segment for the respective temporal section from a representation in the plurality of representations that has a higher bit rate than other ones of the representations, and

the priority value for a first segment group of the plurality of segment groups is different from the priority value for a second, different segment group of the plurality of segment groups, the first segment group and the second segment group including respective segments from the same representation of the plurality of representations;

performing a selection operation that selects a segment of each respective segment group of the plurality of segment groups, wherein performing the selection operation comprises selecting the segments of each respective segment group based at least in part on the estimated throughput and the priority values for the segment groups; and

processing the selected segments.

2. The method of claim 1, wherein the estimated throughput is a bandwidth of a communication network.

3. The method of claim 2, further comprising requesting the selected segments in accordance with an adaptive streaming network protocol.

4. The method of claim 1, wherein the estimated throughput is an estimate of an amount of media data that one or more processors are able to decode during an upcoming time period.

5. The method of claim 1, wherein selecting the segments comprises selecting the segments of each respective segment group based on the priority values for the segment groups, the estimated throughput, and additionally a current playback duration of video data in a playback buffer.

6. The method of claim 1, wherein:

the estimated throughput is an estimated bandwidth of a communication network, and

an estimated transfer time for the selected segments is less than or equal to a playback duration of the selected segments plus a current playback duration of video data in a playback buffer, the estimated transfer time for the selected segments being based at least in part on the estimated bandwidth and a total number of bits in the selected segments.

7. The method of claim 1, further comprising:

rendering sequences of frames represented by the selected segments; and

presenting the sequences of frames on a display device.

8. The method of claim 1, further comprising:

storing a manifest file, wherein, for each respective segment group of the plurality of segment groups, the manifest file specifies URLs for the segments of the respective segment group and the information indicative of the priority value for the respective segment group; and

using the URLs for the selected segments to generate requests for the selected segments.

9. The method of claim 1, wherein processing the selected segments comprises decoding the selected segments.

10. The method of claim 9, wherein selecting the segments of each respective segment group comprises:

identifying a plurality of candidate combinations, wherein for each respective candidate combination of the plurality of candidate combinations;

the respective candidate combination includes one segment of each of the segment groups that is within an upcoming time period, and

an average of bit rates of the segments in the respective candidate combination is not significantly greater than the estimated throughput;

calculating, based on the priority values for the segment groups within the upcoming time period, a respective score for each respective candidate combination of the plurality of candidate combinations; and

identifying, based on the scores for the candidate combinations, a winning combination from the plurality of candidate combinations, the selected segments being in the winning combination, wherein the winning combination has a score that is greater than or equal to the scores for each other one of the candidate combinations.

11. The method of claim 10, wherein:

performing the selection operation comprises using the priority values for the segment groups within the upcoming time period to calculate respective priority values for each respective segment in the segment groups within the upcoming time period; and

calculating the respective score for each respective candidate combination of the plurality of candidate combinations comprises, for each respective candidate combination of the plurality of candidate combinations, calculating the respective score for the respective candidate combination by summing the priority values for the segments in the respective candidate combination.

12. The method of claim 10,

wherein calculating the respective score for each respective candidate combination of the plurality of candidate combinations comprises, for each respective candidate combination of the plurality of candidate combinations, calculating the respective score for the respective candidate combination based on peak signal-to-noise ratio values of the segments in the respective candidate combination.

13. The method of claim 10, wherein:

performing the selection operation comprises using the priority values for the segment groups within the upcoming time period to calculate respective priority values for each respective segment in the segment groups within the upcoming time period; and

calculating the respective score for each respective candidate combination of the plurality of candidate combinations comprises, for each respective candidate combination of the plurality of candidate combinations, calculating the respective score for the respective candidate combination by calculating a geometric mean of the priority values for the segments in the respective candidate combination.

14. A computing device comprising:

one or more non-transitory computer storage media configured to store video data; and

one or more processors configured to;

determine an estimated throughput;

receive, from a server device, information indicative of respective priority values for each respective segment group of a plurality of segment groups, wherein;

for each respective segment group of the plurality of segment groups;

the respective segment group is a respective plurality of segments,

each of the segments of the respective segment group corresponds to a respective temporal section of media content different from temporal sections of the media content to which segments of other segment groups of the plurality of segment groups correspond,

each respective segment of the respective segment group represents the same respective temporal section of the media content, is retrievable individually, and is from a different representation of a plurality of representations of the media content, each of the representations being encoded at a different bit rate, and

the respective priority value for the respective segment group indicates an importance of selecting a segment for the respective temporal section from a representation in the plurality of representations that has a higher bit rate than other ones of the representations, and

the priority value for a first segment group of the plurality of segment groups is different from the priority value for a second, different segment group of the plurality of segment groups, the first segment group and the second segment group including respective segments from the same representation of the plurality of representations;

perform a selection operation that selects a segment of each respective segment group of the plurality of segment groups, wherein the one or more processors are configured such that, as part of performing the selection operation, the one or more processors select the segments of each respective segment group based at least in part on the estimated throughput and the priority values for the segment groups; and

process the selected segments.

15. The computing device of claim 14, wherein the estimated throughput is a bandwidth of a communication network.

16. The computing device of claim 15, wherein the one or more processors are configured to request the selected segments in accordance with an adaptive streaming network protocol.

17. The computing device of claim 14, wherein the estimated throughput is an estimate of an amount of media data that the one or more processors are able to decode during an upcoming time period.

18. The computing device of claim 14, wherein the one or more processors are configured to select the segments of each respective segment group based on the priority values for the segment groups, the estimated throughput, and additionally a current playback duration of video data in a playback buffer.

19. The computing device of claim 14, wherein:

the estimated throughput is an estimated bandwidth of a communication network, and

an estimated transfer time for the selected segments is less than or equal to a playback duration of the selected segments plus a current playback duration of video data in a playback buffer, the estimated transfer time for the selected segments being based on the estimated bandwidth and a total number of bits in the selected segments.

20. The computing device of claim 14, wherein to process the selected segments, the one or more processors:

render sequences of frames represented by the selected segments; and

present the sequences of frames on a display device.

21. The computing device of claim 14, wherein:

the computing device stores a manifest file,

for each respective segment group of the plurality of segment groups, the manifest file specifies URLs for the segments of the respective segment group and the information indicative of the priority value for the respective segment group, and

the one or more processors are configured to use the URLs for the selected segments to generate requests for the selected segments.

22. The computing device of claim 14, wherein the one or more processors are configured to:

decode the selected segments.

23. The computing device of claim 14, wherein the one or more processors are configured to:

identify a plurality of candidate combinations, wherein for each respective candidate combination of the plurality of candidate combinations;

the respective candidate combination includes one segment of each of the segment groups that is within an upcoming time period, and

an average of bit rates of the segments in the respective candidate combination is not significantly greater than the estimated throughput;

calculate, based on the priority values for the segment groups within the upcoming time period, a respective score for each respective candidate combination of the plurality of candidate combinations; and

identify, based on the scores for the candidate combinations, a winning combination from the plurality of candidate combinations, the selected segments being in the winning combination, wherein the winning combination has a score that is greater than or equal to the scores for each other one of the candidate combinations.

24. The computing device of claim 23, wherein the one or more processors are configured to:

use the priority values for the segment groups within the upcoming time period to calculate respective priority values for each respective segment in the segment groups within the upcoming time period; and

for each respective candidate combination of the plurality of candidate combinations, calculate the respective score for the respective candidate combination by summing the priority values for the segments in the respective candidate combination.

25. The computing device of claim 23, wherein the one or more processors are configured to:

for each respective candidate combination of the plurality of candidate combinations, calculate the respective score for the respective candidate combination based on peak signal-to-noise ratio values of the segments in the respective candidate combination.

26. The method of claim 23, wherein the one or more processors are configured to:

use the priority values for the segment groups within the upcoming time period to calculate respective priority values for each respective segment in the segment groups within the upcoming time period; and

for each respective candidate combination of the plurality of candidate combinations, calculate the respective score for the respective candidate combination by calculating a geometric mean of the priority values for the segments in the respective candidate combination.

27. A computing device comprising:

means for determining an estimated throughput;

means for receiving, from a server device, information indicative of a respective priority value for each respective segment group of a plurality of segment groups, wherein;

for each respective segment group of the plurality of segment groups;

the respective segment group is a respective plurality of segments,

each of the segments of the respective segment group corresponds to a respective temporal section of media content different from temporal sections of the media content to which segments of other segment groups of the plurality of segment groups correspond,

each respective segment of the respective segment group represents the same respective temporal section of the media content, is retrievable individually, and is from a different representation of a plurality of representations of the media content, each of the representations being encoded at a different bit rate, and

the respective priority value for the respective segment group indicates an importance of selecting a segment for the respective temporal section from a representation in the plurality of representations that has a higher bit rate than other ones of the representations, and

the priority value for a first segment group of the plurality of segment groups is different from the priority value for a second, different segment group of the plurality of segment groups, the first segment group and the second segment group including respective segments from the same representation of the plurality of representations;

means for performing a selection operation that selects a segment of each respective segment group of the plurality of segment groups, wherein the means for performing the selection operation comprises means for selecting the segments of each respective segment group based at least in part on the estimated throughput and the priority values for the segment groups; and

means for processing the selected segments.

28. The computing device of claim 27, wherein the estimated throughput is a bandwidth of a communication network.

29. The computing device of claim 27, further comprising means for requesting the selected segments in accordance with an adaptive streaming network protocol.

30. The computing device of claim 27, wherein the estimated throughput is an estimate of an amount of media data that one or more processors are able to decode during an upcoming time period.

31. The computing device of claim 27, comprising:

means for decoding the selected segments.

32. The computing device of claim 27, comprising:

means for identifying a plurality of candidate combinations, wherein for each respective candidate combination of the plurality of candidate combinations;

the respective candidate combination includes one segment of each of the segment groups that is within an upcoming time period, and

an average of bit rates of the segments in the respective candidate combination is not significantly greater than the estimated throughput;

means for calculating, based on the priority values for the segment groups within the upcoming time period, a respective score for each respective candidate combination of the plurality of candidate combinations; and

means for identifying, based on the scores for the candidate combinations, a winning combination from the plurality of candidate combinations, the selected segments being in the winning combination, wherein the winning combination has a score that is greater than or equal to the scores for each other one of the candidate combinations.

33. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to:

determine an estimated throughput;

receive, from a server device, information indicative of a respective priority value for each respective segment group of a plurality of segment groups, wherein;

for each respective segment group of the plurality of segment groups;

the respective segment group is a respective plurality of segments,

each of the segments of the respective segment group corresponds to a respective temporal section of media content different from temporal sections of the media content to which segments of other segment groups of the plurality of segment groups correspond,

each respective segment of the respective segment group represents the same respective temporal section of the media content, is retrievable individually, and is from a different representation of a plurality of representations of the media content, each of the representations being encoded at a different bit rate, and

the respective priority value for the respective segment group indicates an importance of selecting a segment for the respective temporal section from a representation in the plurality of representations that has a higher bit rate than other ones of the representations, and

the priority value for a first segment group of the plurality of segment groups is different from the priority value for a second, different segment group of the plurality of segment groups, the first segment group and the second segment group including respective segments from the same representation of the plurality of representations;

perform a selection operation that selects a segment of each respective segment group of the plurality of segment groups, wherein the instructions cause the one or more processors to perform the selection operation in part by causing the one or more processors to select the segments of each respective segment group based at least in part on the estimated throughput and the priority values for the segment groups; and

process the selected segments.

34. The non-transitory computer-readable storage medium of claim 33, wherein the estimated throughput is a bandwidth of a communication network.

35. The non-transitory computer-readable storage medium of claim 34, wherein the instructions, when executed, cause the one or more processors to request the selected segments in accordance with an adaptive streaming network protocol.

36. The non-transitory computer-readable storage medium of claim 33, wherein the estimated throughput is an estimate of an amount of media data that the one or more processors are able to decode during an upcoming time period.

37. The non-transitory computer-readable storage medium of claim 33, wherein the instructions, when executed, cause the one or more processors to decode the selected segments.

38. The non-transitory computer-readable storage medium of claim 33, wherein the instructions, when executed, cause the one or more processors to:

identify a plurality of candidate combinations, wherein for each respective candidate combination of the plurality of candidate combinations;

the respective candidate combination includes one segment of each of the segment groups that is within an upcoming time period, and

an average of bit rates of the segments in the respective candidate combination is not significantly greater than the estimated throughput;

calculate, based on the priority values for the segment groups within the upcoming time period, a respective score for each respective candidate combination of the plurality of candidate combinations; and

identify, based on the scores for the candidate combinations, a winning combination from the plurality of candidate combinations, the selected segments being in the winning combination, wherein the winning combination has a score that is greater than or equal to the scores for each other one of the candidate combinations.

39. A method for providing video data via an adaptive streaming network protocol, the method comprising:

sending information indicative of a respective priority value for each respective segment group of a plurality of segment groups, wherein;

for each respective segment group of the plurality of segment groups;

the respective segment group is a respective plurality of segments,

each of the segments of the respective segment group corresponds to a respective temporal section of media content different from temporal sections of the media content to which segments of other segment groups of the plurality of segment groups correspond,

each respective segment of the respective segment group represents the same respective temporal section of the media content, is retrievable individually, and is from a different representation of a plurality of representations of the media content, each of the representations being encoded at a different bit rate, and

the respective priority value for the respective segment group indicates an importance of selecting a segment for the respective temporal section from a representation in the plurality of representations that has a higher bit rate than other ones of the representations, and

the priority value for a first segment group of the plurality of segment groups is different from the priority value for a second, different segment group of the plurality of segment groups, the first segment group and the second segment group including respective segments from the same representation of the plurality of representations; and

sending requested segments via the adaptive streaming network protocol in response to one or more requests from a client device, the requested segments being ones of the segments specified by the one or more requests.

40. The method of claim 39, further comprising:

assigning the priority values to the segment groups; and

storing the priority values to a manifest file for the media content.

41. A computing device comprising:

one or more non-transitory computer storage media configured to store media data; and

one or more processors configured to;

send information indicative of a respective priority value for each respective segment group of a plurality of segment groups, wherein;

for each respective segment group of the plurality of segment groups;

the respective segment group is a respective plurality of segments,

each of the segments of the respective segment group corresponds to a respective temporal section of media content different from temporal sections of the media content to which segments of other segment groups of the plurality of segment groups correspond,

each respective segment of the respective segment group represents the same respective temporal section of the media content, is retrievable individually, and is from a different representation of a plurality of representations of the media content, each of the representations being encoded at a different bit rate, and

the respective priority value for the respective segment group indicates an importance of selecting a segment for the respective temporal section from a representation in the plurality of representations that has a higher bit rate than other ones of the representations, and

the priority value for a first segment group of the plurality of segment groups is different from the priority value for a second, different segment group of the plurality of segment groups, the first segment group and the second segment group including respective segments from the same representation of the plurality of representations; and

send requested segments via an adaptive streaming network protocol in response to one or more requests from a client device, the requested segments being ones of the segments specified by the requests.

42. The computing device of claim 41, wherein the one or more processors are configured to:

assign the priority values to the segment groups; and

store the priority values to a manifest file for the media content.

43. A computing device comprising:

means for sending information indicative of a respective priority value for each respective segment group of a plurality of segment groups, wherein;

for each respective segment group of the plurality of segment groups;

the respective segment group is a respective plurality of segments,

each of the segments of the respective segment group corresponds to a respective temporal section of media content different from temporal sections of the media content to which segments of other segment groups of the plurality of segment groups correspond,

each respective segment of the respective segment group represents the same respective temporal section of the media content, is retrievable individually, and is from a different representation of a plurality of representations of the media content, each of the representations being encoded at a different bit rate, and

the respective priority value for the respective segment group indicates an importance of selecting a segment for the respective temporal section from a representation in the plurality of representations that has a higher bit rate than other ones of the representations, and

the priority value for a first segment group of the plurality of segment groups is different from the priority value for a second, different segment group of the plurality of segment groups, the first segment group and the second segment group including respective segments from the same representation of the plurality of representations; and

means for sending requested segments via an adaptive streaming network protocol in response to one or more requests from a client device, the requested segments being ones of the segments specified by the requests.

44. The computing device of claim 43, further comprising:

means for assigning the priority values to the segment groups; and

means for storing the priority values to a manifest file for the media content.

45. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by one or more processors of a computing device, cause the computing device to:

send information indicative of a respective priority value for each respective segment group of a plurality of segment groups, wherein;

for each respective segment group of the plurality of segment groups;

the respective segment group is a respective plurality of segments,

each of the segments of the respective segment group corresponds to a respective temporal section of media content different from temporal sections of the media content to which segments of other segment groups of the plurality of segment groups correspond,

each respective segment of the respective segment group represents the same respective temporal section of the media content, is retrievable individually, and is from a different representation of a plurality of representations of the media content, each of the representations being encoded at a different bit rate, and

the respective priority value for the respective segment group indicates an importance of selecting a segment for the respective temporal section from a representation in the plurality of representations that has a higher bit rate than other ones of the representations, and

the priority value for a first segment group of the plurality of segment groups is different from the priority value for a second, different segment group of the plurality of segment groups, the first segment group and the second segment group including respective segments from the same representation of the plurality of representations; and

send requested segments via an adaptive streaming network protocol in response to one or more requests from a client device, the requested segments being ones of the segments specified by the requests.

46. The non-transitory computer-readable storage medium of claim 45, wherein the instructions, when executed by the one or more processors, cause the computing device to:

assign the priority values to the segment groups; and

store the priority values to a manifest file for the media content.