SIGNALING CHARACTERISTICS OF SEGMENTS FOR NETWORK STREAMING OF MEDIA DATA

• US 20130103849A1
• Filed: 09/20/2012
• Published: 04/25/2013
• Est. Priority Date: 09/21/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Mao, Yinian, Gillies, Donald W.
• Granted Patent: US 9,445,136 B2
• Time in Patent Office: Days
• US Class Current: 709/231
• CPC Class Codes: H04L 65/1083 In-session procedures
• CPC Class Codes: H04L 65/65 Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 21/234327 by decomposing into layers, e.g. base layer and one or more enhancement layers
• CPC Class Codes: H04N 21/23439 for generating different versions
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/44209 Monitoring of downstream path of the transmission network originating from a server, e.g. bandwidth variations of a wireless network arrangements for maintenance or administration in data switching networks involving bandwidth and capacity management H04L41/0896
• CPC Class Codes: H04N 21/4424 Monitoring of the internal components or processes of the client device, e.g. CPU or memory load, processing speed, timer, counter or percentage of the hard disk space used error monitoring in general G06F11/30; arrangements for monitoring conditions of receiving stations in broadcast systems H04H60/32; diagnosis, testing or measuring for television receivers H04N17/04
• CPC Class Codes: H04N 21/6332 directed to client
• CPC Class Codes: H04N 21/64322 IP
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments

Abstract

Video data is retrieved from a server. During retrieval of the video data, a client device receives information indicating bit rates of representations of multimedia content. In addition, the client device receives information indicating priority values for segments of the representations. The segments correspond to particular temporal sections of the representations. The client device requests selected ones of the segments based on the priority values for the segments and an estimated throughput. In some instances, the client device requests the segments in accordance with an adaptive streaming network protocol, such as Dynamic Adaptive Streaming over HTTP (DASH).

Claims

1. A method for processing video data, the method comprising:

determining an estimated throughput;

selecting, based at least in part on the estimated throughput and priority values for segments of representations of multimedia content, one or more of the segments, wherein the representations include video data are encoded at different bit rates, each of the segments are retrievable individually, and the segments correspond to particular temporal sections of the representations; and

processing the one or more selected segments.

2. The method of claim 1, wherein the estimated throughput is a bandwidth of a communication network.

3. The method of claim 2, further comprising requesting the one or more selected segments in accordance with an adaptive streaming network protocol.

4. The method of claim 1, wherein the estimated throughput is an estimate of an amount of media data that one or more processors are able to decode during an upcoming time period.

5. The method of claim 1, wherein selecting one or more of the segments based on the priority values for the segments and the estimated throughput comprises selecting one or more of the segments based on the priority values for the segments, the estimated throughput, and additionally a current playback duration of video data in a playback buffer.

6. The method of claim 5, wherein the estimated throughput is an estimated bandwidth of a communication network and an estimated transfer time for the one or more selected segments is less than or equal to a playback duration of the one or more selected segments plus the current playback duration of the video data in the playback buffer, the estimated transfer time for the one or more selected segments being based at least in part on the estimated bandwidth and a total number of bits in the one or more selected segments.

7. The method of claim 1, further comprising:

rendering sequences of frames represented by the one or more selected segments; and

presenting the sequences of frames on a display device.

8. The method of claim 1,

wherein the method further comprises storing a manifest file, the manifest file specifying URLs for the segments and the information that indicates the priority values for the segments;

and

using the URLs for the one or more selected segments to request the one or more selected segments.

9. The method of claim 1,

wherein ones of the segments that correspond to different temporal sections of the media content are in different segment groups;

wherein the method further comprises selecting a segment combination for an upcoming time period, the segment combination including one segment for each of the segment groups that is within the upcoming time period; and

wherein processing the one or more selected segments comprises decoding the segments in the segment combination.

10. The method of claim 9, wherein selecting the segment combination comprises:

identifying a plurality of candidate combinations, wherein for each of the candidate combinations, an average of bit rates of the segments in the candidate combination is not significantly greater than the estimated throughput;

calculating QoS scores for the candidate combinations based on the priority values associated with the segments in the candidate combinations; and

identifying a winning combination from among the plurality of candidate combinations based on the QoS scores for the candidate combinations.

11. The method of claim 10,

wherein the candidate combinations include a given candidate combination;

and

wherein calculating the QoS scores for the candidate combinations comprises calculating the QoS score for the given candidate combination by summing the priority values associated with the segments in the given candidate combination.

12. The method of claim 10,

wherein the candidate combinations include a given candidate combination;

and

wherein calculating the QoS scores for the candidate combinations comprises calculating the QoS score for the given candidate combination based on peak signal-to-noise ratio values of the segments in the given candidate combination.

13. The method of claim 10,

wherein the candidate combinations include a given candidate combination;

and

wherein calculating the QoS scores for the candidate combinations comprises calculating the QoS score for the given candidate combination by calculating a geometric mean of the priority values associated with the segments in the given candidate combination.

14. The method of claim 1, wherein the priority values of the segments are correlated with amounts of computed distortion in the segments.

15. The method of claim 1, wherein the priority values of the segments are correlated with amounts of perceptual distortion perceivable in the segments by a human nervous system.

16. A computing device comprising one or more processors configured to:

determine an estimated throughput;

select, based at least in part on the estimated throughput and priority values for segments of representations of multimedia content, one or more of the segments, wherein the representations include video data are encoded at different bit rates, each of the segments are retrievable individually, and the segments correspond to particular temporal sections of the representations; and

process the one or more selected segments.

17. The computing device of claim 16, wherein the estimated throughput is a bandwidth of a communication network.

18. The computing device of claim 17, wherein the one or more processors are further configured to request the one or more selected segments in accordance with an adaptive streaming network protocol.

19. The computing device of claim 16, wherein the estimated throughput is an estimate of an amount of media data that one or more processors are able to decode during an upcoming time period.

20. The computing device of claim 16, wherein the one or more processors are configured to select one or more of the segments based on the priority values for the segments, the estimated throughput, and additionally a current playback duration of video data in a playback buffer.

21. The computing device of claim 16, wherein the estimated throughput is an estimated bandwidth of a communication network and an estimated transfer time for the one or more selected segments is less than or equal to a playback duration of the one or more selected segments plus the current playback duration of the video data in the playback buffer, the estimated transfer time for the one or more selected segments being based on the estimated bandwidth and a total number of bits in the one or more selected segments.

22. The computing device of claim 16, wherein the one or more processors are further configured to:

render sequences of frames represented by the one or more selected segments; and

present the sequences of frames on a display device.

23. The computing device of claim 16,

wherein the computing device stores a manifest file, the manifest file specifying URLs for the segments and the information that indicates the priority values for the segments;

and

where one or more processors are configured to use the URLs for the one or more selected segments to generate requests for the one or more selected segments.

24. The computing device of claim 16,

wherein ones the segments that correspond to different temporal sections of the media content are in different segment groups;

and

wherein the one or more processors are further configured to;

select a segment combination for an upcoming time period, the segment combination including one segment for each of the segment groups that is within the upcoming time period; and

decode the segments in the segment combination.

25. The computing device of claim 24, wherein one or more processors are configured to:

identify a plurality of candidate combinations, wherein for each of the candidate combinations, an average of bit rates of the segments in the candidate combination is not significantly greater than the estimated throughput;

calculate QoS scores for the candidate combinations based on the priority values associated with the segments in the candidate combinations; and

identify a winning combination from among the plurality of candidate combinations based on the QoS scores for the candidate combinations.

26. The computing device of claim 25,

wherein the candidate combinations include a given candidate combination;

and

wherein the one or more processors are configured to calculate the QoS score for the given candidate combination by summing the priority values associated with the segments in the given candidate combination.

27. The computing device of claim 25,

wherein the candidate combinations include a given candidate combination;

and

wherein the one or more processors are configured to calculate the QoS score for the given candidate combination based on peak signal-to-noise ratio values of the segments in the given candidate combination.

28. The method of claim 25,

wherein the candidate combinations include a given candidate combination;

and

wherein the one or more processors are configured to calculate the QoS score for the given candidate combination by calculating a geometric mean of the priority values associated with the segments in the given candidate combination.

29. The computing device of claim 16, wherein the priority values of the segments are correlated with amounts of computed distortion in the segments.

30. The computing device of claim 16, wherein the priority values of the segments are correlated with amounts of perceptual distortion perceivable in the segments by a human nervous system.

31. A computing device that comprises:

means for determining an estimated throughput;

means for selecting, based at least in part on the estimated throughput and priority values for segments of representations of multimedia content, one or more of the segments, wherein the representations include video data are encoded at different bit rates, each of the segments are retrievable individually, and the segments correspond to particular temporal sections of the representations; and

means for processing the one or more selected segments.

32. The computing device of claim 31, wherein the estimated throughput is a bandwidth of a communication network.

33. The computing device of claim 31, further comprising means for requesting the one or more selected segments in accordance with an adaptive streaming network protocol.

34. The computing device of claim 31, wherein the estimated throughput is an estimate of an amount of media data that one or more processors are able to decode during an upcoming time period.

35. The computing device of claim 31,

wherein ones the segments that corresponds to different temporal sections of the media content are in different segment groups;

wherein the computing device comprises;

means for selecting a segment combination for an upcoming time period, the segment combination including one segment for each of the segment groups that is within the upcoming time period; and

means for requesting the segments in the segment combination.

36. The computing device of claim 35, comprising:

means for identifying a plurality of candidate combinations, wherein for each of the candidate combinations, an average of bit rates of the segments in the candidate combination is not significantly greater than the estimated throughput;

means for calculating QoS scores for the candidate combinations based on the priority values associated with the segments in the candidate combinations; and

means for identifying a winning combination from among the plurality of candidate combinations based on the QoS scores for the candidate combinations.

37. The computing device of claim 31, wherein the priority values of the segments are correlated with amounts of computed distortion in the segments.

38. The computing device of claim 31, wherein the priority values of the segments are correlated with amounts of perceptual distortion perceivable in the segments by a human nervous system.

39. A computer-readable storage medium having stored thereon instructions that, when executed, cause a processor to:

determine an estimated throughput;

select, based at least in part on the estimated throughput and priority values for segments of representations of multimedia content, one or more of the segments, wherein the representations include video data are encoded at different bit rates, each of the segments are retrievable individually, and the segments correspond to particular temporal sections of the representations; and

process the one or more selected segments.

40. The computer-readable storage medium of claim 39, wherein the estimated throughput is a bandwidth of a communication network.

41. The computer-readable storage medium of claim 40, further comprising requesting the one or more selected segments in accordance with an adaptive streaming network protocol.

42. The computer-readable storage medium of claim 39, wherein the estimated throughput is an estimate of an amount of media data that one or more processors are able to decode during an upcoming time period.

43. The computer-readable storage medium of claim 39,

wherein ones the segments that correspond to different temporal sections of the media content are in different segment groups;

and

wherein execution of the instructions further causes the computing device to;

select a segment combination for an upcoming time period, the segment combination including one segment for each of the segment groups that is within the upcoming time period; and

request the segments in the segment combination.

44. The computer-readable storage medium of claim 39, wherein execution of the instructions causes the computing device to:

identify a plurality of candidate combinations, wherein for each of the candidate combinations, an average of bit rates of the segments in the candidate combination is not significantly greater than the estimated throughput;

calculate QoS scores for the candidate combinations based on the priority values associated with the segments in the candidate combinations; and

identify a winning combination from among the plurality of candidate combinations based on the QoS scores for the candidate combinations.

45. The computer-readable storage medium of claim 39, wherein the priority values of the segments are correlated with amounts of computed distortion in the segments.

46. The computer-readable storage medium of claim 39, wherein the priority values of the segments are correlated with amounts of perceptual distortion perceivable in the segments by a human nervous system.

47. A method for providing video data via an adaptive streaming network protocol, the method comprising:

sending information that indicates priority values for segments, each of the segments being retrievable individually, the segments corresponding to particular temporal sections of representations of multimedia content, each of the representations of the multimedia content including video data encoded at a different bit rate; and

sending requested segments via the adaptive streaming network protocol in response to one or more requests from a client device, the requested segments being ones of the segments specified by the requests.

48. The method of claim 47, further comprising:

assigning the priority values to the segments; and

storing the priority values to a manifest file for the multimedia content.

49. The method of claim 48, wherein assigning the priority values to the segments comprises calculating the priority values based on characteristics of the segments.

50. The method of claim 49, wherein the characteristics of the segments include amounts of computed distortion in the segments.

51. The method of claim 49, wherein assigning the priority values to the segments comprises assigning the priority value to the segments such that the priority values of the segments are correlated with amounts of perceptual distortion perceivable in the segments by a human nervous system.

52. A computing device comprising one or more processors configured to:

send information that indicates priority values for segments, each of the segments being retrievable individually, the segments corresponding to particular temporal sections of representations of multimedia content, each of the representations of the multimedia content including video data encoded at a different bit rate; and

send requested segments via the adaptive streaming network protocol in response to one or more requests from a client device, the requested segments being ones of the segments specified by the requests.

53. The computing device of claim 52, wherein the one or more processors are configured to:

assign the priority values to the segments; and

store the priority values to a manifest file for the multimedia content.

54. The computing device of claim 53, wherein the one or more processors are configured to calculate the priority values based on characteristics of the segments.

55. The computing device of claim 54, wherein the characteristics of the segments include amounts of computed distortion in the segments.

56. The computing device of claim 52, wherein the one or more processors are configured to assign the priority values to the segments such that the priority values of the segments are correlated with amounts of perceptual distortion perceivable in the segments by a human nervous system.

57. A computing device comprising:

means for sending information that indicates priority values for segments, each of the segments being retrievable individually, the segments corresponding to particular temporal sections of representations of multimedia content, each of the representations of the multimedia content including video data encoded at a different bit rate; and

means for sending requested segments via the adaptive streaming network protocol in response to one or more requests from a client device, the requested segments being ones of the segments specified by the requests.

58. The computing device of claim 57, further comprising:

means for assigning the priority values to the segments; and

means for storing the priority values to a manifest file for the multimedia content.

59. The computing device of claim 58, further comprising means for calculating the priority values based on characteristics of the segments.

60. The computing device of claim 59, wherein the characteristics of the segments include amounts of computed distortion in the segments.

61. The computing device of claim 57, wherein the means for assigning the priority values to the segments comprises means for assigning the priority values to the segments such that the priority values of the segments are correlated with amounts of perceptual distortion perceivable in the segments by a human nervous system.

62. A computer-readable storage medium having stored thereon instructions that, when executed by one or more processors of a computing device, cause the computing device to:

send information that indicates priority values for segments, each of the segments being retrievable individually, the segments corresponding to particular temporal sections of representations of multimedia content, each of the representations of the multimedia content including video data encoded at a different bit rate; and

send requested segments via the adaptive streaming network protocol in response to one or more requests from a client device, the requested segments being ones of the segments specified by the requests.

63. The computer-readable storage medium of claim 62, wherein execution of the instructions by the one or more processors causes the computing device to:

assign the priority values to the segments; and

store the priority values to a manifest file for the multimedia content.

64. The computer-readable storage medium of claim 63, wherein execution of the instructions by the one or more processors causes the computing device to calculate the priority values based on characteristics of the segments.

65. The computer-readable storage medium of claim 64, wherein the characteristics of the segments include amounts of computed distortion in the segments.

66. The computer-readable storage medium of claim 62, wherein execution of the instructions by the one or more processors causes the computing device to assign the priority values to the segments such that the priority values of the segments are correlated with amounts of perceptual distortion perceivable in the segments by a human nervous system.