METHOD AND APPARATUS FOR SUBSCRIPTION-BASED BANDWIDTH BALANCING FOR INTERACTIVE HETEROGENEOUS CLIENTS

• US 20110022705A1
• Filed: 07/20/2010
• Published: 01/27/2011
• Est. Priority Date: 07/21/2009
• Status: Active Grant

• Current Assignee: Polycom Incorporated (Plantronics, Inc.)
• Original Assignee: ViVu, Inc. (Plantronics, Inc.)
• Inventors: Katasani, Simha Sundeep Reddy, Yellamraju, SivaKiran Venkata
• Granted Patent: US 8,626,499 B2
• Time in Patent Office: Days
• US Class Current: 709/224
• CPC Class Codes: H04L 65/1069 Session establishment or de-establishment
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04L 65/80 Responding to QoS

Abstract

A method or system for enabling client devices connected to a network to subscribe to a one or more multimedia signals available in a selection of bandwidths, based on an analysis of capabilities of the client, the quality of the connection, and the real-time changes in the ability of the client to process the signal.

Claims

1. A system configured to enable a client device to publish and subscribe one or more media streams over a network, where each of the of media streams are available in one or more bandwidths and media types, the system comprising:

a server, coupled to a network, the server comprising a computer, a storage, and a memory;

a first module, coupled to the server, configured to accept information about media streams and capabilities of client devices, associate media stream and client devices with a session, calculate the capabilities of a client device based on the network it is connected to, and report the available media streams associated with a session on request, wherein the information about media streams comprises the identity of the stream, type of stream, bandwidth of stream, and identity of the publisher of the stream;

a smart client device, coupled to a network, the smart client device comprising a computer, a storage and a memory; and

a second module, coupled to the smart client device, configured to;

join a session,

accept information about media streams associated with that session including identifier, media type and bandwidth,

play one or more of the media streams associated with that session based on calculated priority and available bandwidth,

monitor the quality of the one or more media streams played,

respond to changes in the ability of the client device to play a media stream by skipping portions of the stream; and

responding to a high frequency of skipping portions of a media stream by either playing a different stream of lower quality, or stopping playing the stream altogether if such a stream is unavailable,

wherein the quality of the media stream is a combination of the latency of the played media signal relative to its input media stream, bit error rate, and signal strength.

2. A method for enabling client devices coupled to a network to subscribe to one or more media streams, where each of the of media streams are available in one or more bandwidths and media types, the method comprising:

enabling a client device to join a session;

determining and reporting media stream information about streams which the client device can publish to the network;

determining a set of capabilities associated with the client device which enables it to accept media streams of various bandwidths and media types;

notifying the client device as to what media streams are available and in what bandwidths and media types;

accepting requests from client devices to subscribe to one or more of the plurality of multimedia signals, each at a specified bandwidth and media type;

measuring the signal quality as a function of the varying bandwidth of the subscribed plurality of multimedia signals and the ability of the media stream player to play the signal without falling behind the live stream; and

responding to inability for the client device to keep up with the live stream by skipping a part of the signal; and

responding to frequent skipping of the signal by either switching to a lower quality version of the multimedia signal or stop playing the multimedia stream where a lower quality signal is unavailable, wherein the quality of the signal is a combination of the bandwidth and media type,

wherein the signal portions are skipped by moving ahead in a signal buffer at short intervals, the size of the skipped signal portions and intervals determined by the quality of the signal based on measurements of trend of signal latency.

3. The method in claim 2, wherein the enabling of the client device to join the session further comprises detecting whether the client device is an interactive device based on its mode of connection, and if that mode of connection is determined to not allow interactivity, selecting the stream for the client device to subscribe to.

4. The method in claim 3, further comprising:

accepting requests from a client device associated with a session to publish or unpublish one or more of a plurality of multimedia signals associated with the client device associated with the session; and

notifying all interactive client devices other than the client device publishing the stream of the availability of multimedia signals associated with the client device.

5. A method for managing the impact of signal latency on the reception of a multimedia signal by a client device coupled to a network, the method comprising;

defining a first period of time as the size of the signal to skip;

defining a second period of time as the minimum time between signal skips;

defining a third period of time as the length of the signal to buffer;

calculating a first moving average of the signal latency over a first period of time;

calculating a minimum of a first moving average of the latency over a session;

measuring a change in signal latency over some configured period of time;

buffering a part of the signal equal to the third period of time prior to presenting it to the client device;

skipping the first period of time of the signal if the change in signal latency exceeds some configured value;

skipping the first period of time of the signal if the signal latency exceeds some configured percentage of the measurement of the minimum of the first moving average; and

skipping the first period of time of the signal if the signal latency exceeds some configured percentage of the first moving average of the signal latency some configured period of time after the last time a part of the signal was skipped.

6. The method in claim 5, where method further comprises:

calculating a second moving average over a second period of time, wherein the second period of time is longer than the first period of time;

shorter than the expected length of a session; and

skipping the first period of time of the signal if the first moving average exceeds the second moving average at least the second period of time after the last time a part of the signal was skipped.

7. The method in claim 5, where the determining of the measurement of the minimum signal latency further comprises initializing the minimum signal latency as a current latency measured some configured time into the current session, thereafter for the duration of the session comparing a current value of the minimum signal latency to a current latency value, and setting the minimum signal latency value to the current latency value if the current latency value is less that the minimum signal latency value.

8. The method in claim 5, where the third period of time is larger than the expected signal latency but smaller than the expected length of a session.

9. The method in claim 5, where the third period of time is no less than 15 seconds and no more than 1 minute.

10. The method in claim 5, where the first period of time is smaller than the buffer length.

11. The method in claim 10, where the first period of time is less than 0.5 seconds.

12. The method in claim 5, where the change in signal latency is calculated by comparing the latency measured at the first configured period of time prior to the most recent measurement.

13. The method in claim 5, where the first configured period of time is shorter than the size of the buffer but longer than the second configured period of time.

14. The method in claim 13, where the first configured period of time is at least 3 seconds but not more than 10 seconds, and the second period of time is at least 30 seconds but not more than 2 minutes.

15. The method in claim 5, where the first configured percentage is at least 50% and no more than 200%.

16. The method in claim 5, where the second configured percentage is at least 25% and no more than 100%.

17. The method in claim 5, wherein the third configured period of time is at least the length of time of the first period of time.

18. An apparatus for enabling client devices coupled to a network to play a plurality of media streams, where each of the plurality of media streams are available in a plurality of bandwidths and media types, the apparatus comprising:

a computer, comprising of one or more CPUs, memory, and an interface coupled to a network;

a Signal Buffer, coupled to the computer, configured to hold a configured time period of a digitized media stream;

a Media Exchange Table Handler coupled to the computer, configured to manage the plurality of multimedia streams based on available bandwidth and media type;

a Monitoring Module, coupled to the computer, configured to report on the latency of a media stream at a given point in time;

a Client Stream Processor coupled to the computer, configured to manage the reception of the subscribed streams, including any quality issues associated with those streams; and

an Adaptive Latency Filter, coupled to the computer, configured to accept reports from the Monitoring Module and calculate the length of a signal buffer and the current position of a Client Stream Processor in the Signal Buffer.

19. The apparatus in claim 18, further comprising a Subscription Manager, coupled to the computer, configured to monitor the quality of the incoming signal such as signal strength, reporting such to the Media Exchange Table Handler and commanding the Client Stream Processor associated with the incoming signal to downgrade the signal if the quality of the signal goes below some configured threshold.

20. The apparatus in claim 18, further comprising a Publishing Manager, coupled to the computer, configured to manage the requests for and notifications to client devices of the availability of multimedia streams for subscription.

21. A method for bandwidth balancing multiple media streams among heterogeneous clients, the method comprising:

accepting information about a session;

accepting a media stream information to publish to that session, wherein the media stream is defined by an identifier, bandwidth requirements, and type of media stream;

determining the capabilities of a client to play a media stream based on its bandwidth and player capabilities;

sending media stream information to a client;

monitoring a quality of the media stream as received by the client, wherein the quality is a combination of the actual signal quality and the latency of the client processing of the media stream; and

determining a course of action based on the quality of the media stream, wherein the course of action comprises skipping ahead in the buffer, resetting to the beginning of the buffer, stopping playing the media stream, or selecting a different media stream to play.

22. A computer readable medium having stored thereon programming instructions that, when executed by a computer, causes the computer to perform a method for bandwidth balancing multiple media streams among heterogeneous clients, the medium comprising:

programming instructions for accepting information about a session;

programming instructions for accepting a media stream information to publish to that session, wherein the media stream is defined by an identifier, bandwidth requirements, and type of media stream;

programming instructions for determining the capabilities of a client to play a media stream based on its bandwidth and player capabilities;

programming instructions for sending media stream information to a client;

programming instructions for monitoring a quality of the media stream as received by the client, wherein the quality is a combination of the actual signal quality and the latency of the client processing of the media stream; and

programming instructions for determining a course of action based on the quality of the media stream, wherein the course of action comprises skipping ahead in the buffer, resetting to the beginning of the buffer, stopping playing the media stream, or selecting a different media stream to play.

23. An apparatus for maintaining the publishing and subscription of a media stream among multiple client devices, the apparatus comprising:

a computer, the computer comprising one or more processors, one or more storage devices, a memory, one or more interfaces coupled to a network, and one or more devices capable of playing media streams;

a Media Exchange Table Manager, coupled to the computer, configured to maintain and report information about a media stream associated with a session, including the publisher, session, bandwidth, and media type information;

a Publishing Manager, coupled to the computer, configured to accept information associated with a media stream to be associated with a session, and report information about the media streams associated with a session on request;

a Capability Manager coupled to the computer, configured to accept information from a client device about its capabilities, or infer the capabilities of a client device based on the network it is connected to; and

a Subscription Manager coupled to the computer, configured to accept requests to subscribe to a session or automatically subscribe client devices to media streams, based on their reported or assumed capabilities.

24. A method for maintaining the output quality of a media stream, the method comprising:

initializing a signal buffer to hold at least 30 seconds of a media signal at the start of a session;

measuring the latency of the signal processing relative to the real-time position in the signal buffer;

calculating the derivative of the latency over time;

calculating a first moving average of the latency over the first 30 second time period;

defining a first threshold associated with the first moving average, greater than the first moving average by some fixed percentage; and

determining if the absolute value of the signal latency is above the first threshold, and if so commanding the device performing the signal processing to perform a series of skips of the signal over a first period until the derivative of the signal latency is no longer exceeds the first threshold.

25. The method in claim 24, further comprising:

calculating a second moving average of the derivative of the latency over the most recent 5 second time period;

calculating a third moving average of the derivative of the latency over the most recent 30 second time period; and

determining if the second moving average exceeds the third moving average, and if so commanding the device performing the signal processing to perform a series of skips of the signal over a first period until the third moving average exceeds the second moving average.

26. The method in claim 25, further comprising:

defining a first value, associated with expected number of skips over a first period of time greater than the largest moving average but less than 10 minutes;

defining a second value associated with the first value, greater than the first value by some fixed percentage;

defining a third value associated with the first value, less than the first value by some fixed percentage;

determining if the number of skips over the last first period of time exceeds the second value, and if so command the device performing the signal processing to select a signal of lower quality if available, or stop playing otherwise; and

determining if the number of skips over the last first period of time is less than the third value, and if so command the device performing the signal processing to select a signal of greater quality if available.

27. The method in claim 25, further comprising:

defining a fourth value associated with the maximum number of skips allowed over some period of time, between the first value and the second value;

defining a fifth value associated with the minimum number of skips allowed over some period of time, between the first value and the third value;

defining a sixth value associated with the initial time between signal skips;

determining if the number of skips over the last first period of time exceeds the fourth value, and if so command the device performing the signal processing to skip a larger portion of the signal over a period larger than the sixth value, both increased by a fixed percentage, until the number of skips over the last period of time is below the fourth value; and

determining if the number of skips over the last first period of time is less than the fifth value, and if so command the device performing the signal processing to skip a smaller portion of the signal over a period less than the sixth value, both decreased by a fixed percentage, until the number of skips over the last period of time is above the fifth value.

28. The method in claim 25, further comprising:

determining if the derivative of the signal latency exceeds 150% of the first moving average, and if so commanding the device performing the signal processing to perform a series of skips of the signal over a first period until the derivative of the signal latency no longer exceeds 150% of the first moving average.

29. A computer readable medium having stored thereon instructions that, when executed by a computer, causes the computer to perform a method for maintaining the output quality of a media stream, the medium comprising:

programming instructions for initializing a signal buffer to hold at least 30 seconds of a media signal at the start of a session;

programming instructions for measuring the latency of the signal processing relative to the real-time position in the signal buffer;

programming instructions for calculating the derivative of the latency over time;

programming instructions for calculating a first moving average of the latency over the first 30 second time period;

programming instructions for defining a first threshold associated with the first moving average, greater than the first moving average by some fixed percentage; and

programming instructions for determining if the absolute value of the signal latency is above the first threshold, and if so commanding the device performing the signal processing to perform a series of skips of the signal over a first period until the derivative of the signal latency is no longer exceeds the first threshold.

30. The computer readable media in claim 29, further comprising:

programming instructions for calculating a second moving average of the derivative of the latency over the most recent 5 second time period;

programming instructions for calculating a third moving average of the derivative of the latency over the most recent 30 second time period; and

programming instructions for determining if the second moving average exceeds the third moving average, and if so commanding the device performing the signal processing to perform a series of skips of the signal over a first period until the third moving average exceeds the second moving average.

31. The computer readable media in claim 29, further comprising:

programming instructions for defining a first value, associated with expected number of skips over a first period of time greater than the largest moving average but less than 10 minutes;

programming instructions for defining a second value associated with the first value, greater than the first value by some fixed percentage;

programming instructions for defining a third value associated with the first value, less than the first value by some fixed percentage;

programming instructions for determining if the number of skips over the last first period of time exceeds the second value, and if so command the device performing the signal processing to select a signal of lower quality if available, or stop playing otherwise; and

programming instructions for determining if the number of skips over the last first period of time is less than the third value, and if so command the device performing the signal processing to select a signal of greater quality if available.

32. The computer readable media in claim 29, further comprising:

programming instructions for defining a fourth value associated with the maximum number of skips allowed over some period of time, between the first value and the second value;

programming instructions for defining a fifth value associated with the minimum number of skips allowed over some period of time, between the first value and the third value;

programming instructions for defining a sixth value associated with the initial time between signal skips;

programming instructions for determining if the number of skips over the last first period of time exceeds the fourth value, and if so command the device performing the signal processing to skip a larger portion of the signal over a period larger than the sixth value, both increased by a fixed percentage, until the number of skips over the last period of time is below the fourth value; and

programming instructions for determining if the number of skips over the last first period of time is less than the fifth value, and if so command the device performing the signal processing to skip a smaller portion of the signal over a period less than the sixth value, both decreased by a fixed percentage, until the number of skips over the last period of time is above the fifth value.

33. The computer readable media of claim 29, further comprising:

computer instructions for determining if the derivative of the signal latency exceeds 150% of the first moving average, and if so commanding the device performing the signal processing to perform a series of skips of the signal over a first period until the derivative of the signal latency no longer exceeds 150% of the first moving average.