Method and system for evaluating quality of service for streaming audio and video

• US 7,197,557 B1
• Filed: 05/29/2001
• Issued: 03/27/2007
• Est. Priority Date: 05/29/2001
• Status: Expired due to Term

• Current Assignee: Dynatrace LLC (Dynatrace, Inc.)
• Original Assignee: Keynote Systems (Dynatrace, Inc.)
• Inventors: Asar, Adnan, Aoki, Don, Yee, Kent, Gupta, Umang, Wolf, Ronald, Peng, Ou-Dan, Parks, Matthew, Overton, Chris, Spicer, Nicholas, Bright, Kenny
• Primary Examiner(s): PWU, JEFFREY C
• Application Number:
  US09/870,127
• Time in Patent Office: 2,128 Days
• Field of Search: 709224-226, 709231-232, 709/203, 709/200, 709/202, 709/105, 709/208, 709/209, 709/210, 709/217, 709/218, 709/219, 709/235, 709/238, 709/220, 714/4, 714/2, 725/87, 725/98, 386/95, 707/104.1, 707/103.R
• US Class Current: 709/224
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 41/5009 Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF]
• CPC Class Codes: H04L 41/509 wherein the managed service relates to media content delivery, e.g. audio, video or TV
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 43/12 Network monitoring probes
• CPC Class Codes: H04L 65/612 for unicast
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A system for measuring performance of streaming media sent from a server over a network. The system includes a data acquisition agent connected to the network and operable to send a request for streaming media to the server. The agent is configured to receive streaming media, collect performance measurements, and send the performance measurements to a storage device.

Claims

1. A method of measuring performance of streaming media over a network, the method comprising:

connecting a data acquisition agent to the network;

receiving a list of measurement targets, associated configuration parameters, and schedule at the data acquisition agent;

sending a request for streaming media, based on the received list of measurement targets and schedule, from the data acquisition agent to a media source identified as one of the measurement targets connected to the network;

receiving streaming media from the measurement target in response to the request for media at the data acquisition agent;

monitoring the received streaming media;

collecting performance measurements based on the received streaming media, said performance measurements comprising initial connection and redirection times; and

sending the collected performance measurements to a data collection server configured for collecting and storing data received from a plurality of data acquisition agents.

2. The method of claim 1 wherein receiving streaming media comprises receiving streaming audio.

3. The method of claim 1 wherein receiving streaming media comprises receiving streaming video.

4. The method of claim 1 wherein receiving streaming media comprises receiving streaming audio and video.

5. The method of claim 1 wherein collecting performance measurements comprises collecting number of redirects.

6. The method of claim 1 wherein collecting performance measurements comprises collecting packet information.

7. The method of claim 1 wherein collecting performance measurements comprises collecting rebuffer information.

8. The method of claim 1 wherein collecting performance measurements comprises collecting frame information.

9. The method of claim 1 wherein collecting performance measurements comprises collecting bandwidth information.

10. The method of claim 1 wherein collecting performance measurements comprises identifying the codec used to encode a clip.

11. The method of claim 1 wherein collecting performance measurements comprises identifying errors.

12. The method of claim 1 wherein the network is the Internet.

13. The method of claim 12 wherein the media source is a Web server.

14. The method of claim 1 wherein connecting a data acquisition agent to the network comprises connecting a plurality of data acquisition agents to the network at a plurality of locations.

15. The method of claim 14 wherein each of the data acquisition agents include an internal clock coordinated with the other data acquisition agent clocks.

16. The method of claim 1 further comprising displaying the performance measurements on a Web site.

17. The method of claim 1 further comprising determining a stream quality rating based on the collected performance measurements.

18. The method of claim 1 wherein collecting performance measurements comprises collecting measurements for a predetermined period of time.

19. The method of claim 1 wherein collecting performance measurements comprises collecting measurements at predetermined intervals.

20. The method of claim 1 wherein collecting performance measurements comprises collecting measurements at specified times.

21. The method of claim 1 wherein the configuration parameters comprise a URL containing media to be played and performance measurements to be collected.

22. The method of claim 1 further comprising sending schedule information to a schedule generator and receiving the schedule.

23. The method of claim 22 wherein the schedule information includes playback time.

24. The method of claim 1 wherein connecting a data acquisition agent to the network comprises utilizing a connection other than a dial-up connection.

25. A method of measuring performance of streaming media over a network, the method comprising:

connecting a data acquisition agent to the network;

receiving a list of measurement targets, associated configuration parameters, and schedule at the data acquisition agent;

sending a request for streaming media, based on the received list of measurement targets and schedule, from the data acquisition agent to a media source identified as one of the measurement targets connected to the network;

receiving streaming media from the measurement target in response to the request for media at the data acquisition agent;

monitoring the streaming media;

collecting performance measurements for the streaming media;

sending the performance measurements to a data collection server configured for collecting and storing data received from a plurality of data acquisition agents; and

determining a stream quality rating based on the collected performance measurements,

wherein the stream quality rating is calculated based on a percentage of a startup score, audio score, and video score.

26. The method of claim 25 wherein the video and audio scores are based on rendering and encoding scores.

27. The method of claim 26 wherein the rendering scores are based on relative packets and bandwidth.

28. The method of claim 26 wherein the encoding scores are based on bandwidth data.

29. A computer program product for measuring streaming media over a network, the product comprising:

computer code that connects a data acquisition agent to the network;

computer code that is configured for reading a received list of measurement targets at the data acquisition agent, associated configuration parameters, and schedule;

computer code that sends a request for streaming media, based on the received list of measurement targets and schedule, from the data acquisition agent to a media source connected to the network;

computer code that receives streaming media in response to the request for media at the data acquisition agent;

computer code that monitors the received streaming media;

computer code that collects performance measurements based on the received streaming media, said performance measurements comprising initial connection and redirection times;

computer code that sends the collected performance measurements to a data collection server configured for collecting and storing data received from a plurality of data acquisition agents; and

a computer readable storage medium that stores said computer codes.

30. The computer product of claim 29 wherein the computer readable medium is selected from the group consisting of CD-ROM, floppy disk, tape, flash memory, system memory, hard drive, and a data signal embodied in a carrier wave.

31. A system for measuring performance of streaming media sent from a server over a network, the system comprising:

a preference delivery device operable to generate a list of measurement targets and associated configuration parameters;

a scheduler operable to generate a schedule based on data received for each of the measurement targets; and

a data acquisition agent connected to the network and operable to receive the list of measurement targets, the configuration parameters associated with each of said measurement targets, and the schedule for requesting streaming media from each of said measurement targets;

send a request for streaming media to the server, the agent configured to receive the streaming media, collect performance measurements based on the received streaming media, and send the collected performance measurements to a storage device, said performance measurements comprising initial connection and redirection times.

32. The system of claim 31 wherein the data acquisition agent includes a media player.

33. The system of claim 31 further comprising a data delivery system operable to receive data collected by the agent and store said data in a database.

34. The system of claim 31 wherein the performance measurements include a list of errors.

35. The system of claim 31 wherein the performance measurements comprise packet information.

36. The system of claim 31 wherein the performance measurements comprise frame information.

37. The system of claim 31 wherein the performance measurements comprise bandwidth information.

38. The system of claim 31 wherein the performance measurements comprise the codec used to encode a clip.

39. The system of claim 31 wherein the network is the Internet.

40. The system of claim 39 wherein the server is a Web server.

41. The system of claim 31 further comprising a plurality of data acquisition agents located in different geographic locations.

42. The system of claim 31 further comprising a processor configured to determine a stream quality rating based on the collected performance measurements.

43. The system of claim 42 wherein the stream quality rating is based on a startup score, audio score, and video score.

44. The system of claim 43 wherein the video and audio scores are based on rendering and encoding scores.

45. The system of claim 44 wherein the rendering scores are based on relative packets and bandwidth.

46. The system of claim 44 wherein the encoding scores are based on bandwidth data.

47. A system for measuring performance of streaming media sent from a server over a network, the system comprising:

a preference delivery device operable to generate a list of measurement targets and associated configuration parameters;

a scheduler operable to generate a schedule based on data received for each of the measurement targets; and

a data acquisition agent connected to the network and operable to receive the list of measurement targets, the configuration parameters associated with each of said measurement targets, and the schedule for requesting streaming media from each of said measurement targets;

send a request for streaming media to the server, the agent being configured to receive the streaming media, collect performance measurements, and send the performance measurements to a storage device;

wherein the performance measurements comprise initial connection and redirection times.

48. A system for measuring performance of streaming media sent from a server over a network, the system comprising:

a preference delivery device operable to generate a list of measurement targets and associated configuration parameters;

a scheduler operable to generate a schedule based on data received for each of the measurement targets; and

a data acquisition agent connected to the network and operable to receive the list of measurement targets, the configuration parameters associated with each of said measurement targets, and the schedule for requesting streaming media from each of said measurement targets;

send a request for streaming media to the server, the agent being configured to receive the streaming media, collect performance measurements, and send the performance measurements to a storage device;

wherein the performance measurements comprise number of redirects.

49. A system for measuring performance of streaming media sent from a server over a network, the system comprising

a preference delivery device operable to generate a list of measurement targets and associated configuration parameters;

a scheduler operable to generate a schedule based on data received for each of the measurement targets; and

a data acquisition agent connected to the network and operable to receive the list of measurement targets, the configuration parameters associated with each of said measurement targets, and the schedule for requesting streaming media from each of said measurement targets;

send a request for streaming media to the server, the agent being configured to receive the streaming media, collect performance measurements, and send the performance measurements to a storage device;

wherein the performance measurements comprise rebuffer information.

50. A system for measuring performance of streaming media sent from a server over a network, the system comprising:

a preference delivery device operable to generate a list of measurement targets and associated configuration parameters;

a scheduler operable to generate a schedule based on data received for each of the measurement targets; and

a data acquisition agent connected to the network and operable to receive the list of measurement targets, the configuration parameters associated with each of said measurement targets, and the schedule for requesting streaming media from each of said measurement targets;

send a request for streaming media to the server, the agent configured to receive the streaming media, collect performance measurements based on the received streaming media, and send the collected performance measurements to a storage device, wherein the agent is configured to at least temporarily store the collected performance measurements.

51. A system for measuring performance of streaming media sent from a server over a network, the system comprising:

a preference delivery device operable to generate a list of measurement targets and associated configuration parameters;

a scheduler operable to generate a schedule based on data received for each of the measurement targets, wherein the scheduler is in communication with the data acquisition agent and is configured to compute a schedule based on measurement interval, start time, and end time for each of the measurement targets; and

a data acquisition agent connected to the network and operable to receive the list of measurement targets, the configuration parameters associated with each of said measurement targets, and the schedule for requesting streaming media from each of said measurement targets;

send a request for streaming media to the server, the agent configured to receive the streaming media, collect performance measurements based on the received streaming media, and send the collected performance measurements to a storage device.

52. A method of measuring performance of streaming media over a network, the method comprising:

receiving at a data acquisition agent, a list of measurement targets, associated configuration parameters, and a schedule file;

sending a request for streaming media from the data acquisition agent to one of the measurement targets at a time specified in the schedule file;

receiving streaming media in response to the request for media at the data acquisition agent;

collecting performance measurements for the streaming media; and

sending performance measurements to a storage device;

wherein the data acquisition agent is configured to at least temporarily store the collected performance measurements.

53. The method of claim 52 wherein receiving a schedule file comprises:

generating a measurement schedule at the data acquisition agent based on the list of measurement targets and associated configuration parameters;

sending the measurement schedule to a schedule generator; and

receiving a schedule file from the schedule generator.

54. A method of measuring performance of streaming media over a network, the method comprising:

receiving at a data acquisition agent, a list of measurement targets, associated configuration parameters, and a schedule file;

sending a request for streaming media from the data acquisition agent to one of the measurement targets at a time specified in the schedule file;

receiving streaming media in response to the request for media at the data acquisition agent;

collecting performance measurements for the streaming media; and

sending performance measurements to a storage device;

wherein the performance measurements comprise initial connection time, redirection time, and buffering time.

55. A method of measuring performance of streaming media over a network, the method comprising:

receiving at a data acquisition agent, a list of measurement targets, associated configuration parameters, and a schedule file;

sending a request for streaming media from the data acquisition agent to one of the measurement targets at a time specified in the schedule file;

receiving streaming media in response to the request for media at the data acquisition agent;

collecting performance measurements for the streaming media; and

sending performance measurements to a storage device;

wherein the performance measurements comprise;

average frames per second, average video and audio bandwidth, and encoding rate.

56. The A method of measuring performance of streaming media over a network, the method comprising:

receiving at a data acquisition agent, a list of measurement targets, associated configuration parameters, and a schedule file;

sending a request for streaming media from the data acquisition agent to one of the measurement targets at a time specified in the schedule file;

receiving streaming media in response to the request for media at the data acquisition agent;

collecting performance measurements for the streaming media;

sending performance measurements to a storage device; and

calculating a stream quality rating based on a percentage of a startup score, audio score, and video score.

57. The method of claim 56 wherein the audio and video score account for encoding, delivery, and rendering factors.