Method and system for evaluating the quality of packet-switched voice signals

• US 7,085,230 B2
• Filed: 02/07/2001
• Issued: 08/01/2006
• Est. Priority Date: 12/24/1998
• Status: Expired due to Term

• Current Assignee: Far North Patents, LLC
• Original Assignee: MCI LLC (Verizon Communications Inc.)
• Inventors: Hardy, William Christopher
• Primary Examiner(s): Nguyen, Chau
• Assistant Examiner(s): Moore, Ian N.
• Application Number:
  US09/778,186
• Publication Number:
  US 20020114296A1
• Time in Patent Office: 2,001 Days
• Field of Search: 370/252, 370/253, 370/232, 370/233, 370/234, 370/352, 370/356, 370/357, 370/358, 370/242, 370/244, 370/245, 370/250, 704/228, 704/270.1, 704/200, 704/233, 704/234, 709/223, 709/224, 709/225, 709/226
• US Class Current: 370/232
• CPC Class Codes: H04L 41/06 Management of faults, events, alarms or notifications
• CPC Class Codes: H04L 41/5067 Customer-centric QoS measurements
• CPC Class Codes: H04L 43/00 Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/0829 Packet loss
• CPC Class Codes: H04L 43/0841 Round trip packet loss
• CPC Class Codes: H04L 43/0852 Delays
• CPC Class Codes: H04L 43/0864 Round trip delays
• CPC Class Codes: H04L 43/16 Threshold monitoring
• CPC Class Codes: H04L 43/50 Testing arrangements
• CPC Class Codes: H04L 43/55 Testing of service level quality, e.g. simulating service usage
• CPC Class Codes: H04L 65/103 in the network
• CPC Class Codes: H04L 65/104 in the network
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04M 2207/203 composed of PSTN and data network, e.g. the Internet
• CPC Class Codes: H04M 3/2227 Quality of service monitoring
• CPC Class Codes: H04M 3/2236 Quality of speech transmission monitoring
• CPC Class Codes: H04M 3/2254 in networks
• CPC Class Codes: H04M 7/006 Networks other than PSTN/ISDN providing telephone service, e.g. Voice over Internet Protocol (VoIP) , including next generation networks with a packet-switched transport layer H04L65/00 takes precedence; aspects not specific to the type of network H04M3/00; special services in those networks H04M3/42
• CPC Class Codes: H04Q 2213/13034 A/D conversion, code compression/expansion
• CPC Class Codes: H04Q 2213/13093 : Personal computer, PC
• CPC Class Codes: H04Q 2213/13389 : LAN, internet

Abstract

A method and apparatus for determining what level of performance must be maintained by a second communication service so as to be perceived as being of equal quality to a first communication service, especially where the second communication service is subject to additional impairments not experienced by the first communication service. In one example embodiment, the present invention is applied to determining what combinations of packet loss rate and packet delay are tolerable where a packet switched telephony service is desired to be perceived to be substantially equivalent to a traditional toll-quality non-packet-switched telephone service. Empirically derived models are used to relate user perception to objectively measurable characteristics, such as packet loss rate.

Claims

1. A method for determining acceptability of quality of a second communications service, in comparison to a first communications service which is deemed to exhibit acceptable quality, comprising the steps of:

obtaining a first quality index pertaining to the first communications service;

obtaining a second quality index pertaining to the second communications service; and

determining that the second communication service is of unacceptable quality if the second quality index differs from the first quality index service by more than a selected amount.

2. The method of claim 1 wherein said first and second quality indices are mean opinion scores.

3. The method of claim 1 wherein said first and second quality indices relate to an average proportion of communications that would be rated as objectionable by users.

4. The method of claim 1 wherein obtaining a quality index pertaining to the second communications service comprises the steps of:

measuring performance characteristics of the second network; and

computing an expected quality index for the second communications service.

5. The method of claim 4 wherein said step of computing an expected quality index is performed by applying an effects matrix.

6. A method for determining the quality performance required of a second communications service in comparison to a first communications service comprising the steps of:

obtaining a first quality index representing the quality of the first communication service;

determining the effect of at least one performance characteristic of the second communication service upon a second quality index pertaining to the second communication service; and

determining a value for the performance characteristic required to maintain the second quality index acceptably near the value of the first quality index.

7. The method of claim 6 wherein said first and second quality indices are mean opinion scores.

8. The method of claim 6 wherein said first and second quality indices relate to an average proportion of communications that would be rated as objectionable by users.

9. The method of claim 6 wherein said performance characteristic is packet loss rate.

10. The method of claim 6 wherein said performance characteristic is packet delay.

11. A method for determining the quality performance required of a second communications service in comparison to a first communications service comprising the steps of:

obtaining a first quality index representing the quality of the first communication service;

determining the effect of a first performance characteristic of the second communication service upon a second quality index pertaining to the second communication service;

determining the effect of a second performance characteristic of the second communication service upon the second quality index pertaining to the second communication service;

assuming a selected value for the first performance characteristic; and

in the context of the selected value for the first performance characteristic, determining a value for the second performance characteristic required to maintain the second quality index acceptably near the value of the first quality index.

12. The method of claim 11 further comprising the step of determining that the first performance characteristic has an effect upon the second quality index that is independent of any other performance characteristics.

13. The method of claim 11 further comprising the step of selecting a first performance characteristic which has an effect upon the second quality index that is independent of any other performance characteristics.

14. The method of claim 11 further comprising the step of selecting a first performance characteristic which has an effect upon the second quality index that is independent of the second performance characteristic.

15. The method of claim 11 wherein said first and second quality indices relate to an average proportion of communications that would be rated as objectionable by users.

16. The method of claim 11 wherein said first and second quality indices are mean opinion scores.

17. The method of claim 11 wherein said first performance characteristic is packet delay.

18. The method of claim 11 wherein said second performance characteristic is packet loss rate.

19. A method for determining the quality performance required of a second communications service in comparison to a first communications service comprising the steps of:

obtaining a first quality index representing the quality of the first communication service;

determining a second quality index representing the quality of the second communication service subject to at least one degraded performance characteristic;

determining an averaged composite quality index for communications occurring through the second communications network, said averaged composite quality index being an average value resulting from a mixture of first communications occurring without the degraded performance characteristic and second communications occurring with the degraded performance characteristic; and

expressing the required quality performance of the second communication service as a proportion between said first communications and said second communications required to maintain said averaged composite quality index acceptably near the value of the first quality index.

20. The method of claim 19 wherein said first, second, and averaged composite quality indices are mean opinion scores.

21. The method of claim 19 wherein said first, second, and averaged composite quality indices are values associated with a percentage (P) of calls or connections that elicit unusable (U), difficult (D), or irritating (I) responses (P(UDI)).

22. A method for determining how a first performance characteristic having a given value affects the quality of a communication service, the method comprising:

obtaining an original data set pertaining to occurrences of various values of at least one second performance characteristic within the communication service;

determining the effect that the first performance characteristic has upon the occurrences of values of the second performance characteristic;

computing an altered data set by changing, in the original data set, the occurrences of values of the second performance characteristic assuming the first performance characteristic is set to said given value; and

computing a quality index for the communication service based upon the altered data set.

23. The method of claim 22 wherein said step of computing a quality index is performed by convolving the altered data set with an effects matrix.

24. The method of claim 22 wherein said quality index is a mean opinion score.

25. The method of claim 22 wherein said quality index relates to an average proportion of communications that would be rated as objectionable by users.

26. A method for determining acceptable quality of a second communication service, in comparison to a first communication service which exhibits acceptable quality, comprising the steps of:

measuring at least one performance characteristic for the first communication service;

from the measured performance characteristic for the first communication service, determining a first mean opinion score pertaining to the first communication service;

measuring least one performance characteristic for the second communication service;

from the measured performance characteristic for the second communication service, determining a second mean opinion score pertaining to the second communication service; and

determining that the second communication service is of unacceptable quality if the second mean opinion score is less than the first mean opinion score by more than a perceptible difference threshold.

27. The method of claim 26 further comprising the step of determining a P(UDI) value of the second communication service, said P(UDI) value relate to an average proportion of communications that would be rated as objectionable by users, and determining that the second communications is of unacceptable quality if the P(UDI) exceeds a threshold value.

28. The method of claim 27 wherein said threshold value for the P(UDI) of the second communications service is 0.06.

29. The method of claim 26 wherein said second communication service is subject to at least one impairment that does not affect the first communication service.

30. The method of claim 29 wherein said impairment is packet loss.

31. The method of claim 29 wherein said impairment is packet delay.

32. A method for determining the performance required of a second communications service to achieve acceptable quality in comparison to a first communications service, comprising the steps of:

obtaining a permissible P(UDI) limit, where P(UDI) relates to a percentage (P) of calls or connections that elicit unusable (U), difficult (D), or irritating (I) responses;

obtaining a first mean opinion score representing the quality of the first communication service;

determining a value of packet delay for the second communications service;

determining a reduced P(UDI) margin by subtracting the effects of the value of packet delay from the permissible P(UDI) limit;

obtaining an original data set comprising occurrences of values for performance characteristics absent the effects of packet loss;

transforming the original data set into a transformed data set based upon a given packet loss rate;

applying an effects matrix to the transformed data set to compute a predicted second mean opinion score and second P(UDI) for the second communications service;

determining a first maximum proportion of communications that may exhibit said second mean opinion score while maintaining an averaged mean opinion score that is acceptably close to said first mean opinion score;

determining a second maximum proportion of communications that may exhibit said second P(UDI) such that the averaged P(UDI) remains less than the reduced P(UDI) margin; and

selecting the lesser of the first and second maximum proportions as the maximum proportion of communications in the second communications service that may experience the given packet delay and packet loss rate yet still allow the second communications service to be perceived as having the same quality as the first communications service.