Metrics-related testing of an operational support system (OSS) of an incumbent provider for compliance with a regulatory scheme

• US 6,636,585 B2
• Filed: 06/26/2001
• Issued: 10/21/2003
• Est. Priority Date: 06/26/2000
• Status: Expired due to Fees

• Current Assignee: PricewaterhouseCoopers LLP (Pricewaterhousecoopers International Limited)
• Original Assignee: Bearingpoint Incorporated
• Inventors: Blockus, Linda G., King, Charles H. Jr., Salzberg, Alan J., Scott, Jonathan C., Sears, Raymond W. III
• Primary Examiner(s): Kuntz, Curtis
• Assistant Examiner(s): Taylor, Barry W
• Application Number:
  US09/892,243
• Publication Number:
  US 20030086536A1
• Time in Patent Office: 847 Days
• Field of Search: 379/15.01, 379/9.03, 379/2, 379/15.03, 379/22, 379/25, 379/26.01, 379/27.01, 370/244
• US Class Current: 379/22
• CPC Class Codes: H04M 15/00 Arrangements for metering, time-control or time indication ; Metering, charging or billing arrangements for voice wireline or wireless communications, e.g. VoIP
• CPC Class Codes: H04M 15/43 Billing software details
• CPC Class Codes: H04M 15/51 for resellers, retailers or service providers
• CPC Class Codes: H04M 15/70 Administration or customization aspects; Counter-checking correct charges
• CPC Class Codes: H04M 15/73 Validating charges
• CPC Class Codes: H04M 15/81 Dynamic pricing, e.g. change of tariff during call
• CPC Class Codes: H04M 15/82 Criteria or parameters used for performing billing operations
• CPC Class Codes: H04M 15/83 Notification aspects
• CPC Class Codes: H04M 2215/0112 Dynamic pricing, e.g. change of tariff during call
• CPC Class Codes: H04M 2215/2026 Wireless network, e.g. GSM, PCS, TACS
• CPC Class Codes: H04M 2215/32 Involving wireless systems
• CPC Class Codes: H04M 2215/54 Resellers-retail or service providers billing, e.g. agreements with telephone service operator, activation, charging/recharging of accounts
• CPC Class Codes: H04M 2215/70 Administration aspects, modify settings or limits or counter-check correct charges
• CPC Class Codes: H04M 2215/7072 Validate charges
• CPC Class Codes: H04M 2215/78 Metric aspects
• CPC Class Codes: H04M 2215/82 Advice-of-Charge [AOC], i.e. notify subscriber of charges/cumulative charge; meter at the substation
• CPC Class Codes: H04M 3/22 Arrangements for supervision, monitoring or testing
• CPC Class Codes: H04M 3/2254 in networks
• CPC Class Codes: H04M 3/2263 Network management
• CPC Class Codes: H04M 3/242 Software testing
• CPC Class Codes: H04W 4/24 : Accounting or billing

Abstract

Methods are provided for metrics-related testing of one or more operational support systems (OSSs) of an incumbent provider for compliance with a regulatory scheme, the method performed by an independent testing entity attempting to emulate a competitive provider that would access the OSSs in attempting to compete with the incumbent provider in an open competitive market. Each method includes an active testing phase in which one or more OSSs are tested. Performance of the incumbent provider during active testing is evaluated according to predetermined evaluation criteria. Test results are generated according to the evaluation and a final report is issued. The final report includes the test results and provides a sufficient basis for a regulatory entity administering the regulatory scheme to determine compliance of the incumbent provider with the regulatory scheme.

Claims

1. A method for testing one or more operational support systems (OSSs) of an incumbent provider for compliance with a regulatory scheme, the method performed by an independent testing entity attempting to emulate a competitive provider that would access the OSSs in attempting to compete with the incumbent provider in an open competitive market, the method comprising:

conducting active testing of the OSSs with respect to one or more variables;

evaluating performance of the incumbent provider during active testing with respect to the one or more variables according to predetermined evaluation criteria comprising one or more performance metrics, evaluation of incumbent provider performance comprising;

for performance metrics requiring the incumbent provider to meet a particular performance standard, using a one-sample t-test for continuous variable performance metrics and a binomial test for discrete variable performance metrics; and

for performance metrics requiring parity between incumbent provider performance for operations of the incumbent provider associated with its interaction with competitive providers and incumbent provider performance for its internal operations, using a two-sample t-test for continuous variable performance metrics and a hyper-geometric test for discrete variable performance metrics;

generating test results according to the evaluation; and

issuing a final report comprising the test results and providing a sufficient basis for a regulatory entity administering the regulatory scheme to determine compliance of the incumbent provider with the regulatory scheme.

2. The method of claim 1, wherein the testing is performed to allow the regulatory entity to consider whether and to what extent the incumbent provider provides competitive providers non-discriminatory access to, use of, and support in connection with the OSSs relative to that available to the incumbent provider in its internal operations.

3. The method of claim 1, wherein:

the incumbent provider comprises an incumbent local exchange carrier (ILEC);

the competitive providers comprise competitive local exchange carriers (CLECs);

the regulatory entity is associated with a State within the United States of America; and

the regulatory scheme comprises Section 271 of The Telecommunications Act of 1996, codified in the laws of the United States of America at 47 U.S.C. §

§

151 et seq.

4. The method of claim 1, wherein:

a continuous variable performance metric comprises time; and

a discrete variable performance metric comprises whether an event occurred.

5. The method of claim 1, wherein:

incumbent provider operations associated with its interaction with competitive providers comprise retail operations; and

internal incumbent provider operations comprise wholesale operations.

6. The method of claim 1, wherein for continuous variable performance metrics where the sample size is relatively small according to a predetermined scale, evaluation of incumbent provider performance further comprises using a permutation test.

7. The method of claim 1, wherein the test results comprise a p-value representing a significance level.

8. A method for testing one or more operational support systems (OSSs) of an incumbent provider for compliance with a regulatory scheme, the testing being associated with metrics capture, tracking, and reporting activities of the incumbent provider with respect to a particular operational domain, the method being performed by an independent testing entity attempting to emulate a competitive provider that would access the OSSs in attempting to compete with the incumbent provider in an open competitive market, the method comprising for testing of an OSS:

within a data integrity testing component during active testing, to determine whether filtered data the incumbent provider uses to calculate values for each of one or more metrics is accurately derived from raw data the incumbent provider collects in connection with transactions associated with the OSS;

identifying one or more fields in the filtered data that the incumbent provider uses in calculating values for the metric;

identifying a random sample of transactions associated with the OSS;

obtaining from the incumbent provider copies of records associated with the identified transactions, the records containing raw data for these transactions from which filtered data for these transactions in the identified fields was derived; and

for each identified field, determining whether the filtered data for the identified transactions in the identified field is consistent with the raw data for the identified transactions contained in the associated records;

within a metrics replication testing component during active testing, for each of one or more metrics for which the incumbent provider has calculated values;

obtaining from the incumbent provider a description of the metric, filtered data on which the incumbent provider based its calculation of values for the metric, and an algorithm according to which the incumbent provider made its calculation of values for the metric, the filtered data being associated with a specified time period, the filtered data being derived from raw data the incumbent provider collected in connection with transactions associated with the OSS;

re-calculating values for the metric according to the description, data, and algorithm obtained from the incumbent provider; and

determining whether the re-calculated values are consistent with the values calculated by the incumbent provider;

evaluating performance of the incumbent provider during active testing according to predetermined evaluation criteria;

generating test results according to the evaluation; and

issuing a final report comprising the test results and providing a sufficient basis for a regulatory entity administering the regulatory scheme to determine compliance of the incumbent provider with the regulatory scheme.

9. The method of claim 8, wherein the testing is performed to allow the regulatory entity to consider whether and to what extent the incumbent provider provides competitive providers non-discriminatory access to, use of, and support in connection with the OSSs relative to that available to the incumbent provider in its internal operations.

10. The method of claim 8, wherein:

the incumbent provider comprises an incumbent local exchange carrier (ILEC);

the competitive providers comprise competitive local exchange carriers (CLECs);

the regulatory entity is associated with a State within the United States of America; and

the regulatory scheme comprises Section 271 of The Telecommunications Act of 1996, codified in the laws of the United States of America at 47 U.S.C. §

§

151 et seq.

11. The method of claim 8, wherein the regulatory entity requires the incumbent provider to calculate values for the metrics and report calculated values for the metrics to the regulatory entity for oversight purposes.

12. The method of claim 8, wherein:

the operational domain comprises a pre-ordering, ordering, and provisioning domain;

the OSS is associated with pre-ordering, ordering, and provisioning activities of the incumbent provider; and

the transactions comprise orders for local telecommunications services.

13. The method of claim 12, wherein the orders comprise Local Service Requests (LSRs).

14. The method of claim 8, wherein the identified fields are those the incumbent provider uses relatively often in calculating values for the metric.

15. The method of claim 8, wherein identifying the records within the data integrity testing component comprises:

requesting the incumbent provider to provide a location of the records; and

requesting the incumbent provider to identify one or more incumbent provider personnel to explain the records, determination of which identified records to inspect being based at least in part on interaction with the identified incumbent provider personnel.

16. The method of claim 8, further comprising within the data integrity testing component, requesting the incumbent provider to research and explain any discrepancies between the filtered data and corresponding raw data before evaluating the performance of the incumbent provider with respect to the data integrity testing component.

17. The method of claim 8, wherein:

the operational domain comprises a pre-ordering, ordering, and provisioning domain;

the OSS is associated with pre-ordering, ordering, and provisioning activities of the incumbent provider;

the transactions comprise orders for local telecommunications services; and

the method further comprises tracking progress of orders during a test period encompassing the active testing and activities of the testing entity occurring before the active testing, the testing entity requesting from the incumbent provider raw data for orders submitted to the OSS during an initial portion of the test period and filtered data generated for provisioned and pending orders during substantially all of the test period, the testing entity comparing filtered data for provisioned and pending orders with raw data for corresponding submitted orders to determine whether the filtered data is consistent with the raw data.

18. The method of claim 8, further comprising developing one or more software components to implement the obtained algorithms at a computer system of the testing entity.

19. The method of claim 8, further comprising within a statistical analysis testing component during active testing:

accessing correctly calculated values for one or more metrics associated with internal operations of the incumbent provider and correctly calculated values for these metrics associated with operations of the incumbent provider involving competitive providers;

interacting with one or more incumbent provider personnel to determine how data on which the correctly calculated values are based is collected and processed at the incumbent provider to confirm that this data can be expected to allow for correct inferences based on comparisons of correctly calculated values;

statistically comparing the correctly calculated values for internal operations with the correctly calculated values for operations involving competitive providers; and

according to the statistical comparison, drawing one or more inferences as to parity between the correctly calculated values for internal operations and the correctly calculated values for operations involving competitive providers.

20. The method of claim 19, wherein the correctly calculated values are calculated using data collected during an active testing period under conditions that are statistically similar to conditions outside of the active testing period.

21. The method of claim 19, wherein the correctly calculated values are calculated by the testing entity rather than the incumbent provider.

22. The method of claim 19, wherein:

the internal operations comprise wholesale operations; and

the operations involving competitive providers comprise retail operations.

23. The method of claim 19, further comprising accounting for differences in one or more of geography, transaction size, and transaction type between the correctly calculated values for internal operations and the correctly calculated values for operations involving competitive providers in drawing one or more inferences as to parity.

24. The method of claim 8, wherein the operational domain comprises a maintenance and repair domain, the OSS is associated with maintenance and repair activities of the incumbent provider, and the transactions comprise trouble reports associated with local telecommunications services.

25. The method of claim 8, further comprising in a surveillance testing component during active testing:

observing incumbent provider activities within the operational domain;

reviewing incumbent provider documentation within the operational domain;

interviewing incumbent provider personnel identified as being familiar with the operational domain;

generating a process map associated with the operational domain according to the observing, reviewing, and interviewing;

submitting the process map to the incumbent provider for confirmation that the process map is accurate;

upon confirmation that the process map is accurate, using the process map to identify any differences between a process of the incumbent provider used for its wholesale operations within the operational domain and a process of the incumbent provider used for its retail operations within the operational domain; and

evaluating process parity between the wholesale and retail operations of the incumbent provider according to any identified differences.

26. A method for testing one or more operational support systems (OSSs) of an incumbent provider for compliance with a regulatory scheme, the testing being associated with performance metrics capture, tracking, and reporting activities of the incumbent provider across a plurality of operational domains, the method performed by an independent testing entity attempting to emulate a competitive provider that would access the OSSs in attempting to compete with the incumbent provider in an open competitive market, the method comprising:

conducting a metrics data collection and storage test during active testing to evaluate incumbent provider policies and practices for collecting and storing data used for calculating performance metric values, both for internal operations of the incumbent provider and for operations of the incumbent provider associated with its interaction with competitive providers;

conducting a metrics definition documentation and implementation test during active testing to evaluate incumbent provider policies and practices for documenting and implementing performance metric definitions, implementation of performance metrics definitions comprising applying exclusions and business rules in calculating performance metric values based on raw data;

conducting a metrics change management test during active testing to evaluate incumbent provider policies and practices for developing, implementing, monitoring, and reporting changes concerning performance metrics;

conducting a metrics data integrity test during active testing to evaluate incumbent provider policies and practices for processing data necessary to calculate performance metric values;

conducting a metrics calculation and reporting test during active testing to evaluate policies and practices of the incumbent provider for calculating performance metric values, this test comprising;

accessing calculated performance metric values for operations of the incumbent provider associated with its interaction with competitive providers, these values having been calculated by the incumbent provider and thereafter reported to the regulatory entity;

re-calculating the accessed performance metric values;

comparing the re-calculated values with the accessed values to identify any discrepancies;

attempting to reconcile any identified discrepancies between the re-calculated values and the accessed values; and

determining whether the incumbent provider calculated the accessed values consistent with documented instructions for doing so;

conducting a statistical evaluation of transactions test during active testing to evaluate metrics-related service performance of the incumbent provider according to statistical techniques, this test comprising;

accessing test performance metric values calculated based on interaction between the incumbent provider and the testing entity;

accessing benchmark performance metric values calculated based on interaction between the incumbent provider and one or more competitive providers; and

comparing the test performance metric values with the benchmark performance metric values according to statistical techniques to determine whether and to what extent parity exists between the compared values;

evaluating performance of the incumbent provider during active testing according to predetermined evaluation criteria;

generating test results according to the evaluation; and

issuing a final report comprising the test results and providing a sufficient basis for a regulatory entity administering the regulatory scheme to determine a net impact of identified metrics-related issues on the ability of competitive providers to compete with the incumbent provider in an open competitive market.

27. The method of claim 26, wherein the testing is performed to allow the regulatory entity to consider whether and to what extent the incumbent provider provides competitive providers non-discriminatory access to, use of, and support in connection with the OSSs relative to that available to the incumbent provider in its internal operations.

28. The method of claim 26, wherein:

the incumbent provider comprises an incumbent local exchange carrier (ILEC);

the competitive providers comprise competitive local exchange carriers (CLECs);

the regulatory entity is associated with a State within the United States of America; and

the regulatory scheme comprises Section 271 of The Telecommunications Act of 1996, codified in the laws of the United States of America at 47 U.S.C. §

§

151 et seq.

29. The method of claim 26, wherein the regulatory entity requires the incumbent provider to calculate values for metrics and report calculated values for metrics to the regulatory entity for oversight purposes.

30. The method of claim 26, wherein the operational domains comprise one or more of:

a pre-ordering, ordering, and provisioning domain, one or more OSSs being associated with pre-ordering, ordering, and provisioning activities of the incumbent provider;

a maintenance and repair domain, one or more OSSs being associated with maintenance and repair activities of the incumbent provider;

a billing domain, one or more OSSs being associated with billing activities of the incumbent provider; and

a relationship management and infrastructure domain, one or more OSSs being associated with relationship management and infrastructure activities of the incumbent provider.

31. The method of claim 26, wherein the testing is conducted in addition to domain-specific metrics testing in one or more of the operational domains.

32. The method of claim 26, wherein:

internal operations of the incumbent provider comprises wholesale operations; and

operations of the incumbent provider associated with its interaction with competitive providers comprise retail operations.