Apparatus and method for analyzing routing of calls in an automated response system

• US 6,879,685 B1
• Filed: 03/04/2002
• Issued: 04/12/2005
• Est. Priority Date: 03/05/2001
• Status: Expired due to Term

• Current Assignee: Cx360 Incorporated, Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Verizon Corporate Services Group Incorporated (Verizon Communications Inc.)
• Inventors: Peterson, Patrick, Suhm, Bernhard, Godfrey, Katherine
• Primary Examiner(s): Hong, Harry S.
• Application Number:
  US10/090,210
• Time in Patent Office: 1,135 Days
• Field of Search: 379/88.01, 379/88.04, 379/88.18, 379/265.01, 379/265.02, 379/265.06, 379/265.07, 379/265.08, 379/265.11, 379/265.12, 379/265.13, 379/265.14, 705/7, 705/8, 705/9, 705/11
• US Class Current: 379/265.11
• CPC Class Codes: G06Q 10/0639 Performance analysis of employees; Performance analysis of enterprise or organisation operations
• CPC Class Codes: H04M 15/00 Arrangements for metering, time-control or time indication ; Metering, charging or billing arrangements for voice wireline or wireless communications, e.g. VoIP
• CPC Class Codes: H04M 15/43 Billing software details
• CPC Class Codes: H04M 15/58 based on statistics of usage or network monitoring
• CPC Class Codes: H04M 2203/555 Statistics, e.g. about subscribers but not being call statistics
• CPC Class Codes: H04M 2215/0188 Network monitoring; statistics on usage on called/calling number
• CPC Class Codes: H04M 3/36 Statistical metering, e.g. recording occasions when traffic exceeds capacity of trunks
• CPC Class Codes: H04M 3/493 Interactive information services, e.g. directory enquiries ; Arrangements therefor, e.g. interactive voice response [IVR] systems or voice portals
• CPC Class Codes: H04M 3/51 Centralised call answering arrangements requiring operator intervention , e.g. call or contact centers for telemarketing

Abstract

A system for analyzing the performance of an interactive voice response (IVR) system operates to record from end to end plural calls from callers. For a plurality of calls, the system codes and annotates an agent-caller interaction, including a topic discussed during the interaction; detects a routing operation of the IVR system by detecting a sequence of events in the IVR system, with reference to a call-flow file that summarizes the call-flow model of the IVR system, for each call; and cross-tabulates routing information obtained from the detected sequence of events with results from the coded and annotated agent-caller interactions.

Claims

1. A method of analyzing performance of an interactive voice response (IVR) system of a call processing center with respect to routing of calls received by the IVR system from callers, the IVR system being operable to route the calls in accordance with a call-flow model of the IVR system based on inputs of the callers to the IVR system, the performance being defined as accuracy of routing decisions made by the IVR system compared to where the callers should have been routed in accordance with the callers'"'"' true reasons for calling the call processing center, said method comprising the steps of:

recording from end to end plural calls from the callers;

for a plurality of recorded calls, coding and annotating an agent-caller interaction, including a topic discussed during the interaction to obtain coded and annotated agent-caller interactions;

for a plurality of recorded calls, detecting a routing operation of the IVR system by detecting a sequence of events in the IVR system, with reference to a call-flow file that summarizes the call-flow model of the IVR system;

for a plurality of recorded calls, summarizing the detected sequence of events in a call-sequence file;

compiling a list of call-sequence files;

compiling from the list of call-sequence files a summary file that lists information collected from calls interacting with the IVR system, including routing information and a routing destination out of the IVR system for the calls;

creating summary results for the coded and annotated agent-caller interactions; and

cross-tabulating the routing information in the summary file with the summary results from the coded and annotated agent-caller interactions.

2. A method according to claim 1, wherein said recording step is performed off-site in relation to the IVR system.

3. A method according to claim 1, further comprising the step of generating, from a result of the cross-tabulating step, a confusion matrix that correlates actual routing decisions of the IVR system with true reasons for calling, as ascertained from annotations of agent-caller interactions.

4. A method according to claim 1, wherein said coding and annotating step is performed manually by one or more persons listening to the plurality of recorded calls.

5. A method according to claim 1, wherein said coding and annotating step is performed automatically using topic-recognition software.

6. A method according to claim 1, wherein inputs to the IVR system comprise speech and/or DTMF tones.

7. An apparatus for analyzing performance of an interactive voice response (IVR) system of a call processing center with respect to routing of calls received by the IVR system from callers, the IVR system being operable to route the calls in accordance with a call-flow model of the IVR system based on inputs of callers to the IVR system, the performance being defined as accuracy of routing decisions made by the IVR system compared to where callers should have been routed in accordance with the callers'"'"' true reasons for calling the call processing center, said apparatus comprising:

means for recording from end to end plural calls from callers;

means for, for a plurality of recorded calls, coding and annotating an agent-caller interaction, including a topic discussed during the interaction to obtain coded and annotated agent-caller interactions;

means for, for a plurality of recorded calls, detecting a routing operation of the IVR system by detecting a sequence of events in the IVR system, with reference to a call-flow file that summarizes the call-flow model of the IVR system;

means for, for a plurality of recorded calls, summarizing the detected sequence of events in a call-sequence file;

means for compiling a list of call-sequence files;

means for compiling from the list of call-sequence files a summary file that lists information collected from calls interacting with the IVR system, including routing information and a routing destination out of the IVR system for the calls;

means for creating summary results for the coded and annotated agent-caller interactions; and

means for cross-tabulating the routing information in the summary file with the summary results from the coded and annotated interactions.

8. An apparatus according to claim 7, wherein said means for recording is located off-site in relation to the IVR system.

9. An apparatus according to claim 7, further comprising means for generating, from cross-tabulated information generated by said means for cross-tabulating, a confusion matrix that correlates actual routing decisions of the IVR system with true reasons for calling, as ascertained from annotations of agent-caller interactions.

10. An apparatus according to claim 7, wherein said means for coding and annotating performs coding and annotation automatically using topic-recognition software.

11. An apparatus according to claim 7, wherein inputs to the IVR system comprise speech and/or DTMF tones.

12. A system for analyzing performance of an interactive voice response (IVR) system of a call processing center with respect to routing of calls received by the IVR system from callers, the IVR system being operable to route the calls in accordance with a call-flow model of the IVR system based on inputs of callers to the IVR system, the performance being defined as accuracy of routing decisions made by the IVR system compared to where callers should have been routed in accordance with the callers'"'"' true reasons for calling the call processing center, said system being operable to:

record from end to end plural calls from the callers;

for a plurality of recorded calls, code and annotate an agent-caller interaction, including a topic discussed during the interaction to obtain coded and annotated agent-caller interactions;

for a plurality of recorded calls, detect a routing operation of the IVR system by detecting a sequence of events in the IVR system, with reference to a call-flow file that summarizes the call-flow model of the IVR system;

for a plurality of recorded calls, summarize the detected sequence of events in a call-sequence file;

compile a list of the call-sequence files;

compile from the list of call-sequence files a summary file that lists information collected from callers in the IVR system, including routing information and a routing destination out of the IVR system for each call;

create summary results for the coded and annotated agent-caller interactions; and

cross-tabulate the routing information in the summary file with the summary results from the coded and annotated interactions.

13. A system according to claim 12, wherein recording of calls from end to end is performed off-site in relation to the IVR system.

14. A system according to claim 12, said system further being operable to generate, from cross-tabulated information, a confusion matrix that correlates actual routing decisions of the IVR system with true reasons for calling, as ascertained from annotations of agent-caller interactions.

15. A system according to claim 12, wherein coding and annotation are performed automatically by a topic-recognition algorithm.

16. A system according to claim 12, wherein inputs to the IVR system comprise speech and/or DTMF tones.

17. A method of analyzing performance of an interactive voice response (IVR) system of a call processing center with respect to routing of calls received by the IVR system from callers, the IVR system being operable to route the calls in accordance with a call-flow model of the IVR system based on inputs of the callers to the IVR system, the performance being defined as accuracy of routing decisions made by the IVR system compared to where the callers should have been routed in accordance with the callers'"'"' true reasons for calling the call processing center, said method comprising the steps of:

recording from end to end plural calls from the callers;

for a plurality of recorded calls, coding and annotating an agent-caller interaction, including a topic discussed during the interaction to obtain coded and annotated agent-caller interactions;

for a plurality of recorded calls, detecting a routing operation of the IVR system by detecting a sequence of events in the IVR system, with reference to a call-flow file that summarizes the call-flow model of the IVR system; and

cross-tabulating routing information obtained from the detected sequence of events with results from the coded and annotated agent-caller interactions.

18. An apparatus for analyzing performance of an interactive voice response (IVR) system of a call processing center with respect to routing of calls received by the IVR system from callers, the IVR system being operable to route the calls in accordance with a call-flow model of the IVR system based on inputs of the callers to the IVR system, the performance being defined as accuracy of routing decisions made by the IVR system compared to where the callers should have been routed in accordance with the callers'"'"' true reasons for calling the call processing center, said apparatus comprising:

means for recording from end to end plural calls from the callers;

means for, for a plurality of recorded calls, coding and annotating an agent-caller interaction, including a topic discussed during the interaction to obtain coded and annotated agent-caller interactions;

means for, for a plurality of recorded calls, detecting a routing operation of the IVR system by detecting a sequence of events in the IVR system, with reference to a call-flow file that summarizes the call-flow model of the IVR system; and

means for cross-tabulating routing information obtained from the detected sequence of events with results from the coded and annotated agent-caller interactions.

19. A system for analyzing performance of an interactive voice response (IVR) system of a call processing center with respect to routing of calls received by the IVR system from callers, the IVR system being operable to route the calls in accordance with a call-flow model of the IVR system based on inputs of the callers to the IVR system, the performance being defined as accuracy of routing decisions made by the IVR system compared to where the callers should have been routed in accordance with the callers'"'"' true reasons for calling the call processing center, said system being operable to:

record from end to end plural calls from the callers;

for a plurality of recorded calls, code and annotate an agent-caller interaction, including a topic discussed during the interaction to obtain coded and annotated agent-caller interactions;

for a plurality of recorded calls, detect a routing operation of the IVR system by detecting a sequence of events in the IVR system, with reference to a call-flow file that summarizes the call-flow model of the IVR system; and

cross-tabulate routing information obtained from the detected sequence of events with results from the coded and annotated agent-caller interactions.

20. A method of analyzing performance of an automated response system of a contact processing center with respect to routing of contacts received by the automated response system from contactors, the automated response system being operable to route the contacts in accordance with a contact-flow model of the automated response system based on inputs of the contactors to the automated response system, the performance being defined as accuracy of routing decisions made by the automated response system compared to where the contactors should have been routed in accordance with the contactors'"'"' true reasons for contacting the contact processing center, said method comprising the steps of:

recording from end to end plural contacts from the contactors;

for a plurality of recorded contacts, coding and annotating an agent-contactor interaction, including a topic discussed during the interaction to obtain coded and annotated agent-caller interactions;

for a plurality of recorded contacts, detecting a routing operation of the automated response system by detecting a sequence of events in the automated response system, with reference to a contact-flow file that summarizes the contact-flow model of the automated response system; and

cross-tabulating routing information obtained from the detected sequence of events with results from the coded and annotated agent-contactor interactions.

21. An apparatus for analyzing performance of an automated response system of a contact processing center with respect to routing of contacts received by the automated response system from contactors, the automated response system being operable to route the contacts in accordance with a contact-flow model of the automated response system based on inputs of the contactors to the automated response system, the performance being defined as accuracy of routing decisions made by the automated response system compared to where the contactors should have been routed in accordance with the contactors'"'"' true reasons for contacting the contact processing center, said apparatus comprising:

means for recording from end to end plural contacts from the contactors;

means for, for a plurality of recorded contacts, coding and annotating an agent-contactor interaction,

including a topic discussed during the interaction to obtain coded and annotated agent-caller interactions;

means for, for a plurality of recorded contacts, detecting a routing operation of the automated response system by detecting a sequence of events in the automated response system, with reference to a contact-flow file that summarizes the contact-flow model of the automated response system; and

means for cross-tabulating routing information obtained from the detected sequence of events with results from the coded and annotated agent-contactor interactions.

22. A system for analyzing performance of an automated response system of a contact processing center with respect to routing of contacts received by the automated response system from contactors, the automated response system being operable to route the contacts in accordance with a contact-flow model of the automated response system based on inputs of the contactors to the automated response system, the performance being defined as accuracy of routing decisions made by the automated response system compared to where the contactors should have been routed in accordance with the contactors'"'"' true reasons for calling the contact processing center, said system being operable to:

record from end to end plural contacts from the contactors;

for a plurality of recorded contacts, code and annotate an agent-contactor interaction, including a topic discussed during the interaction to obtain coded and annotated agent-caller interactions;

for a plurality of recorded contacts, detect a routing operation of the automated response system by detecting a sequence of events in the automated response system, with reference to a contact-flow file that summarizes the contact-flow model of the automated response system; and

cross-tabulate routing information obtained from the detected sequence of events with results from the coded and annotated agent-contactor interactions.