Regression testing

• US 9,589,579 B2
• Filed: 06/11/2014
• Issued: 03/07/2017
• Est. Priority Date: 01/15/2008
• Status: Active Grant

• Current Assignee: Verint Americas Incorporated (Verint Systems Incorporated)
• Original Assignee: Next IT Corporation (Verint Systems Incorporated)
• Inventors: Brown, Fred, Zartler, Mark, Miller, Tanya M, Buzan, Scott
• Primary Examiner(s): Utama, Robert J
• Application Number:
  US14/302,096
• Publication Number:
  US 20140343924A1
• Time in Patent Office: 1,000 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/63 Querying
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 25/48 specially adapted for particular use

Abstract

Various embodiments provide a tool, referred to herein as “Active Lab” that can be used to develop, debug, and maintain knowledge bases. These knowledge bases (KBs) can then engage various applications, technology, and communications protocols for the purpose of task automation, real time alerting, system integration, knowledge acquisition, and various forms of peer influence. In at least some embodiments, a KB is used as a virtual assistant that any real person can interact with using their own natural language. The KB can then respond and react however the user wants: answering questions, activating applications, or responding to actions on a web page.

Claims

1. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

evaluating a natural language processing system with a group of input-result pairs, each input-result pair of the group of input-result pairs mapping an input to a result, the evaluating including;

processing each input of the group of input-result pairs with the natural language processing system to identify a result; and

determining, for each input of the group of input-result pairs, whether or not the result of the processing matches the result of the input-result pair; and

in response to determining that a result of the processing does not match a corresponding result of the input-result pair, providing a user interface with an indicator for the result of the processing that does not match the corresponding result of the input-result pair to enable a user to modify the natural language processing system.

2. The one or more non-transitory computer-readable storage media of claim 1, wherein the natural language processing system is implemented at least in part by a knowledge base, the knowledge base including a set of concept-based units that are each paired with a response.

3. The one or more non-transitory computer-readable storage media of claim 1, wherein each result of the group of input-results pairs comprises at least one of:

a concept of the corresponding input;

a unit that includes a pattern of concepts for the corresponding input;

or

a response for the corresponding input.

4. The one or more non-transitory computer-readable storage media of claim 1, wherein the user interface includes elements that enable modification of at least one of:

a response that is determined from the processing;

a concept that is determined from the processing;

a building block that is determined from the processing;

a vocab term that is determined from the processing;

a variable that is utilized for the processing;

an application call that is triggered by the processing;

a pattern of concepts that is utilized for the processing;

a pattern of units that is utilized for the processing;

a data mining process;

or

a cascade that is utilized for the processing.

5. The one or more non-transitory computer-readable storage media of claim 1, wherein the operations further comprise:

after modifying the natural language processing system, processing additional input with the natural language processing system to determine whether or not a result of processing the additional input matches a result that has been previously associated by a user with the additional input.

6. The one or more non-transitory computer-readable storage media of claim 1, wherein each input of the group of input-result pairs is mapped to a result by a user.

7. A method comprising:

identifying a natural language processing system that is associated with a virtual assistant;

evaluating the natural language processing system by processing test user input with the natural language processing system, the test user input having been previously associated with a particular result;

determining that the processing does not identify the particular result; and

causing display of a user interface with an indicator associated with the test user input to enable a user to modify an element of the natural language processing system that is associated with at least one of the test user input or the particular result that has been previously associated with the test user input, the indicator indicating that the processing does not identify the particular result.

8. The method of claim 7, wherein the test user input is associated with the particular result by at least one of the user or another user.

9. The method of claim 7, wherein the natural language processing system is implemented at least in part with a knowledge base.

10. The method of claim 9, further comprising:

upon modifying the element of the natural language processing system, evaluating the natural language processing system with a batch of test user input that is associated with the knowledge base.

11. The method of claim 7, wherein the natural language processing system utilizes a plurality of components, each of the plurality of components comprising at least one of:

a unit that associates one or more concepts with a response;

a concept;

or

a vocab term comprising a grouping of at least one of unambiguous synonyms of a word or spelling variations of a word.

12. The method of claim 7, wherein the natural language processing system utilizes a plurality of concept-based components, each of the plurality of concept-based components comprising a unit that associates one or more concepts with a response;

and the method further comprises;

presenting information that identifies a concept of the one or more concepts that caused the process to not identify the particular result.

13. The method of claim 7, wherein the element of the natural language processing system that is enabled to be modified by the user comprises at least one of:

a response that is determined from the processing;

a concept that is determined from the processing;

a building block that is determined from the processing;

a vocab term that is determined from the processing;

a variable that is utilized for the processing;

an application call that is triggered by the processing;

a pattern of concepts that is utilized for the processing;

a pattern of units that is utilized for the processing;

a data mining process;

or

a cascade that is utilized for the processing.

14. A system comprising:

one or more processors; and

memory communicatively coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

evaluating a natural language processing system by causing the natural language processing system to process test user input that has been previously associated by a user with a particular result; and

in the event that the evaluation indicates that the processing does not satisfy one or more criteria, presenting a user interface with an icon for the test user input to enable at least one of the user or another user to modify the natural language processing system, the icon indicating that the processing of the test user input does not satisfy the one or more criteria.

15. The system of claim 14, wherein the natural language processing system is implemented at least in part by a knowledge base, the knowledge base including a plurality of concept-based components that each associate a pattern of concepts with a response.

16. The system of claim 15, wherein the particular result that is associated with the test user input comprises a concept-based component of the plurality of concept-based components, and the operations further comprise:

determining that the processing does not satisfy the one or more criteria based at least in part on a concept-based component that is identified from the processing not matching the concept-based component that is associated with the test user input.

17. The system of claim 16, wherein the operations further comprise:

presenting information to the user or the other user that indicates that the concept-based component that is identified from the processing does not match the concept-based component that is associated with the test user input.

18. The system of claim 14, wherein the user interface includes elements that enable modification of at least one of:

a response that is determined from the processing;

a concept that is determined from the processing;

a building block that is determined from the processing;

a vocab term that is determined from the processing;

a variable that is utilized for the processing;

an application call that is triggered by the processing;

a pattern of concepts that is utilized for the processing;

a pattern of units that is utilized for the processing;

a data mining process;

or

a cascade that is utilized for the processing.

19. The system of claim 14, wherein the operations further comprise:

causing a virtual assistant to be presented to facilitate a user conversation;

receiving user input during the conversation;

causing the natural language processing system to process the user input to formulate a response to the user input; and

causing the virtual assistant to provide the response.

20. The system of claim 14, wherein the operations further comprise:

receiving the test user input from the user and receiving input from the user to associate the test user input with the particular result; and

associating the test user input with the particular result.

21. A system comprising:

one or more processors; and

memory communicatively coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

identifying a natural language processing system that is associated with a virtual assistant;

performing a first evaluation of the natural language processing system by;

accessing a plurality of test user inputs from a database, each of the plurality of test user inputs being associated with a respective predetermined result;

performing natural language processing on each of the plurality of test user inputs with the natural language processing system to analyze an accuracy of the natural language processing system; and

generating test result information indicating whether or not the natural language processing system identifies the respective predetermined result for each of the plurality of test user inputs;

providing a graphical user interface that displays the test result information and that enables a user to provide input regarding a modification to an element of the natural language processing system;

based at least in part on the modification to the element of the natural language processing system, performing a second evaluation of the natural language processing system by;

accessing the plurality of test user inputs from the database;

performing natural language processing on each of the plurality of test user inputs with the natural language processing system to analyze an accuracy of the natural language processing system; and

generating additional test result information indicating whether or not the natural language processing system identifies the respective predetermined result for each of the plurality of test user inputs; and

causing display of the additional test result information in the graphical user interface.