Regression testing
First Claim
1. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
- evaluating a natural language processing system with a group of input-result pairs, each input-result pair of the group of input-result pairs mapping an input to a result, the evaluating including;
processing each input of the group of input-result pairs with the natural language processing system to identify a result; and
determining, for each input of the group of input-result pairs, whether or not the result of the processing matches the result of the input-result pair; and
in response to determining that a result of the processing does not match a corresponding result of the input-result pair, providing a user interface with an indicator for the result of the processing that does not match the corresponding result of the input-result pair to enable a user to modify the natural language processing system.
3 Assignments
0 Petitions
Accused Products
Abstract
Various embodiments provide a tool, referred to herein as “Active Lab” that can be used to develop, debug, and maintain knowledge bases. These knowledge bases (KBs) can then engage various applications, technology, and communications protocols for the purpose of task automation, real time alerting, system integration, knowledge acquisition, and various forms of peer influence. In at least some embodiments, a KB is used as a virtual assistant that any real person can interact with using their own natural language. The KB can then respond and react however the user wants: answering questions, activating applications, or responding to actions on a web page.
187 Citations
21 Claims
-
1. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
-
evaluating a natural language processing system with a group of input-result pairs, each input-result pair of the group of input-result pairs mapping an input to a result, the evaluating including; processing each input of the group of input-result pairs with the natural language processing system to identify a result; and determining, for each input of the group of input-result pairs, whether or not the result of the processing matches the result of the input-result pair; and in response to determining that a result of the processing does not match a corresponding result of the input-result pair, providing a user interface with an indicator for the result of the processing that does not match the corresponding result of the input-result pair to enable a user to modify the natural language processing system. - View Dependent Claims (2, 3, 4, 5, 6)
-
-
7. A method comprising:
-
identifying a natural language processing system that is associated with a virtual assistant; evaluating the natural language processing system by processing test user input with the natural language processing system, the test user input having been previously associated with a particular result; determining that the processing does not identify the particular result; and causing display of a user interface with an indicator associated with the test user input to enable a user to modify an element of the natural language processing system that is associated with at least one of the test user input or the particular result that has been previously associated with the test user input, the indicator indicating that the processing does not identify the particular result. - View Dependent Claims (8, 9, 10, 11, 12, 13)
-
-
14. A system comprising:
-
one or more processors; and memory communicatively coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising; evaluating a natural language processing system by causing the natural language processing system to process test user input that has been previously associated by a user with a particular result; and in the event that the evaluation indicates that the processing does not satisfy one or more criteria, presenting a user interface with an icon for the test user input to enable at least one of the user or another user to modify the natural language processing system, the icon indicating that the processing of the test user input does not satisfy the one or more criteria. - View Dependent Claims (15, 16, 17, 18, 19, 20)
-
-
21. A system comprising:
-
one or more processors; and memory communicatively coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising; identifying a natural language processing system that is associated with a virtual assistant; performing a first evaluation of the natural language processing system by; accessing a plurality of test user inputs from a database, each of the plurality of test user inputs being associated with a respective predetermined result; performing natural language processing on each of the plurality of test user inputs with the natural language processing system to analyze an accuracy of the natural language processing system; and generating test result information indicating whether or not the natural language processing system identifies the respective predetermined result for each of the plurality of test user inputs; providing a graphical user interface that displays the test result information and that enables a user to provide input regarding a modification to an element of the natural language processing system; based at least in part on the modification to the element of the natural language processing system, performing a second evaluation of the natural language processing system by; accessing the plurality of test user inputs from the database; performing natural language processing on each of the plurality of test user inputs with the natural language processing system to analyze an accuracy of the natural language processing system; and generating additional test result information indicating whether or not the natural language processing system identifies the respective predetermined result for each of the plurality of test user inputs; and causing display of the additional test result information in the graphical user interface.
-
Specification