System and method for eliciting open-ended natural language responses to questions to train natural language processors

• US 9,786,277 B2
• Filed: 09/06/2016
• Issued: 10/10/2017
• Est. Priority Date: 09/07/2015
• Status: Active Grant

• Current Assignee: Cerence Inc.
• Original Assignee: VoiceBox Technologies Corporation (Microsoft Corporation)
• Inventors: Rothwell, Spencer John, Braga, Daniela, Elshenawy, Ahmad Khamis, Carter, Stephen Steele
• Primary Examiner(s): Sirjani, Fariba
• Application Number:
  US15/257,217
• Publication Number:
  US 20170068659A1
• Time in Patent Office: 399 Days
• Field of Search: None
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 40/232 Orthographic correction, e.g. spell checking or vowelisation
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Systems and methods gathering text commands in response to a command context using a first crowdsourced are discussed herein. A command context for a natural language processing system may be identified, where the command context is associated with a command context condition to provide commands to the natural language processing system. One or more command creators associated with one or more command creation devices may be selected. A first application one the one or more command creation devices may be configured to display command creation instructions for each of the one or more command creators to provide text commands that satisfy the command context, and to display a field for capturing a user-generated text entry to satisfy the command creation condition in accordance with the command creation instructions. Systems and methods for reviewing the text commands using second and crowdsourced jobs are also presented herein.

Claims

1. A computer-implemented method for eliciting command variants from crowd users to train a natural language processing system, the method being implemented in a computer system having one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the computer system to perform the method, the method comprising:

identifying a computer recognized command to be processed by a computer;

generating at least a first task, wherein the first task includes command creation instructions that prompt a user to input a variant of the computer recognized command;

transmitting the first task to a plurality of users via a network;

receiving, from at least a first user among the plurality of users, a first user-defined variant of the computer recognized command;

receiving, from at least a second user among the plurality of users, a second user-defined variant of the computer recognized command;

storing the first user-defined variant and the second user-defined variant in association with the computer recognized command;

generating at least a second task, wherein the second task includes command review instructions that prompt a given user to verify the first user-defined variant;

transmitting the second task to a second plurality of users via the network;

receiving, from at least a second user among the second plurality of users, a verification of the first user-defined variant; and

training the natural language processing system to recognize variants of the computer recognized command based on the verified first user-defined variant and the second user-defined variant.

2. The method of claim 1, wherein the first task configures a first application executing on a command creation device to display a plurality of computer recognized commands, and to capture, for each of the plurality of computer recognized commands, a plurality of user-defined variants.

3. The method of claim 1, wherein the first task configures the first application to periodically display a non-machine readable caption on the first application.

4. The method of claim 1, wherein the non-machine readable caption comprises one or more of a non-machine readable image and a non-machine readable sound bite.

5. The method of claim 1, wherein the command review instructions instruct the second plurality of users to review the syntax of the first user-defined variant.

6. The method of claim 1, wherein the command review instructions instruct the second plurality of users to compare the first user-defined variant with model text language variants for the computer recognized command.

7. The method of claim 6, wherein the command review instructions further includes the first user-defined variant and the computer recognized command alongside the model text language variants, the method further comprising:

receiving an indication from each of the second plurality of users of whether the first user-defined variant is similar to one of the model text language variants.

8. The method of claim 1, further comprising checking syntax of the first user-defined variant.

9. The method of claim 1, wherein storing the first user-defined variant and the second user-defined variant comprises storing the first user-defined variant and the second user-defined variant in a transcription library.

10. The method of claim 1, further comprising using the first user-defined variant and the second user-defined variant for transcription of voice data during an end-user deployment phase of the natural language processing system.

11. The method of claim 1, further comprising:

counting a number of times that a given user-defined variant of the computer recognized command has been received from the plurality of users;

determining that the number of times exceeds a threshold number;

preventing further tasks from accepting the given user-defined variant as a variant of the computer recognized command responsive to the determination that the number of times exceeds the threshold number so as to elicit other user-defined variants of the computer recognized command.

12. The method of claim 1, wherein transmitting the first task to the plurality of users comprises transmitting the first task via a crowd sourced job.

13. A system for eliciting command variants from crowd users to train a natural language processing system comprising:

a memory;

one or more physical processors programmed with one or more computer program instructions which, when executed, cause the one or more physical processors to;

identify a computer recognized command to be processed by a computer;

generate at least a first task, wherein the first task includes command creation instructions that prompt a user to input a variant of the computer recognized command;

transmit the first task to a plurality of users via a network;

receive, from at least a first user among the plurality of users, a first user-defined variant of the computer recognized command;

receive, from at least a second user among the plurality of users, a second user-defined variant of the computer recognized command; and

store the first user-defined variant and the second user-defined variant in association with the computer recognized command;

generate at least a second task, wherein the second task includes command review instructions that prompt a given user to verify the first user-defined variant;

transmit the second task to a second plurality of users via the network;

receive, from at least a second user among the second plurality of users, a verification of the first user-defined variant; and

train the natural language processing system to recognize variants of the computer recognized command based on the verified first user-defined variant and the second user-defined variant.

14. The system of claim 13, wherein the first task configures a first application executing on a command creation device to display a plurality of computer recognized commands, and to capture, for each of the plurality of computer recognized commands, a plurality of user-defined variants.

15. The system of claim 13, wherein the first task configures the first application to periodically display a non-machine readable caption on the first application.

16. The system of claim 13, wherein the non-machine readable caption comprises one or more of a non-machine readable image and a non-machine readable sound bite.

17. The system of claim 13, wherein the command review instructions instruct the second plurality of users to review the syntax of the first user-defined variant.

18. The system of claim 13, wherein the command review instructions instruct the second plurality of users to compare the first user-defined variant with model text language variants for the computer recognized command.

19. The system of claim 18, wherein the command review instructions further includes the first user-defined variant and the computer recognized command alongside the model text language variants, and wherein the instructions which, when executed, further cause the one or more physical processors to receive an indication from each of the second plurality of users of whether the first user-defined variant is similar to one of the model text language variants.

20. The system of claim 13, further comprising checking syntax of the first user-defined variant.

21. The system of claim 13, wherein to store the first user-defined variant and the second user-defined variant, the one or more physical processors are further cause to store the first user-defined variant and the second user-defined variant in a transcription library.

22. The system of claim 13, wherein the instructions which, when executed, further cause the one or more physical processors to use the first user-defined variant and the second user-defined variant for transcription of voice data during an end-user deployment phase of the natural language processing system.

23. A computer program product comprising:

one or more tangible, non-transitory computer-readable storage devices;

program instructions, stored on at least one of the one or more tangible, non-transitory computer-readable tangible storage devices that, when executed, cause a computer to;

identify a computer recognized command to be processed by a computer;

generate at least a first task, wherein the first task includes command creation instructions that prompt a user to input a variant of the computer recognized command;

transmit the first task to a plurality of users via a network;

receive, from at least a first user among the plurality of users, a first user-defined variant of the computer recognized command;

receive, from at least a second user among the plurality of users, a second user-defined variant of the computer recognized command;

store the first user-defined variant and the second user-defined variant in association with the computer recognized command;

generate at least a second task, wherein the second task includes command review instructions that prompt a given user to verify the first user-defined variant;

transmit the second task to a second plurality of users via the network;

receive, from at least a second user among the second plurality of users, a verification of the first user-defined variant; and

train the natural language processing system to recognize variants of the computer recognized command based on the verified first user-defined variant and the second user-defined variant.