System and method of recording utterances using unmanaged crowds for natural language processing

• US 9,448,993 B1
• Filed: 09/07/2015
• Issued: 09/20/2016
• Est. Priority Date: 09/07/2015
• Status: Active Grant

• Current Assignee: Cerence Inc.
• Original Assignee: VoiceBox Technologies Corporation (Microsoft Corporation)
• Inventors: Braga, Daniela, Rothwell, Spencer John, Romani, Faraz, Elshenawy, Ahmad Khamis, Carter, Stephen Steele, Kennewick, Michael
• Primary Examiner(s): Albertalli, Brian
• Application Number:
  US14/846,926
• Time in Patent Office: 379 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/24 the user being prompted to utter a password or a predefined phrase

Abstract

A system and method of recording utterances for building Named Entity Recognition (“NER”) models, which are used to build dialog systems in which a computer listens and responds to human voice dialog. Utterances to be uttered may be provided to users through their mobile devices, which may record the user uttering (e.g., verbalizing, speaking, etc.) the utterances and upload the recording to a computer for processing. The use of the user'"'"'s mobile device, which is programmed with an utterance collection application (e.g., configured as a mobile app), facilitates the use of crowd-sourcing human intelligence tasking for widespread collection of utterances from a population of users. As such, obtaining large datasets for building NER models may be facilitated by the system and method disclosed herein.

Claims

1. A computer implemented method of recording utterances from unmanaged crowds for natural language processing, the method being implemented in an end user device having one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the end user device to perform the method, the method comprising:

obtaining, by the end user device, a token;

obtaining, by the end user device, one or more campaign configuration parameters based on the token;

configuring, by the end user device, the computer program instructions based on the one or more campaign configuration parameters;

obtaining, by the end user device, one or more utterances to be uttered by a user based on the token;

displaying, by the end user device, the one or more utterances to be uttered by the user;

generating, by the end user device, an audio recording of the one or more utterances; and

causing, by the end user device, the audio recording to be provided to a remote device via a network.

2. The method of claim 1, wherein the token comprises an alphanumeric string.

3. The method of claim 2, wherein obtaining the one or more campaign configuration parameters comprises parsing the alphanumeric string to obtain a campaign identifier that is associated with the one or more campaign configuration parameters.

4. The method of claim 3, wherein obtaining the one or more utterances to be uttered comprises parsing the alphanumeric string to obtain a session script identifier that is associated with the one or more utterances to be uttered.

5. The method of claim 3, wherein the one or more campaign configuration parameters comprises a configuration parameter that indicates that an audit check should be performed, the method further comprising:

configuring, by the end user device, the computer program instructions to determine whether a number of failed audits associated with the campaign identifier with which the end user device was involved exceeds a predetermined threshold number of failed audits; and

responsive to a determination that the number of failed audits with which the end user device was involved exceeds the predetermined threshold number of failed audits, preventing the end user device from participating in additional utterance recordings associated with the campaign identifier.

6. The method of claim 1, wherein the one or more campaign configuration parameters comprises a calibration parameter that indicates that a calibration test should be performed, the method further comprising:

configuring, by the end user device, the computer program instructions to perform a calibration test prior to generating the audio recording, by;

determining, by the end user device, a level of ambient noise;

determining, by the end user device, whether the level of ambient noise satisfies the calibration test, wherein the one or more utterances to be uttered are displayed responsive to a determination that the ambient level of audio satisfies the calibration test.

7. The method of claim 6, wherein the calibration parameter specifies a minimum level of ambient noise to satisfy the calibration test.

8. The method of claim 6, wherein the calibration parameter specifies a maximum level of ambient noise to satisfy the calibration test.

9. The method of claim 1, the method further comprising:

generating, by the end user device, a completion code responsive to a determination that a corresponding audio recording has been generated for each one of one or more utterances, wherein the completion code is configured to provide information that validates the one or more utterances have been uttered and recorded.

10. The method of claim 1, wherein causing the audio recording to be provided to a remote device via a network comprises:

immediately uploading, by the end user device, the audio recording to the remote device instead of through a batch process.

18. The system of claim 6, wherein the calibration parameter specifies a maximum level of ambient noise to satisfy the calibration test.

11. A system of recording utterances from unmanaged crowds for natural language processing, the system comprising:

an end user device having one or more physical processors programmed with computer program instructions that, when executed by the one or more physical processors, cause the end user device to;

obtain a token;

obtain one or more campaign configuration parameters based on the token;

configure the computer program instructions based on the one or more campaign configuration parameters;

obtain one or more utterances to be uttered by a user based on the token;

display the one or more utterances to be uttered by the user;

generate an audio recording of the one or more utterances; and

cause the audio recording to be provided to a remote device via a network.

12. The system of claim 11, wherein the token comprises an alphanumeric string.

13. The system of claim 12, wherein to obtain the one or more campaign configuration parameters, the end user device is further programmed to:

parse the alphanumeric string to obtain a campaign identifier that is associated with the one or more campaign configuration parameters.

14. The system of claim 13, wherein to obtain the one or more utterances to be uttered, the end user device is further programmed to:

parse the alphanumeric string to obtain a session script identifier that is associated with the one or more utterances to be uttered.

15. The system of claim 13, wherein the one or more campaign configuration parameters comprises a configuration parameter that indicates that an audit check should be performed, and wherein the end user device is further programmed to:

configure the computer program instructions to determine whether a number of failed audits associated with the campaign identifier with which the end user device was involved exceeds a predetermined threshold number of failed audits; and

responsive to a determination that the number of failed audits with which the end user device was involved exceeds the predetermined threshold number of failed audits, prevent the end user device from participating in additional utterance recordings associated with the campaign identifier.

16. The system of claim 11, wherein the one or more campaign configuration parameters comprises a calibration parameter that indicates that a calibration test should be performed, and wherein the end user device is further programmed to:

configure the computer program instructions to perform a calibration test prior to generating the audio recording, wherein to perform the calibration test, the end user device is further programmed to;

determine a level of ambient noise;

determine whether the level of ambient noise satisfies the calibration test, wherein the one or more utterances to be uttered are displayed responsive to a determination that the ambient level of audio satisfies the calibration test.

17. The system of claim 16, wherein the calibration parameter specifies a minimum level of ambient noise to satisfy the calibration test.

19. The system of claim 11, wherein the end user device is further programmed to:

generate a completion code responsive to a determination that a corresponding audio recording has been generated for each one of one or more utterances, wherein the completion code is configured to provide information that validates the one or more utterances have been uttered and recorded.

20. The system of claim 11, wherein to cause the audio recording to be provided to a remote device via a network, the end user device is further programmed to:

immediately upload the audio recording to the remote device instead of through a batch process.