Updating language understanding classifier models for a digital personal assistant based on crowd-sourcing

• US 9,508,339 B2
• Filed: 01/30/2015
• Issued: 11/29/2016
• Est. Priority Date: 01/30/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kannan, Vishwac Sena, Uzelac, Aleksandar, Hwang, Daniel J.
• Primary Examiner(s): PULLIAS, JESSE SCOTT
• Application Number:
  US14/611,042
• Publication Number:
  US 20160225370A1
• Time in Patent Office: 669 Days
• Field of Search: 704/1, 704/9, 704/10, 704/257, 704270-275
• US Class Current: 1/1
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/0636 Threshold criteria for the updating
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

A method for updating language understanding classifier models includes receiving via one or more microphones of a computing device, a digital voice input from a user of the computing device. Natural language processing using the digital voice input is used to determine a user voice request. Upon determining the user voice request does not match at least one of a plurality of pre-defined voice commands in a schema definition of a digital personal assistant, a GUI of an end-user labeling tool is used to receive a user selection of at least one of the following: at least one intent of a plurality of available intents and/or at least one slot for the at least one intent. A labeled data set is generated by pairing the user voice request and the user selection, and is used to update a language understanding classifier.

Claims

1. A server computer, comprising:

a processing unit; and

memory coupled to the processing unit;

the server computer configured to perform operations for updating language understanding classifier models, the operations comprising;

receiving from at least one computing device of a plurality of computing devices communicatively coupled to the server computer, a first user selection of at least one of the following;

at least one intent of a plurality of available intents and/or at least one slot for the at least one intent, wherein;

the at least one intent is associated with at least one action used to perform at least one function of a category of functions for a domain;

the at least one slot indicating a value used for performing the at least one action; and

the first user selection associated with a digital voice input received at the at least one computing device; and

upon receiving from at least another computing device of the plurality of computing devices, a plurality of subsequent user selections that are identical to the first user selection and a plurality of subsequent digital voice inputs corresponding to the plurality of subsequent user selections, wherein the plurality of subsequent digital voice inputs are substantially similar to the digital voice input;

generating a labeled data set by pairing the digital voice input with the first user selection;

selecting a language understanding classifier from a plurality of available language understanding classifiers associated with one or more agent definitions, the selecting based at least on the at least one intent; and

updating the selected language understanding classifier based on the generated labeled data set.

2. The server computer according to claim 1, the operations further comprising:

determining a number of the plurality of subsequent user selections; and

when the number of the plurality of subsequent user selections is higher than a first threshold, automatically updating the selected language understanding classifier based on the generated labeled data set.

3. The server computer according to claim 1, the operations further comprising:

receiving the digital voice input from the at least one computing device;

performing natural language processing using the digital voice input to determine a user voice request; and

storing one or both of the digital voice input and the user voice request in an utterances database.

4. The server computer according to claim 3, the operations further comprising:

retrieving one or both of the digital voice input and the user voice request from the utterances database;

generating the labeled data set by pairing the first user selection with one or both of the digital voice input and the user voice request; and

storing the generated labeled data set in a labeled data database.

5. The server computer according to claim 1, the operations further comprising:

determining a number of the plurality of subsequent user selections which comprise at least one intent and at least one slot that are different from the at least one intent and the at least one slot of the first user selection.

6. The server computer according to claim 5, the operations further comprising:

when the determined number of the plurality of subsequent user selections is higher than a second threshold, generating a request for manual updating of the selected language understanding classifier by an administrator of the server computer.

7. The server computer according to claim 6, the operations further comprising:

in response to the request for manual updating, receiving input selecting the at least one intent and the at least one slot of the first user selection or the at least one intent and the at least one slot of the plurality of subsequent user selections.

8. The server computer according to claim 7, the operations further comprising:

receiving input updating the selected language understanding classifier based on the selected at least one intent and the at least one slot.

9. A method for updating language understanding classifier models, the method comprising:

receiving via one or more microphones of a computing device, a digital voice input from a user of the computing device;

performing natural language processing using the digital voice input to determine a user voice request;

upon determining the user voice request does not match at least one of a plurality of pre-defined tasks in an agent definition of a digital personal assistant running on the computing device;

receiving using a graphical user interface of an end-user labeling tool (EULT) of the computing device, a user selection of at least one of the following;

an intent of a plurality of available intents and at least one slot for the intent, wherein;

the intent is associated with at least one action used to perform at least one function of a category of functions for a domain; and

the at least one slot indicating a value used for performing the at least one action;

generating a labeled data set by pairing the user voice request and the user selection;

selecting a language understanding classifier from a plurality of available language understanding classifiers associated with the agent definition, the selecting based at least on the intent selected by the user; and

updating the selected language understanding classifier based on the generated labeled data set.

10. The method according to claim 9, wherein the plurality of available language understanding classifiers associated with the agent definition are stored in local storage at the computing device and the method further comprises:

associating the updated language understanding classifier with a profile of the user within the computing device; and

storing the updated language understanding classifier in the local storage.

11. The method according to claim 10, further comprising:

designating the updated language understanding classifier as a common resource that can be shared between the digital personal assistant and at least one third-party application running on the computing device.

12. The method according to claim 9, wherein the updating comprises:

replacing an association of the selected language understanding classifier with at least one of a previous intent and/or slot with a new association with at least one of the intent and/or the at least one slot of the user selection.

13. The method according to claim 12, further comprising:

creating an association of the selected language understanding classifier with one or both of the digital voice input and the user voice request.

14. The method according to claim 9, wherein the user selection comprises the intent, and the method further comprises:

selecting the at least one slot automatically, based on the intent and the user voice request.

15. The method according to claim 9, wherein the agent definition comprises at least one of a voice command definition (VCD) schema and a reactive agent definition (RAD) schema associated with the digital personal assistant.

16. A computer-readable storage medium storing computer-executable instructions for causing a computing device to perform operations for updating language understanding classifier models, the operations comprising:

determining a user request based on user input received at a computing device, the user request received via at least one of text input and voice input, the request for a functionality of a digital personal assistant running on the computing device;

determining the user request does not match at least one of a plurality of pre-defined voice commands in an agent definition of the digital personal assistant;

generating a confidence score by applying a plurality of available language understanding classifiers associated with the agent definition to the user request;

upon determining that the confidence score is less than a threshold value;

receiving using a graphical user interface of an end-user labeling tool (EULT) of the computing device, a user selection of at least one of the following;

at least one intent of a plurality of available intents and at least one slot for the at least one intent, wherein;

the at least one intent is associated with at least one action used to perform at least one function of a category of functions for a domain; and

the at least one slot indicating a value used for performing the at least one action;

generating a labeled data set by pairing the user voice request and the user selection;

selecting a language understanding classifier from the plurality of available language understanding classifiers associated with the agent definition, the selecting based at least on the at least one intent selected by the user; and

generating an updated language understanding classifier by training the selected language understanding classifier using the generated labeled data set.

17. The computer-readable storage medium according to claim 16, the operations further comprising:

receiving using the graphical user interface of the end-user labeling tool (EULT) of the computing device, a selection of the domain associated with the at least one intent.

18. The computer-readable storage medium according to claim 16, the operations further comprising:

during the training, associating the selected language understanding classifier with the user request, the at least one intent and the at least one slot of the user selection.

19. The computer-readable storage medium according to claim 16, wherein the agent definition comprises at least one of a voice command definition (VCD) schema and a reactive agent definition (RAD) schema associated with the digital personal assistant.