Tool for facilitating the development of new language understanding scenarios

• US 10,417,346 B2
• Filed: 01/23/2016
• Issued: 09/17/2019
• Est. Priority Date: 01/23/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kim, Young-Bum, Sarikaya, Ruhi, Rochette, Alexandre
• Primary Examiner(s): Saint Cyr, Leonard
• Application Number:
  US15/004,901
• Publication Number:
  US 20170212884A1
• Time in Patent Office: 1,333 Days
• Field of Search: 704 7- 10
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30

Abstract

A computer-implemented technique is described for facilitating the creation of a language understanding (LU) component for use with an application. The technique allows a developer to select a subset of parameters from a larger set of parameters. The subset of parameters pertains to a LU scenario to be handled by the application. The larger set of parameters pertains to a plurality of LU scenarios handled by an already-existing generic LU model. The technique creates a constrained LU component that is based on the subset of parameters in conjunction with the generic LU model. At runtime, the constrained LU component interprets input language items using the generic LU model in a manner that is constrained by the subset of parameters that have been selected, to provide an output result. The technique also allows the developer to create new rules and/or supplemental models.

Claims

1. A method implemented by one or more computing devices, the method comprising:

receiving, via an input device, input information specifying at least two different subsets of parameters selected from among a larger set of parameters of a preexisting language understanding (LU) model, the at least two different subsets of parameters being associated with different dialog states of a multi-turn dialog handled by an application, the different dialog states involving different prompts by the application to a user;

storing customization logic identifying the at least two different subsets of parameters in a data store; and

generating a constrained LU component by executing a development tool that applies the customization logic to the preexisting LU model, the customization logic causing the development tool to generate the constrained LU component by;

configuring the constrained LU component to, during a first dialog state at runtime of the application, filter output items generated by the preexisting LU model with respect to a first subset of parameters to provide first output items that conform to the first subset of parameters; and

configuring the constrained LU component to, during a second dialog state at runtime of the application, filter the output items generated by the preexisting LU model with respect to a second subset of parameters to provide second output items that conform to the second subset of parameters.

2. The method of claim 1, wherein the input information is received from a developer.

3. The method of claim 2, wherein said receiving the input information comprises:

receiving a first selection of a first prompt that is associated with the first dialog state of the multi-turn dialog; and

receiving a second selection of a second prompt that is associated with the second dialog state of the multi-turn dialog.

4. The method of claim 1, wherein the preexisting LU model is a statistical model that is trained based at least on a generic set of training examples to recognize various input language items associated with a plurality of LU scenarios.

5. The method of claim 1, wherein said receiving input information comprises:

receiving at least one example query via a user interface presentation;

using the preexisting LU model to identify one or more candidate parameters associated with said at least one example query;

presenting said one or more candidate parameters on the user interface presentation;

receiving a selection, via the user interface presentation, of at least one selected parameter from the one or more candidate parameters; and

including the at least one selected parameter in the first subset or the second subset.

6. The method of claim 1, wherein said receiving input information comprises:

providing a list of at least some of the larger set of parameters on a user interface presentation;

receiving a selection, via the user interface presentation, of one or more selected parameters from the list; and

including the one or more selected parameters in the first subset or the second subset.

7. The method of claim 1, wherein the preexisting LU model comprises a deep neural network configured to determine respective intents of linguistic items input to the application.

8. The method of claim 7, wherein the output items comprise the respective intents output by the deep neural network for the linguistic items, the first output items comprise first intents that conform to the first subset of parameters, and the second output items comprise second intents that conform to the second subset of parameters.

9. The method of claim 1, wherein the preexisting LU model comprises a Conditional Random Field tagging model configured to determine tags for words input to the application.

10. The method of claim 9, wherein the output items comprise the tags determined by the Conditional Random Field tagging model for the words, the first output items comprise first tags that conform to the first subset of parameters, and the second output items comprise second tags that conform to the second subset of parameters.

11. The method of claim 1, wherein the constrained LU component causes the application to take different actions at runtime during the multi-turn dialog based at least on the first output items and the second output items.

12. The method of claim 1, further comprising:

receiving rule input information that specifies a new rule to be used in conjunction with the constrained LU component;

creating the new rule based at least on the rule input information; and

including the new rule in the customization logic applied to generate the constrained LU component.

13. The method of claim 12, wherein said receiving of the rule input information involves receiving a selection of a specified state in the multi-turn dialog, or at least one characteristic of that state, to provide state information.

14. One or more computing devices comprising:

one or more hardware processor devices; and

one or more computer-readable storage media storing machine-readable instructions which, when executed by the one or more hardware processor devices, cause the one or more hardware processor devices to;

access a language understanding (LU) model having a plurality of classifiers configured to output parameters reflecting results of classification operations performed by the classifiers;

receive, via an input device, input information specifying a subset of specific parameters selected from among the parameters of the LU model, the subset of specific parameters pertaining to a language understanding scenario to be handled by an application;

store customization logic identifying the subset of specific parameters in a data store; and

execute a development tool that applies the customization logic to the LU model to generate a constrained LU component, the constrained LU component being configured to;

perform interpretation of input language items using a subset of classifiers of the LU model, the subset of classifiers excluding at least some classifiers of the LU model that do not output the specific parameters; and

cause the application to take certain actions based at least on the interpretation of the input language items.

15. The one or more computing devices of claim 14, wherein the machine-readable instructions, when executed by the one or more hardware processor devices, cause the one or more hardware processor devices to:

receive selections of a first subset of specific parameters corresponding to a first state in a multi-turn dialog performed by the application and a second subset of specific parameters corresponding to a second state in the multi-turn dialog performed by the application, the customization logic identifying the first subset and the second subset,

wherein executing the development tool configures the constrained LU component to, during the first state, exclude individual classifiers of the LU model that do not output the first subset of specific parameters, and, during the second state, exclude other individual classifiers of the LU model that do not output the second subset of specific parameters.

16. The one or more computing devices of claim 14, wherein the subset of classifiers includes binary classifiers.

17. The one or more computing devices of claim 14, wherein the machine-readable instructions, when executed by the one or more hardware processor devices, cause the one or more hardware processor devices to:

display a user interface presentation;

receive, via the user interface presentation, example linguistic items associated with the language understanding scenario to be handled by the application;

identify multiple suggested parameters of the LU model that are implicated by the example linguistic items; and

display the multiple suggested parameters in the user interface presentation; and

receive the input information specifying the subset of specific parameters via the user interface presentation, the input information identifying at least some of the multiple suggested parameters to include in the subset of specific parameters.

18. The one or more computing devices of claim 14, wherein the machine-readable instructions, when executed by the one or more hardware processor devices, cause the one or more hardware processor devices to:

execute a runtime environment that uses the constrained LU component to perform the interpretation of input language items using the subset of classifiers; and

within the runtime environment, execute a query based at least on the interpretation of the input language items.

19. The one or more computing devices of claim 18, wherein the machine-readable instructions, when executed by the one or more hardware processor devices, cause the one or more hardware processor devices to:

within the runtime environment, execute the query using at least one intent value and at least one slot value provided by the constrained LU component to obtain a query answer; and

within the runtime environment, provide the query answer to the application, the application taking an individual action based at least on the query answer.

20. A system comprising:

one or more processor devices; and

a computer-readable storage medium storing computer-readable instructions which, when executed by the one or more processor devices, cause the one or more processor devices to;

identify a subset of specific parameters that have been selected, via an input device, from among a larger set of parameters of a preexisting language understanding (LU) model, the subset of specific parameters pertaining to an LU scenario to be handled by an application, and the larger set of parameters pertaining to a plurality of LU scenarios handled by the preexisting LU model, the preexisting LU model having a plurality of nodes representing the parameters of the larger set and corresponding state transitions among the nodes;

store customization logic identifying the subset of specific parameters; and

generate a constrained LU component by executing a development tool that applies the customization logic to the preexisting LU model, the customization logic causing the development tool to generate the constrained LU component by;

deactivating at least some state transitions of the preexisting LU model that occur between individual nodes that do not have corresponding specific parameters in the subset; and

retaining at least some other state transitions of the preexisting LU model in the constrained LU component,

the constrained LU component being configured to interpret language items input to an application to obtain corresponding results that cause the application to take specific actions in response to receiving the results.

21. The system of claim 20, wherein the nodes represent specific tags assigned to words by the preexisting LU model.

22. The system of claim 20, wherein the computer-readable instructions which when executed by the one or more processor devices, cause the one or more processor devices to:

after the at least some state transitions have been deactivated, interpret the language items by finding paths through remaining state transitions present in the constrained LU component.

23. The system of claim 22, the paths being lowest-cost paths available through the remaining state transitions, the lowest-cost paths having corresponding nodes with associated tags for the language items.

24. A method performed using at least one computing device, the method comprising:

accessing a language understanding (LU) model having a plurality of classifiers configured to output parameters reflecting results of classification operations performed by the classifiers;

receiving, via an input device, input information specifying a subset of specific parameters selected from among the parameters of the LU model, the subset of specific parameters pertaining to a language understanding scenario to be handled by an application;

storing customization logic identifying the subset of specific parameters in a data store;

executing a development tool that applies the customization logic in the data store to the LU model to generate a constrained LU component;

executing the application in a runtime environment with the constrained LU component;

within the runtime environment, performing interpretation of input language items with the constrained LU component, the constrained LU component performing the interpretation using a subset of classifiers of the LU model that excludes at least some classifiers of the LU model that do not output the specific parameters; and

providing at least one answer to the application within the runtime environment, based at least on the interpretation of the input language items by the constrained LU component.