Generation of language understanding systems and methods

• US 10,460,720 B2
• Filed: 04/03/2015
• Issued: 10/29/2019
• Est. Priority Date: 01/03/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Williams, Jason Douglas, Niraula, Nobal Bikram, Dasigi, Pradeep, Lakshmiratan, Aparna, Zweig, Geoffrey G., Kolobov, Andrey, Garcia Jurado Suarez, Carlos, Chickering, David Maxwell
• Primary Examiner(s): Bost, Dwayne D
• Assistant Examiner(s): Brinich, Stephen M
• Application Number:
  US14/678,315
• Publication Number:
  US 20160196820A1
• Time in Patent Office: 1,670 Days
• Field of Search: 704 1- 10, 704270, 704250, 704255-257, 704266, 704E15007, 704E15008
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/07 to the speaker
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Domain-specific language understanding models that may be built, tested and improved quickly and efficiently are provided. Methods, systems and devices are provided that enable a developer to build user intent detection models, language entity extraction models, and language entity resolution models quickly and without specialized machine learning knowledge. These models may be built and implemented via single model systems that enable the models to be built in isolation or in an end-to-end pipeline system that enables the models to be built and improved in a simultaneous manner.

Claims

1. A method for providing language understanding on a computer, comprising:

receiving an utterance;

building, by the computer, two separate models in isolation from one another, the building of the two separate models comprising;

building a user intent detection model based on a user intent determined from the received utterance, the user intent detection model for determining a user intent in a subsequent utterance, wherein the user intent detection model is updateable via interactive learning to identify an unlabeled user intent associated with functionality within an associated application;

building an entity extraction model based on one or more language entities determined from the received utterance, the entity extraction model for determining one or more language entities in the subsequent utterance, wherein the entity extraction model is updateable via interactive learning to identify unlabeled language entities associated with functionality within the associated application;

automatically generating labeling for the user intent and the one or more language entities determined from the received utterance;

building a function call for calling a function in the associated application from the automatically generated labeling;

wherein in response to receiving the subsequent utterance;

the user intent detection model and the entity extraction model are applied against the subsequent utterance to identify, based on the automatically generated labeling for the subsequent utterance, the function call;

the function call is passed to the associated application to execute the function on the computer; and

the function is executed by the associated application on the computer.

2. The method of claim 1, further comprising storing the automatically generated labeling for subsequent use for labeling the user intent and the one or more language entities extracted from the subsequent utterance.

3. The method of claim 1, prior to building, by the computer, the two separate models in isolation from one another, determining the user intent from one or more words extracted from the received utterance.

4. The method of claim 3, prior to building, by the computer, the two separate models in isolation from one another, determining the one or more language entities from one or more words extracted from the received utterance.

5. The method of claim 4, further comprising converting the received utterance from speech to text prior to extracting one or more words from the utterance.

6. The method of claim 1, further comprising:

receiving a correction to the labeling automatically generated for one or both of the user intent and the one or more language entities determined from the subsequent utterance; and

updating, by the computer, one or both of the user intent detection model and the entity extraction model based on the received correction to the automatically generated labeling.

7. The method of claim 6, further comprising:

re-applying the user intent detection model and the entity extraction model against the subsequent utterance after updating, by the computer, one or both of the user intent detection model and the entity extraction model based on the received correction to the automatically generated labeling; and

automatically generating an updated labeling for the user intent and for the one or more language entities determined from the subsequent utterance.

8. The method of claim 6, further comprising:

processing the subsequent utterance against the user intent detection model and the entity extraction model after updating, by the computer, one or both of the user intent detection model and the entity extraction model based on the received correction to the automatically generated labeling.

9. The method of claim 1, further comprising prior to automatically generating the labeling for the one or more language entities determined from the subsequent utterance, resolving one or more of the one or more language entities to canonical identifications for one or more of the one or more language entities.

10. The method of claim 1, further comprising:

wherein a plurality of results are received from the associated application in response to the function call, ranking the plurality of results received from the associated application based on a correlation of the received results with the labeling for the user intent and for the one or more language entities used for building the function call.

11. The method of claim 1, wherein the received utterance is broken into one or more words contained therein based on pauses and common word patterns.

12. The method of claim 1, wherein a given language entity of the one or more language entities includes a span of one or more words into which the received utterance has been broken.

13. The method of claim 1, wherein a given language entity from the one or more language entities is resolved into a canonical form for a known entity.

14. A method for providing language understanding on a computer, comprising:

extracting, by the computer, from one or more unlabeled utterances a user intent and one or more language entities;

building, by the computer, two separate models in isolation from one another, the building of the two separate models comprising;

building a user intent detection model based on the user intent extracted from the one or more unlabeled utterances for determining user intents in subsequent utterances, wherein the user intent detection model is updateable via interactive learning to identify an unlabeled user intent associated with functionality within an associated application;

building an entity extraction model based on the one or more language entities extracted from the one or more unlabeled utterances for determining language entities in the subsequent utterances, wherein the entity extraction model is updateable via interactive learning to identify unlabeled language entities associated with functionality within the associated application;

testing a first of additional unlabeled utterances against the user intent detection model and the entity extraction model;

automatically generating labeling for a user intent and one or more language entities determined from the first of the additional unlabeled utterances upon application of the user intent detection model and the entity extraction model;

generating a function call associated with a desired function identified from the automatically generated labeling; and

passing the function call to an application for execution of the desired function and obtaining one or more results, wherein the desired function is executed by the application.

15. The method of claim 14, further comprising storing the automatically generated labeling for subsequent use for labeling a user intent and one or more language entities extracted from other unlabeled utterances.

16. The method of claim 14, further comprising:

receiving a correction to the automatically generated labeling for one or both of the user intent and the one or more language entities determined from the first of the additional unlabeled utterances; and

updating, by the computer, one or both of the user intent detection model and the entity extraction model based on the received correction to the automatically generated labeling.

17. The method of claim 16, further comprising:

re-testing the first of the additional unlabeled utterances against the user intent detection model and the entity extraction model after updating, by the computer, one or both of the user intent detection model and the entity extraction model based on the received correction to the automatically generated labeling; and

automatically generating an updated labeling for the user intent and for the one or more language entities determined from the first of the additional unlabeled utterances upon application of the user intent detection model and the entity extraction model for improving performance of the user intent detection model and the entity extraction model.

18. The method of claim 16, further comprising:

processing a second of the additional unlabeled utterances against the user intent detection model and the entity extraction model after updating, by the computer, one or both of the user intent detection model and the entity extraction model based on the received correction to the automatically generated labeling.

19. The method of claim 14, prior to extracting, by the computer, the user intent and the one or more language entities from the one or more unlabeled utterances, collecting, by the computer, the one or more unlabeled utterances and converting the one or more unlabeled utterances from speech to text from which one or more individual words are extracted for use in determining the user intent and the one or more language entities.

20. A language understanding system, comprising:

one or more processors; and

a memory coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the language understanding system to;

receive an utterance;

build two separate models in isolation from one another, the building of the two separate models comprising to;

build a user intent detection model based on a user intent determined from the received utterance, the user intent detection model for determining a user intent in a subsequent utterance, wherein the user intent detection model is updateable via interactive learning to identify an unlabeled user intent associated with functionality within an application; and

build an entity extraction model based on one or more language entities determined from the received utterance, the entity extraction model for determining one or more language entities in the subsequent utterance, wherein the entity extraction model is updateable via interactive learning to identify unlabeled language entities associated with functionality within the application;

wherein, in response to receiving the subsequent utterance, the user intent detection model and the entity extraction model are applied against the subsequent utterance for automatically generating labeling for the user intent and for the one or more language entities determined from the subsequent utterance,

wherein a function call associated with a desired function identified from the automatically generated labeling for the user intent and for the one or more language entities determined from the subsequent utterance is generated by the language understanding system, and

wherein the function call is passed by the language understanding system to the application running on a computer for execution of the desired function and obtaining one or more results; and

wherein the application running on the computer executes the desired function.