Method and system for conveying an example in a natural language understanding application

• US 20070143099A1
• Filed: 12/15/2005
• Published: 06/21/2007
• Est. Priority Date: 12/15/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Lewis, James, Balchandran, Rajesh, Boyer, Linda, Metz, Brent
• Granted Patent: US 8,612,229 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 2015/0638 Interactive procedures
• CPC Class Codes: H04M 2203/2061 Language aspects
• CPC Class Codes: H04M 3/4936 Speech interaction details speech recognition per se G10L15/00

Abstract

A method (300) and system (100) is provided to add the creation of examples at a developer level in the generation of Natural Language Understanding (NLU) models, tying the examples into a NLU sentence database (130), automatically validating (310) a correct outcome of using the examples, and automatically resolving (316) problems the user has using the examples. The method (300) can convey examples of what a caller can say to a Natural Language Understanding (NLU) application. The method includes entering at least one example associated with an existing routing destination, and ensuring an NLU model correctly interprets the example unambiguously for correctly routing a call to the routing destination. The method can include presenting the example sentence in a help message (126) within an NLU dialogue as an example of what a caller can say for connecting the caller to a desired routing destination. The method can also include presented a failure dialogue for displaying at least one example that failed to be properly interpreted to ensure that ambiguous or incorrect examples are not presented in a help message.

Claims

1. A method for interacting with a natural language understanding system (NLU) comprising the steps of:

entering at least one example sentence during development of a natural language understanding (NLU) application; and

presenting said example sentence, wherein said example sentence is interpreted by an NLU model.

2. The method of claim 1, wherein said at least one example sentence is obtained from at least one of an application developer, an external source, sentences from a data collection, or an NLU system user.

3. The method of claim 1, further comprising:

presenting said at least one example sentence in a help message of said NLU application as example input to the NLU application.

4. The method of claim 1, further comprising presenting a failure dialog to a developer for displaying at least one example that failed to be correctly interpreted.

5. The method of claim 1, further comprising biasing the NLU model during application development, wherein the at least one example that failed to be correctly interpreted, is used to correct the association in the NLU model, and wherein when an example is interpreted correctly, it is used to strengthen the association in the NLU model.

6. The method of claim 1, wherein an NLU model forms a correct association between an example sentence and a routing destination during an NLU dialogue such that a caller speaking a sentence is connected to the correct corresponding routing destination.

7. The method of claim 3, wherein said help message comprises a preamble followed by at least one example sentence the caller can say, wherein the preamble introduces the at least one example sentence.

8. The method of claim 1, further comprising the steps of:

adding the example sentence to an NLU database; and

marking the example sentence within the NLU database to produce a marked example sentences for consideration priority.

9. The method of claim 8, wherein said step of ensuring includes building NLU models comprising the steps of:

training an NLU model using the NLU database that includes the marked example sentences;

testing the NLU model with the marked example sentences; and

biasing the NLU model to reinforce an association.

10. The method of claim 9, wherein said biasing is controlled to prevent the system from incorrectly interpreting sentences that were previously correctly identified before the biasing for producing a correctly interpreted example.

11. The method of claim 10, wherein more biasing can be applied to example sentences that fail by increasing a frequency of an example.

12. The method of claim 9, wherein said biasing stops at a threshold number of maximum iterations.

13. A system for creating a Natural Language Understanding (NLU) application comprising:

an example planner that prompts a developer for at least one example sentence; and

a validation unit connected to the example planner to increase a likelihood that an NLU model properly interprets said example sentence.

14. The system of claim 13, further comprising a help message presenting at least one example of speech.

15. The system of claim 13, further comprising a failure dialog for displaying at least one example that failed to be properly interpreted.

16. The system of claim 13, wherein the example planner adds the example sentence with a count as a marked example sentences into a NLU database that indicates a consideration priority during training.

17. The system of claim 16, wherein the validation unit further comprises

a training unit coupled to the NLU database for updating the statistical model, the NLU database including the marked example sentences;

a testing unit to assess performance of the statistical model receiving as input a batch of example sentences; and

a biasing unit for reinforcing an association by biasing the statistical model with the marked example sentences for producing at least one correctly interpreted example.

18. The system of claim 17, wherein the biasing unit resolves a discrepancy by increasing a frequency of an example during a biasing of the statistical model.

19. The system of claim 18, wherein said biasing is controlled to prevent the system from incorrectly interpreting sentences that were previously correctly identified before the biasing for producing a correctly interpreted example.

20. The system of claim 18, wherein said biasing stops when a threshold of corrected sentences to newly broken sentences is reached.

21. The system of claim 15, wherein the failure dialogue checks for at least one error which failed to be correctly interpreted, wherein a list of errors is presented in a ranked order of probability.