Applying a structured language model to information extraction

• US 20030216905A1
• Filed: 05/20/2002
• Published: 11/20/2003
• Est. Priority Date: 05/20/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Chelba, Ciprian, Mahajan, Milind
• Granted Patent: US 7,805,302 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/00 Handling natural language data speech analysis or synthesis, speech recognition G10L
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/237 Lexical tools
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06F 40/56 Natural language generation
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/04 Segmentation; Word boundary detection
• CPC Class Codes: G10L 15/05 Word boundary detection
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

One feature of the present invention uses the parsing capabilities of a structured language model in the information extraction process. During training, the structured language model is first initialized with syntactically annotated training data. The model is then trained by generating parses on semantically annotated training data enforcing annotated constituent boundaries. The syntactic labels in the parse trees generated by the parser are then replaced with joint syntactic and semantic labels. The model is then trained by generating parses on the semantically annotated training data enforcing the semantic tags or labels found in the training data. The trained model can then be used to extract information from test data using the parses generated by the model.

Claims

1. A method of training an information extraction system to extract information from a natural language input, comprising:

generating parses with a structured language model using annotated training data that has semantic constituent labels with semantic constituent boundaries identified;

while generating parses, constraining parses to match the semantic constituent boundaries; and

while generating parses, constraining the parses to match the semantic constituent labels.

2. The method of claim 1 and further comprising:

prior to generating parses, initializing the structured language model with syntactically annotated training data.

3. The method of claim 2 wherein initializing comprises:

initializing the structured language model with syntactically annotated training data parsed from in-domain sentences.

4. The method of claim 2 wherein initializing comprises:

initializing the structured language model with syntactically annotated training data parsed from out-of-domain sentences.

5. The method of claim 1 wherein generating parses comprises:

generating syntactic parses with syntactic labels wherein the parses conform to the semantic constituent boundaries;

enriching the syntactic labels with semantic labels; and

generating semantic parses with semantic labels wherein the semantic labels conform to the semantic constituent labels in the annotated training data.

6. The method of claim 1 wherein generating parses comprises:

generating the parses as binary parse trees.

7. The method of claim 1 wherein generating parses comprises:

generating the parses in a left-to-right fashion.

8. The method of claim 1 wherein generating parses comprises:

generating parses in a bottom-up fashion.

9. A method of extracting information from a natural language input, comprising:

parsing the natural language input with a structured language model to obtain a parse having a semantic frame label and one or more constituents of the natural language input each having a semantic slot label; and

identifying an information extraction frame corresponding to the natural language input based on the frame label and filling in slots in the frame with the one or more constituents labeled by the slot labels.

10. The method of claim 9 wherein parsing comprises:

accessing a semantic schema corresponding to an application program for which information is being extracted; and

constraining parses based on the semantic schema.

11. The method of claim 10 wherein constraining parses comprises:

constraining parses to a structure of frames in the semantic schema.

12. The method of claim 9 wherein identifying comprises:

generating a probability that generated parses occur given a word sequence;

selecting a parse generated during parsing that has a highest probability of occurring; and

retaining only the semantic information in the highest probability parse.

13. The method of claim 9 wherein identifying comprises:

generating a probability that generated parses occur given a word sequence;

summing the probability over all parses having a common semantic parse; and

selecting the semantic parse based on the summed probability.

14. An information extraction system for extracting information from a natural language speech input, comprising:

a speech recognizer, including a structured language model, receiving the natural language speech input and generating a textual representation of the natural language speech input based on language modeling by the structured language model, the structured language model parsing the textual representation to obtain a parse having a semantic frame label and one or more semantic slot labels corresponding to constituents of the textual representation, the semantic frame and slot labels identifying the information to be extracted.

15. The system of claim 14 and further comprising:

a ranking component ranking parses generated by the structured language model.

16. The system of claim 15 wherein the ranking component ranks each semantic parse by summing over all generated parses having the same semantic parse.

17. A method of extracting information from a natural language (NL) input, comprising:

accessing a schema associated with an application program for which the information is extracted, the schema having frames with a frame structure; and

parsing the NL input to obtain a parse having a semantic frame label and one or more semantic slot labels corresponding to constituents of the NL input, the parse being constrained to the frame structure in the schema.

18. The method of claim 17 wherein the NL input is a speech input and further comprising:

performing speech recognition on the speech input for parsing.

19. The method of claim 18 wherein performing speech recognition comprises:

recognizing speech based at least in part on language modeling by a structured language model.

20. The method of claim 19 wherein parsing comprises:

parsing the NL input with the structured language model.

21. The method of claim 17 wherein parsing comprises:

identifying a frame corresponding to the NL input based on the frame label.

22. The method of claim 21 wherein parsing comprises:

filling semantic slots in the identified frame with constituents of the NL input based on the semantic slot labels.