Integrated development tool for building a natural language understanding application

• US 7,191,119 B2
• Filed: 05/07/2002
• Issued: 03/13/2007
• Est. Priority Date: 05/07/2002
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Epstein, Mark E., Jones, Sharon B., Ward, Robert T.
• Primary Examiner(s): Dorvil; Richemond
• Assistant Examiner(s): SAINT CYR, LEONARD
• Application Number:
  US10/140,522
• Publication Number:
  US 20030212543A1
• Time in Patent Office: 1,771 Days
• Field of Search: 704/1, 704/9, 704/10
• US Class Current: 704/10
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

A method of developing natural language understanding (NLU) applications can include determining NLU interpretation information from an NLU training corpus of text using a multi-pass processing technique. The alteration of one pass automatically can alter an input for a subsequent pass. The NLU interpretation information can specify an interpretation of at least part of the NLU training corpus of text. The NLU interpretation information can be stored in a database, and selected items of the NLU interpretation information can be presented in a graphical editor. User specified edits also can be received in the graphical editor.

Claims

1. A method of developing natural language understanding (NLU) applications comprising:

determining NLU interpretation information from an NLU training corpus of text using a multi-pass processing technique, wherein alteration of one pass automatically alters an input for a subsequent pass, said NLU interpretation information specifying an interpretation of at least part of said NLU training corpus of text;

storing said NLU interpretation information in a database;

in a graphical editor, presenting selected items of NLU interpretation information as a meaning tree comprising terminal and non-terminal nodes representative of data items and receiving user specified edits to said NLU interpretation information;

determining an intersection of selected nodes of said meaning tree from said NLU interpretation information;

presenting said intersection of selected nodes as choices for adding an additional node to said meaning tree; and

adding a node above said selected nodes of said meaning tree, wherein said added node is selected from said choices.

2. The method of claim 1, further comprising:

determining a probability indicating whether a portion of said meaning tree is correct; and

visually identifying said portion of said meaning tree if said probability does not exceed a predetermined threshold probability.

3. The method of claim 1, further comprising:

determining a number of occurrences of a substructure of said meaning tree within said NLU interpretation information; and

visually identifying said substructure of said meaning tree if said number of occurrences does not exceed a predetermined threshold.

4. The method of claim 1, further comprising:

responsive to a user command, creating an additional node of said meaning tree, said node representing an additional data item; and

associating said node with a description.

5. The method of claim 4, further comprising:

spell checking said associated description.

6. The method of claim 1, wherein a node of said meaning tree is selected, said method further comprising:

responsive to a user request, displaying a dictionary view comprising a plurality of columns for displaying said node and parameters of said node, wherein said dictionary view includes a data item represented by said node.

7. The method of claim 1, further comprising:

automatically completing said meaning tree according to predetermined NLU interpretation information selected from the group consisting of a dictionary of data items, and a model specifying text interpretations.

8. The method of claim 7, said automatically completing step comprising:

determining whether a single data item from said dictionary of data items is associated with a word of said NLU training corpus of text; and

if so, automatically assigning said data item to said word.

9. The method of claim 1, further comprising:

displaying selected items of said NLU interpretation information as a tool tip; and

displaying a probability that said presented meaning tree is a correct interpretation.

10. The method of claim 1, further comprising:

searching said NLU interpretation information for a specified meaning tree structure.

11. The method of claim 1, said presenting step further comprising:

presenting said NLU interpretation information in a dictionary view comprising a plurality of columns for displaying parent and child data items, and parameters of said data items.

12. The method of claim 11, wherein said dictionary view includes a column for indicating children of said data items and another column for indicating parents of said data items.

13. The method of claim 12, further comprising:

sorting said data items according to said child or parent columns.

14. The method of claim 11, further comprising:

visually indicating data items having a probability or count exceeding a predetermined threshold.

15. The method of claim 11, further comprising:

hiding data items having a probability or count not exceeding a predetermined threshold.

16. The method of claim 11, further comprising:

responsive to a user selection of particular data items having an association, searching said NLU interpretation information for a meaning tree comprising terminal and non-terminal nodes representing said association.

17. The method of claim 11, further comprising:

filtering said NLU interpretation information according to a parameter selected from the group consisting of a data item source, a data item target, a direction associated with a data item, an annotator associated with a data item, annotation status, a node count, a data file, a sentence range, and a usage status.

18. The method of claim 11, further comprising:

displaying histogram information derived from said NLU interpretation information.

19. The method of claim 1, said presenting step further comprising:

in a sentence view, presenting said NLU interpretation information associated with individual text phrases, said NLU interpretation information selected from the group consisting of an annotation status, a phrase count, a designated phrase use, phrase collection information, a correctness probability, and a correctness rank.

20. The method of claim 1, said presenting step further comprising:

presenting said NLU interpretation information as a plurality of meaning trees in a split screen view, said split screen view comprising at least a first window for displaying a first meaning tree and a second window for displaying a second meaning tree.

21. The method of claim 20, further comprising:

while displaying said first meaning tree in said first window, presenting different ones of said meaning trees in said second window responsive to a user request.

22. The method of claim 20, wherein said first and second meaning trees are different interpretations of a same text phrase in a same context.

23. The method of claim 22, wherein said first meaning tree represents a correct interpretation of a text phrase, and said second meaning tree represents a predicted interpretation of said text phrase according to a statistical model.

25. The method of claim 20, wherein said first and second meaning trees are different interpretations of a same text phrase in two different contexts.

26. The method of claim 20, wherein said first meaning tree corresponds to a text phrase after a first processing pass, and said second meaning tree corresponds to said text phrase after a subsequent processing pass.

27. The method of claim 20, wherein said first meaning tree represents a first text phrase, said method further comprising:

receiving an edit to said first meaning tree in said first window; and

responsive to said edit, searching for a meaning tree of a different text phrase which corresponds to said edited first meaning tree, and displaying said meaning tree for said different text phrase in said second window.

28. The method of claim 20, further comprising:

visually indicating differences between said first meaning tree and said second meaning tree.

29. The method of claim 20, further comprising:

conforming said first meaning tree to said second meaning tree.

30. The method of claim 1, further comprising:

automatically importing NLU training sentences; and

automatically determining an interpretation of said NLU training sentences according to statistical likelihoods determined from other NLU training corpus of text.

31. The method of claim 1, further comprising:

automatically importing NLU interpretations of training sentences; and

applying said NLU interpretations to said NLU training corpus of text.

32. The method of claim 1, further comprising:

displaying said NLU interpretation information resulting from one of said multi-passes responsive to a user input specifying said one of said passes.

33. The method of claim 1, further comprising:

ordering text phrases of said NLU training corpus of text in a sentence view according to an attribute of NLU interpretation information for said text phrases; and

sequentially displaying said NLU interpretation information for at least two of said text phrases as meaning trees, wherein said meaning trees are displayed according to said ordering of said sentence view.

24. The method of 23, further comprising:

if a score of said correct interpretation is greater than a score of said predicted interpretation, indicating that said statistical model is incorrect.

34. A method of developing natural language understanding (NLU) applications comprising:

determining NLU interpretation information from an NLU training corpus of text using a multi-pass processing technique, wherein alteration of one pass automatically alters an input for a subsequent pass, said NLU interpretation information specifying an interpretation of at least part of said NLU training corpus of text;

storing said NLU interpretation information in a database;

in a graphical editor, presenting selected items of NLU interpretation information as a meaning tree comprised of terminal and non-terminal nodes representing data items and receiving user specified edits to said NLU interpretation information;

identifying an intersection of data items; and

presenting said identified data items as selections for annotating a user specified node of said meaning tree for a sentence of said NLU training corpus of text.

35. An integrated development tool for developing a natural language understanding (NLU) application, said integrated development tool comprising:

a database configured to store items of NLU interpretation information corresponding to an NLU training corpus of text, wherein said items of NLU interpretation information specify an interpretation of at least part of said NLU training corpus of text;

a graphical editor having a plurality of views for manipulating said items of NLU interpretation information, said graphical editor being communicatively linked to said database, wherein a first one of said plurality of views comprises selected items of NLU interpretation information presented as a meaning tree comprised of terminal and non-terminal nodes representing data items, wherein a second one of said plurality of views comprises presenting an intersection of selected nodes of said meaning tree as choices for adding an additional node to said meaning tree; and

a processor configured to determine said items of NLU interpretation information according to a multi-pass system, and to store said items of NLU interpretation information in said database, wherein alteration of one pass automatically alters an input for a subsequent pass, wherein said processor being further configured to determine the intersection of selected nodes of said meaning tree from said items of NLU interpretation information, and to add a node above said selected nodes to said meaning tree, wherein said added nodes are selected from said choices.

36. The integrated development tool of claim 35, further comprising:

a graphical user interface for specifying a search for selected ones of said data items of said NLU interpretation information, said graphical user interface having at least one list of selectable terminals and non-terminals for specifying said search.

37. The integrated development tool of the claim 36, wherein said graphical user interface includes at least one additional area for specifying relationships of said selected data items with other ones of said data items.

38. The integrated development tool of claim 37, further comprising:

a programmed statistical model configured to determine at least one interpretation from said NLU training corpus of text.

39. The integrated development tool of claim 38, said graphical editor having a tree view for presenting said items of NLU interpretation information in a hierarchical tree structure, a dictionary view for presenting individual ones of said items of NLU interpretation information in column format, a sentence view for presenting said items of NLU interpretation information in sentence form, a split screen view for simultaneously presenting two meaning trees, and an import view for importing additional NLU training text and associated interpretation information.

40. A machine-readable storage, having stored thereon a computer program having a plurality of code sections executable by a machine for causing the machine to perform the steps of:

determining NLU interpretation information from an NLU training corpus of text using a multi-pass processing technique, wherein alteration of one pass automatically alters an input for a subsequent pass, said NLU interpretation information specifying an interpretation of at least part of said NLU training corpus of text;

storing said NLU interpretation information in a database;

in a graphical editor, presenting selected items of NLU interpretation information as a meaning tree comprised of terminal and non-terminal nodes representing data items and receiving user specified edits to said NLU interpretation information;

determining an intersection of selected nodes of said meaning tree from said NLU interpretation information;

presenting said intersection of selected nodes as choices for adding an additional node to said meaning tree; and

adding a node above said selected nodes of said meaning tree, wherein said added node is selected from said choices.

41. The machine-readable storage of claim 40, further comprising:

determining a probability indicating whether a portion of said meaning tree is correct; and

visually identifying said portion of said meaning tree if said probability does not exceed a predetermined threshold probability.

42. The machine-readable storage of claim 40, further comprising:

determining a number of occurrences of a substructure of said meaning tree within said NLU interpretation information; and

visually identifying said substructure of said meaning tree if said number of occurrences does not exceed a predetermined threshold.

43. The machine-readable storage of claim 40, further comprising:

responsive to a user command, creating an additional node of said meaning tree, said node representing an additional data item; and

associating said node with a description.

44. The machine-readable storage of claim 43, further comprising:

spell checking said associated description.

45. The machine-readable storage of claim 40, wherein a node of said meaning tree is selected, said machine-readable storage further comprising:

responsive to a user request, displaying a dictionary view comprising a plurality of columns for displaying said node and parameters of said node, wherein said dictionary view includes a data item represented by said node.

46. The machine-readable storage of claim 40, further comprising:

automatically completing said meaning tree according to predetermined NLU interpretation information selected from the group consisting of a dictionary of data items, and a model specifying text interpretations.

47. The machine-readable storage of claim 46, said automatically completing step comprising:

determining whether a single data item from said dictionary of data items is associated with a word of said NLU training corpus of text; and

if so, automatically assigning said data item to said word.

48. The machine-readable storage of claim 40, further comprising:

displaying selected items of said NLU interpretation information as a tool tip; and

displaying a probability that said presented meaning tree is a correct interpretation.

49. The machine-readable storage of claim 40, further comprising:

searching said NLU interpretation information for a specified meaning tree structure.

50. The machine-readable storage of claim 40, said presenting step further comprising:

presenting said NLU interpretation information in a dictionary view comprising a plurality of columns for displaying parent and child data items, and parameters of said data items.

51. The machine-readable storage of claim 50, wherein said dictionary view includes a column for indicating children of said data items and another column for indicating parents of said data items.

52. The machine-readable storage of claim 51, further comprising:

sorting said data items according to said child or parent columns.

53. The machine-readable storage of claim 50, further comprising:

visually indicating data items having a probability or count exceeding a predetermined threshold.

54. The machine-readable storage of claim 50, further comprising:

hiding data items having a probability or count not exceeding a predetermined threshold.

55. The machine-readable storage of claim 50, further comprising:

responsive to a user selection of particular data items having an association, searching said NLU interpretation information for a meaning tree comprising terminal and non-terminal nodes representing said association.

56. The machine-readable storage of claim 50, further comprising:

filtering said NLU interpretation information according to a parameter selected from the group consisting of a data item source, a data item target, a direction associated with a data item, an annotator associated with a data item, annotation status, a node count, a data file, a sentence range, and a usage status.

57. The machine-readable storage of claim 50, further comprising:

displaying histogram information derived from said NLU interpretation information.

58. The machine-readable storage of claim 40, said presenting step further comprising:

in a sentence view, presenting said NLU interpretation information associated with individual text phrases, said NLU interpretation information selected from the group consisting of an annotation status, a phrase count, a designated phrase use, phrase collection information, a correctness probability, and a correctness rank.

59. The machine-readable storage of claim 40, said presenting step further comprising:

presenting said NLU interpretation information as a plurality of meaning trees in a split screen view, said split screen view comprising at least a first window for displaying a first meaning tree and a second window for displaying a second meaning tree.

60. The machine-readable storage of claim 59, further comprising:

while displaying said first meaning tree in said first window, presenting different ones of said meaning trees in said second window responsive to a user request.

61. The machine-readable storage of claim 59, wherein said first and second meaning trees are different interpretations of a same text phrase in a same context.

62. The machine-readable storage of claim 61, wherein said first meaning tree represents a correct interpretation of a text phrase, and said second meaning tree represents a predicted interpretation of said text phrase according to a statistical model.

64. The machine-readable storage of claim 59, wherein said first and second meaning trees are different interpretations of a same text phrase in two different contexts.

65. The machine-readable storage of claim 59, wherein said first meaning tree corresponds to a text phrase after a first processing pass, and said second meaning tree corresponds to said text phrase after a subsequent processing pass.

66. The machine-readable storage of claim 59, wherein said first meaning tree represents a first text phrase, said machine-readable storage further comprising:

receiving an edit to said first meaning tree in said first window; and

responsive to said edit, searching for a meaning tree of a different text phrase which corresponds to said edited first meaning tree, and displaying said meaning tree for said different text phrase in said second window.

67. The machine-readable storage of claim 59, further comprising:

visually indicating differences between said first meaning tree and said second meaning tree.

68. The machine-readable storage of claim 59, further comprising:

conforming said first meaning tree to said second meaning tree.

69. The machine-readable storage of claim 40, further comprising:

automatically importing NLU training sentences; and

automatically determining an interpretation of said NLU training sentences according to statistical likelihoods determined from other NLU training corpus of text.

70. The machine-readable storage of claim 40, further comprising:

automatically importing NLU interpretations of training sentences; and

applying said NLU interpretations to said NLU training corpus of text.

71. The machine-readable storage of claim 40, further comprising:

displaying said NLU interpretation information resulting from one of said multi-passes responsive to a user input specifying said one of said passes.

72. The machine-readable storage of claim 40, further comprising:

ordering text phrases of said NLU training corpus of text in a sentence view according to an attribute of NLU interpretation information for said text phrases; and

sequentially displaying said NLU interpretation information for at least two of said text phrases as meaning trees, wherein said meaning trees are displayed according to said ordering of said sentence view.

63. The machine-readable storage of 62, further comprising:

if a score of said correct interpretation is greater than a score of said predicted interpretation, indicating that said statistical model is incorrect.

73. A machine-readable storage having stored thereon a computer program having a plurality of code sections executable by a machine for causing the machine to perform the steps of:

determining NLU interpretation information from an NLU training corpus of text using a multi-pass processing technique, wherein alteration of one pass automatically alters an input for a subsequent pass, said NLU interpretation information specifying an interpretation of at least part of said NLU training corpus of text;

storing said NLU interpretation information in a database;

in a graphical editor, presenting selected items of NLU interpretation information as a meaning tree comprised of terminal and non-terminal nodes representing data items and receiving user specified edits to said NLU interpretation information;

identifying an intersection of data items; and

presenting said identified data items as selections for annotating a user specified node of said meaning tree for a sentence of said NLU training corpus of text.