Integrated development tool for building a natural language understanding application

• US 20030212543A1
• Filed: 05/07/2002
• Published: 11/13/2003
• Est. Priority Date: 05/07/2002
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Jones, Sharon B., Ward, Robert T., Epstein, Mark E.
• Granted Patent: US 7,191,119 B2
• Time in Patent Office: Days
• US Class Current: 704/9
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

A method of developing natural language understanding (NLU) applications can include determining NLU interpretation information from an NLU training corpus of text using a multi-pass processing technique. The alteration of one pass automatically can alter an input for a subsequent pass. The NLU interpretation information can specify an interpretation of at least part of the NLU training corpus of text. The NLU interpretation information can be stored in a database, and selected items of the NLU interpretation information can be presented in a graphical editor. User specified edits also can be received in the graphical editor.

Claims

1. A method of developing natural language understanding (NLU) applications comprising:

determining NLU interpretation information from an NLU training corpus of text using a multi-pass processing technique, wherein alteration of one pass automatically alters an input for a subsequent pass, said NLU interpretation information specifying an interpretation of at least part of said NLU training corpus of text;

storing said NLU interpretation information in a database; and

in a graphical editor, presenting selected items of NLU interpretation information and receiving user specified edits to said NLU interpretation information.

2. The method of claim 1, said presenting step further comprising:

presenting said NLU interpretation information as a meaning tree comprised of terminal and non-terminal nodes representing data items.

3. The method of claim 2, further comprising:

determining a probability indicating whether a portion of said meaning tree is correct; and

visually identifying said portion of said meaning tree if said probability does not exceed a predetermined threshold probability.

4. The method of claim 2, further comprising:

determining a number of occurrences of a substructure of said meaning tree within said NLU interpretation information; and

visually identifying said substructure of said meaning tree if said number of occurrences does not exceed a predetermined threshold.

5. The method of claim 2, further comprising:

determining an intersection of selected nodes of said meaning tree from said NLU interpretation information;

presenting said intersection of selected nodes as choices for adding an additional node to said meaning tree; and

adding a node above said selected nodes of said meaning tree, wherein said added node is selected from said choices.

6. The method of claim 2, further comprising:

responsive to a user command, creating an additional node of said meaning tree, said node representing an additional data item; and

associating said node with a description.

7. The method of claim 6, further comprising:

spell checking said associated description.

8. The method of claim 2, wherein a node of said meaning tree is selected, said method further comprising:

responsive to a user request, displaying a dictionary view comprising a plurality of columns for displaying said nodes and parameters of said nodes, wherein said dictionary view includes a data item represented by said highlighted node.

9. The method of claim 2, further comprising:

automatically completing said meaning tree according to predetermined NLU interpretation information selected from the group consisting of a dictionary of data items, and a model specifying text interpretations.

10. The method of claim 9, said automatically completing step comprising:

determining whether a single data item from said dictionary of data items is associated with a word of said NLU training corpus of text; and

if so, automatically assigning said data item to said word.

11. The method of claim 2, further comprising:

displaying selected items of said NLU interpretation information as a tool tip; and

displaying a probability that said presented meaning tree is a correct interpretation.

12. The method of claim 2, further comprising:

searching said NLU interpretation information for a specified meaning tree structure.

13. The method of claim 2, further comprising:

identifying an intersection of data items; and

presenting said identified data items as selections for annotating a user specified node of said meaning tree for a sentence of said NLU training corpus of text.

14. The method of claim 1, said presenting step further comprising:

presenting said NLU interpretation information in a dictionary view comprising a plurality of columns for displaying parent and child data items, and parameters of said data items.

15. The method of claim 14, wherein said dictionary view includes a column for indicating children of said data items and another column for indicating parents of said data items.

16. The method of claim 15, further comprising:

sorting said data items according to said child or parent columns.

17. The method of claim 14, further comprising:

visually indicating data items having a probability or count exceeding a predetermined threshold.

18. The method of claim 14, further comprising:

hiding data items having a probability or count not exceeding a predetermined threshold.

19. The method of claim 14, further comprising:

responsive to a user selection of particular data items having an association, searching said NLU interpretation information for a meaning tree comprising terminal and non-terminal nodes representing said association.

20. The method of claim 14, further comprising:

filtering said NLU interpretation information according to a parameter selected from the group consisting of a data item source, a data item target, a direction associated with a data item, an annotator associated with a data item, annotation status, a node count, a data file, a sentence range, and a usage status.

21. The method of claim 14, further comprising:

displaying histogram information derived from said NLU interpretation information.

22. The method of claim 1, said presenting step further comprising:

in a sentence view, presenting said NLU interpretation information associated with individual text phrases, said NLU interpretation information selected from the group consisting of an annotation status, a phrase count, a designated phrase use, phrase collection information, a correctness probability, and a correctness rank.

23. The method of claim 1, said presenting step further comprising:

presenting said NLU interpretation information as a plurality of meaning trees in a split screen view, said split screen view comprising at least a first window for displaying a first meaning tree and a second window for displaying a second meaning tree.

24. The method of claim 23, further comprising:

while displaying said first meaning tree in said first window, presenting different ones of said meaning trees in said second window responsive to a user request.

25. The method of claim 23, wherein said first and second meaning trees are different interpretations of a same text phrase in a same context.

26. The method of claim 25, wherein said first meaning tree represents a correct interpretation of a text phrase, and said second meaning tree represents a predicted interpretation of said text phrase according to a statistical model.

28. The method of claim 23, wherein said first and second meaning trees are different interpretations of a same text phrase in two different contexts.

29. The method of claim 23, wherein said first meaning tree corresponds to a text phrase after a first processing pass, and said second meaning tree corresponds to said text phrase after a subsequent processing pass.

30. The method of claim 23, wherein said first meaning tree represents a first text phrase, said method further comprising:

receiving an edit to said first meaning tree in said first window; and

responsive to said edit, searching for a meaning tree of a different text phrase which corresponds to said edited first meaning tree, and displaying said meaning tree for said different text phrase in said second window.

31. The method of claim 23, further comprising:

visually indicating differences between said first meaning tree and said second meaning tree.

32. The method of claim 23, further comprising:

conforming said first meaning tree to said second meaning tree.

33. The method of claim 1, further comprising:

automatically importing NLU training sentences; and

automatically determining an interpretation of said NLU training sentences according to statistical likelihoods determined from said NLU training corpus of text.

34. The method of claim 1, further comprising:

automatically importing NLU interpretations of training sentences; and

applying said NLU interpretations to said NLU training corpus of text.

35. The method of claim 1, further comprising:

displaying said NLU interpretation information resulting from one of said multi-passes responsive to a user input specifying said one of said passes.

36. The method of claim 1, further comprising:

ordering text phrases of said NLU training corpus of text in a sentence view according to an attribute of NLU interpretation information for said text phrases; and

sequentially displaying said NLU interpretation information for at least two of said text phrases as meaning trees, wherein said meaning trees are displayed according to said ordering of said sentence view.

27. The method of 26, further comprising:

if a score of said correct interpretation is greater than a score of said predicted interpretation, indicating that said statistical model is incorrect.

37. An integrated development tool for developing a natural language understanding (NLU) application, said integrated development tool comprising:

a database configured to store items of NLU interpretation information corresponding to an NLU training corpus of text;

a graphical editor having a plurality of views for manipulating said items of NLU interpretation information, said graphical editor being communicatively linked to said database; and

a processor configured to determine said items of NLU interpretation information according to a multi-pass system, and to store said items of NLU interpretation information in said database.

38. The integrated development tool of claim 37, further comprising:

a graphical user interface for specifying a search for selected ones of said data items of said NLU interpretation information, said graphical user interface having at least one list of selectable terminals and non-terminals for specifying said search.

39. The integrated development tool of the claim 38, wherein said graphical user interface includes at least one additional area for specifying relationships of said selected data items with other ones of said data items.

40. The integrated development tool of claim 39, further comprising:

a programmed statistical model configured to determine at least one interpretation from said NLU training corpus of text.

41. The integrated development tool of claim 40, said graphical editor having a tree view for presenting said items of NLU interpretation information in a hierarchical tree structure, a dictionary view for presenting individual ones of said items of NLU interpretation information in column format, a sentence view for presenting said items of NLU interpretation information in sentence form, a split screen view for simultaneously presenting two meaning trees, and an import view for importing additional NLU training text and associated interpretation information.

42. A machine-readable storage, having stored thereon a computer program having a plurality of code sections executable by a machine for causing the machine to perform the steps of:

determining NLU interpretation information from an NLU training corpus of text using a multi-pass processing technique, wherein alteration of one pass automatically alters an input for a subsequent pass, said NLU interpretation information specifying an interpretation of at least part of said NLU training corpus of text;

storing said NLU interpretation information in a database; and

in a graphical editor, presenting selected items of NLU interpretation information and receiving user specified edits to said NLU interpretation information.

43. The machine-readable storage of claim 42, said presenting step further comprising:

presenting said NLU interpretation information as a meaning tree comprised of terminal and non-terminal nodes representing data items.

44. The machine-readable storage of claim 43, further comprising:

determining a probability indicating whether a portion of said meaning tree is correct; and

visually identifying said portion of said meaning tree if said probability does not exceed a predetermined threshold probability.

45. The machine-readable storage of claim 43, further comprising:

determining a number of occurrences of a substructure of said meaning tree within said NLU interpretation information; and

visually identifying said substructure of said meaning tree if said number of occurrences does not exceed a predetermined threshold.

46. The machine-readable storage of claim 43, further comprising:

determining an intersection of selected nodes of said meaning tree from said NLU interpretation information;

presenting said intersection of selected nodes as choices for adding an additional node to said meaning tree; and

adding a node above said selected nodes of said meaning tree, wherein said added node is selected from said choices.

47. The machine-readable storage of claim 43, further comprising:

responsive to a user command, creating an additional node of said meaning tree, said node representing an additional data item; and

associating said node with a description.

48. The machine-readable storage of claim 47, further comprising:

spell checking said associated description.

49. The machine-readable storage of claim 43, wherein a node of said meaning tree is selected, said machine-readable storage further comprising:

responsive to a user request, displaying a dictionary view comprising a plurality of columns for displaying said nodes and parameters of said nodes, wherein said dictionary view includes a data item represented by said highlighted node.

50. The machine-readable storage of claim 43, further comprising:

automatically completing said meaning tree according to predetermined NLU interpretation information selected from the group consisting of a dictionary of data items, and a model specifying text interpretations.

51. The machine-readable storage of claim 50, said automatically completing step comprising:

determining whether a single data item from said dictionary of data items is associated with a word of said NLU training corpus of text; and

if so, automatically assigning said data item to said word.

52. The machine-readable storage of claim 43, further comprising:

displaying selected items of said NLU interpretation information as a tool tip; and

displaying a probability that said presented meaning tree is a correct interpretation.

53. The machine-readable storage of claim 43, further comprising:

searching said NLU interpretation information for a specified meaning tree structure.

54. The machine-readable storage of claim 43, further comprising:

identifying an intersection of data items; and

presenting said identified data items as selections for annotating a user specified node of said meaning tree for a sentence of said NLU training corpus of text.

55. The machine-readable storage of claim 42, said presenting step further comprising:

presenting said NLU interpretation information in a dictionary view comprising a plurality of columns for displaying parent and child data items, and parameters of said data items.

56. The machine-readable storage of claim 55, wherein said dictionary view includes a column for indicating children of said data items and another column for indicating parents of said data items.

57. The machine-readable storage of claim 56, further comprising:

sorting said data items according to said child or parent columns.

58. The machine-readable storage of claim 55, further comprising:

visually indicating data items having a probability or count exceeding a predetermined threshold.

59. The machine-readable storage of claim 55, further comprising:

hiding data items having a probability or count not exceeding a predetermined threshold.

60. The machine-readable storage of claim 55, further comprising:

responsive to a user selection of particular data items having an association, searching said NLU interpretation information for a meaning tree comprising terminal and non-terminal nodes representing said association.

61. The machine-readable storage of claim 55, further comprising:

filtering said NLU interpretation information according to a parameter selected from the group consisting of a data item source, a data item target, a direction associated with a data item, an annotator associated with a data item, annotation status, a node count, a data file, a sentence range, and a usage status.

62. The machine-readable storage of claim 55, further comprising:

displaying histogram information derived from said NLU interpretation information.

63. The machine-readable storage of claim 42, said presenting step further comprising:

in a sentence view, presenting said NLU interpretation information associated with individual text phrases, said NLU interpretation information selected from the group consisting of an annotation status, a phrase count, a designated phrase use, phrase collection information, a correctness probability, and a correctness rank.

64. The machine-readable storage of claim 42, said presenting step further comprising:

presenting said NLU interpretation information as a plurality of meaning trees in a split screen view, said split screen view comprising at least a first window for displaying a first meaning tree and a second window for displaying a second meaning tree.

65. The machine-readable storage of claim 64, further comprising:

while displaying said first meaning tree in said first window, presenting different ones of said meaning trees in said second window responsive to a user request.

66. The machine-readable storage of claim 64, wherein said first and second meaning trees are different interpretations of a same text phrase in a same context.

67. The machine-readable storage of claim 66, wherein said first meaning tree represents a correct interpretation of a text phrase, and said second meaning tree represents a predicted interpretation of said text phrase according to a statistical model.

69. The machine-readable storage of claim 64, wherein said first and second meaning trees are different interpretations of a same text phrase in two different contexts.

70. The machine-readable storage of claim 64, wherein said first meaning tree corresponds to a text phrase after a first processing pass, and said second meaning tree corresponds to said text phrase after a subsequent processing pass.

71. The machine-readable storage of claim 64, wherein said first meaning tree represents a first text phrase, said machine-readable storage further comprising:

receiving an edit to said first meaning tree in said first window; and

responsive to said edit, searching for a meaning tree of a different text phrase which corresponds to said edited first meaning tree, and displaying said meaning tree for said different text phrase in said second window.

72. The machine-readable storage of claim 64, further comprising:

visually indicating differences between said first meaning tree and said second meaning tree.

73. The machine-readable storage of claim 64, further comprising:

conforming said first meaning tree to said second meaning tree.

74. The machine-readable storage of claim 42, further comprising:

automatically importing NLU training sentences; and

automatically determining an interpretation of said NLU training sentences according to statistical likelihoods determined from said NLU training corpus of text.

75. The machine-readable storage of claim 42, further comprising:

automatically importing NLU interpretations of training sentences; and

applying said NLU interpretations to said NLU training corpus of text.

76. The machine-readable storage of claim 42, further comprising:

displaying said NLU interpretation information resulting from one of said multi-passes responsive to a user input specifying said one of said passes.

77. The machine-readable storage of claim 42, further comprising:

ordering text phrases of said NLU training corpus of text in a sentence view according to an attribute of NLU interpretation information for said text phrases; and

sequentially displaying said NLU interpretation information for at least two of said text phrases as meaning trees, wherein said meaning trees are displayed according to said ordering of said sentence view.

68. The machine-readable storage of 67, further comprising:

if a score of said correct interpretation is greater than a score of said predicted interpretation, indicating that said statistical model is incorrect.