Data disambiguation systems and methods

• US 20060004826A1
• Filed: 05/04/2004
• Published: 01/05/2006
• Est. Priority Date: 05/04/2004
• Status: Active Grant

• Current Assignee: Verint Americas Incorporated (Verint Systems Incorporated)
• Original Assignee: Next IT Corporation (Verint Systems Incorporated)
• Inventors: Hust, Robert, Zartler, Mark
• Granted Patent: US 7,565,368 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3344 using natural language analysis
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure

Abstract

Various embodiments provide a state-based, regular expression parser in which data, such as generally unstructured text, is received into the system and undergoes a tokenization process which permits structure to be imparted to the data. Tokenization of the data effectively enables various patterns in the data to be identified. In some embodiments, one or more components can utilize stimulus/response paradigms to recognize and react to patterns in the data.

Claims

1. A system comprising:

one or more computer-readable media;

a functional presence engine embodied on the one or more computer-readable media and configured as a probabilistic parser to receive text and perform lexical analysis on the text effective to tokenize text portions; and

a knowledge base component embodied on the one or more computer-readable media and operably associated with the functional presence engine and configured to define;

cases of text that can be matched to text received by the functional presence engine; and

responses that are triggered in an event of a match;

2. The system of claim 1, wherein the functional presence engine is configured to use:

one or more lexical files, individual lexical files specifying one or more matching rules and corresponding output symbols; and

a lexical analysis program which processes text to produce tokenized text portions in accordance with the one or more matching rules specified by the one or more lexical files.

3. The system of claim 2, wherein the one or more rules are specified as regular expressions.

4. The system of claim 2, wherein the one or more rules are specified as regular expressions, and wherein a default behavior of the functional presence engine is to attempt to match all regular expressions and then select the rule that produces the best match.

5. The system of claim 2, wherein the one or more rules are specified as regular expressions, and wherein a behavior can be specified for the functional presence engine in which a first successfully matched rule is selected as the rule whose output symbol will be utilized.

6. The system of claim 2, wherein individual rules can produce more than one output symbol.

7. The system of claim 2, wherein the lexical analysis program is configured to select a rule that produces a best match and responsive thereto, utilize the output symbol associated with the rule that produced the best match.

8. The system of claim 2, wherein the one or more lexical files can comprise a macro section that specifies macro values that are substitutable for macro names, and a lex section that specifies lexical rewrite rules.

9. The system of claim 2, wherein the one or more lexical files can comprise a macro section that specifies macro values that are substitutable for macro names, and a lex section that specifies lexical rewrite rules, and wherein the lex section comprises a main section that contains rules that are executed at a top level of a tokenization process, and a sub-section associated with a rule in the main section, the sub-section containing a group rules that get executed only if the associated main section rule produces the best match.

10. A method comprising:

providing a functional presence engine configured as a probabilistic parser to receive text and perform lexical analysis on the text effective to tokenize text portions; and

providing a knowledge base component operably associated with the functional presence engine and configured to define;

cases of text that can be matched to text received by the functional presence engine; and

responses that are triggered in an event of a match.

11. The method of claim 10, wherein the act of providing the functional presence engine comprises providing a functional presence engine that is configured to use:

one or more lexical files, individual lexical files specifying one or more matching rules and corresponding output symbols; and

a lexical analysis program which processes text to produce tokenized text portions in accordance with the one or more matching rules specified by the one or more lexical files.

12. The method of claim 11, wherein the one or more rules are specified as regular expressions.

13. The method of claim 11, wherein the one or more rules are specified as regular expressions, and wherein the act of providing the functional presence engine comprises providing a functional presence engine that is configured to attempt to match all regular expressions and then select the rule that produces the best match.

14. The method of claim 11, wherein the one or more rules are specified as regular expressions, and wherein the act of providing the functional presence engine comprises providing a functional presence engine having a behavior in which a first successfully matched rule is selected as the rule whose output symbol will be utilized.

15. The method of claim 11, wherein the lexical analysis program is configured to select a rule that produces a best match and responsive thereto, utilize the output symbol associated with the rule that produced the best match.

16. The method of claim 11, wherein the one or more lexical files can comprise a macro section that specifies macro values that are substitutable for macro names, and a lex section that specifies lexical rewrite rules.

17. The method of claim 11, wherein the one or more lexical files can comprise a macro section that specifies macro values that are substitutable for macro names, and a lex section that specifies lexical rewrite rules, and wherein the lex section comprises a main section that contains rules that are executed at a top level of a tokenization process, and a sub-section associated with a rule in the main section, the sub-section containing a group rules that get executed only if the associated main section rule produces the best match.

18. A system comprising:

one or more computer-readable media;

a functional presence engine embodied on the one or more computer-readable media and configured as a probabilistic parser to receive text and perform lexical analysis on the text effective to tokenize text portions; and

a knowledge base component embodied on the one or more computer-readable media and operably associated with the functional presence engine, the knowledge base component being associated with one or more files that define;

cases of text that can be matched to text received by the functional presence engine; and

responses that are triggered in an event of a match;

the files being defined in a hierarchical tag-based language.

19. The system of claim 18, wherein the hierarchical tag-based language comprises an extensible markup language.

20. The system of claim 18, wherein the responses can comprise conversational responses.

21. The system of claim 18, wherein the responses can comprise non-conversational responses.

22. The system of claim 18, wherein one tag comprises a top level tag that encapsulates all other tags.

23. The system of claim 22, wherein another tag comprises a tag that specifies a file to be loaded.

24. The system of claim 22, wherein another tag comprises a tag that specifies a file that is to be loaded and used to perform the lexical analysis.

25. The system of claim 22, wherein another tag comprises a tag that specifies a case whose sub-tags identify text patterns that are attempted to be matched and associated responses that should be taken in an event of a match.

26. The system of claim 25, wherein text patterns are defined as regular expressions.

27. The system of claim 25, wherein another tag comprises a tag that supports recursion of responses.

28. The system of claim 22, wherein another tag comprises a conditional tag that defines one or more conditions that control execution flow.

29. The system of claim 22, wherein another tag comprises a tag that is used to randomly select an encapsulated element for generating a response.

30. The system of claim 22, wherein another tag comprises a tag that is used to set a variable to a specified value.

31. The system of claim 22, wherein another tag comprises a tag that is used to retrieve a variable value.

32. The system of claim 22, wherein another tag comprises a tag that is used to retrieve unified wildcard values.

33. The system of claim 22, wherein another tag comprises a tag that is used to communicate with an application.

34. A method comprising:

providing a functional presence engine configured as a probabilistic parser to receive text and perform lexical analysis on the text effective to tokenize text portions; and

providing a knowledge base component operably associated with the functional presence engine, the knowledge base component being associated with one or more files that define;

cases of text that can be matched to text received by the functional presence engine; and

responses that are triggered in an event of a match;

the files being defined in a hierarchical tag-based language.

35. The method of claim 34, wherein the hierarchical tag-based language comprises an extensible markup language.

36. The method of claim 34, wherein the responses can comprise conversational responses.

37. The method of claim 34, wherein the responses can comprise non-conversational responses.

38. The method of claim 34, wherein one tag comprises a top level tag that encapsulates all other tags.

39. The method of claim 38, wherein another tag comprises a tag that specifies a file to be loaded.

40. The method of claim 38, wherein another tag comprises a tag that specifies a file that is to be loaded and used to perform the lexical analysis.

41. The method of claim 38, wherein another tag comprises a tag that specifies a case whose sub-tags identify text patterns that are attempted to be matched and associated responses that should be taken in an event of a match.

42. The method of claim 38, wherein another tag comprises a tag that specifies a case whose sub-tags identify text patterns that are attempted to be matched and associated responses that should be taken in an event of a match, and wherein text patterns are defined as regular expressions.

43. The method of claim 38, wherein another tag comprises a tag that specifies a case whose sub-tags identify text patterns that are attempted to be matched and associated responses that should be taken in an event of a match, and wherein another tag comprises a tag that supports recursion of responses.

44. The method of claim 38, wherein another tag comprises a conditional tag that defines one or more conditions that control execution flow.

45. The method of claim 38, wherein another tag comprises a tag that is used to randomly select an encapsulated element for generating a response.

46. The method of claim 38, wherein another tag comprises a tag that is used to set a variable to a specified value.

47. The method of claim 38, wherein another tag comprises a tag that is used to retrieve a variable value.

48. The method of claim 38, wherein another tag comprises a tag that is used to retrieve unified wildcard values.

49. The method of claim 38, wherein another tag comprises a tag that is used to communicate with an application.

50. A software architecture comprising:

one or more computer-readable media;

one or more runtime objects embodied on the one or more computer-readable media, the one or more runtime objects being configured to receive and process text;

one or more knowledge base objects embodied on the one or more computer-readable media, individual knowledge base objects being associated with one or more runtime objects,

individual knowledge base objects being associated with a hierarchical, tag-based file that defines;

cases of text that can be matched to text received by associated runtime objects; and

responses that are triggered in an event of a match;

the runtime objects and knowledge base objects collectively processing data received by the runtime objects to tokenize text portions.

51. The software architecture of claim 50, wherein individual runtime objects can be associated with more than one knowledge base object.

52. The software architecture of claim 50, wherein runtime objects are configured to maintain state information associated with the text that it receives.

53. The software architecture of claim 50, wherein runtime objects are configured to maintain state information associated with processing that takes place on one or more knowledge base objects.

54. The software architecture of claim 50, wherein knowledge base objects are configured to process text received from runtime objects and return a score and an indication of associated node that generated the score to runtime objects from which text was received.

55. The software architecture of claim 50, wherein:

knowledge base objects are configured to;

process text received from runtime objects, and

return a score and an indication of associated node that generated the score to runtime objects from which text was received, and runtime objects are configured to;

receive scores and associated node indications, and responsive thereto, call a method to implement a response associated with the node that generated a score.

56. A system comprising:

one or more computer-readable media;

a functional presence engine embodied on the one or more computer-readable media and configured as a probabilistic parser to receive documents and perform lexical analysis on the documents effective to tokenize document portions; and

a knowledge base component embodied on the one or more computer-readable media and operably associated with the functional presence engine and configured to define;

cases of text that can appear in documents and be matched to documents received by the functional presence engine; and

responses that are triggered in an event of a match;

a table embodied on the one or more computer-readable media and associated with the functional presence engine and knowledge base component and configured to contain entries associated with tokens assigned in the tokenization process and documents in which the tokens appear; and

an information retrieval component configured to search for tokens and identify associated documents.

57. The system of claim 56, wherein the tokenized document portions comprise individual words that appear in the documents.

58. The system of claim 56, wherein the information retrieval component is configured to use regular expressions to search for tokens.

59. The system of claim 56, wherein the functional presence engine is configured to use:

one or more lexical files, individual lexical files specifying one or more matching rules and corresponding output symbols; and

a lexical analysis program which processes documents to produce tokenized document portions in accordance with the one or more matching rules specified by the one or more lexical files.

60. The system of claim 59, wherein the one or more rules are specified as regular expressions.

61. The system of claim 56, wherein the knowledge base component is associated with one or more files that define cases of text that can be matched to text in documents received by the functional presence engine and responses that are triggered in an event of a match, the files being defined in a hierarchical tag-based language.

62. The system of claim 61, wherein the hierarchical tag-based language comprises an extensible markup language.

63. The system of claim 61, wherein one tag comprises a top level tag that encapsulates all other tags.

64. The system of claim 61, wherein one tag comprises a top level tag that encapsulates all other tags, and wherein another tag comprises a tag that specifies a file to be loaded.

65. The system of claim 61, wherein one tag comprises a top level tag that encapsulates all other tags, and wherein another tag comprises a tag that specifies a file that is to be loaded and used to perform the lexical analysis.

66. The system of claim 61, wherein one tag comprises a top level tag that encapsulates all other tags, and wherein another tag comprises a tag that specifies a case whose sub-tags identify text patterns that are attempted to be matched and associated responses that should be taken in an event of a match.

67. The system of claim 61, wherein one tag comprises a top level tag that encapsulates all other tags, and wherein another tag comprises a tag that specifies a case whose sub-tags identify text patterns that are attempted to be matched and associated responses that should be taken in an event of a match, wherein the text patterns are defined as regular expressions.

68. A method comprising:

providing a functional presence engine configured as a probabilistic parser to receive documents and perform lexical analysis on the documents effective to tokenize document portions;

providing a knowledge base component operably associated with the functional presence engine and configured to define;

cases of text that can appear in documents and be matched to documents received by the functional presence engine, and

responses that are triggered in an event of a match;

using the functional presence engine and the knowledge base component, building a table that contains entries associated with tokens assigned in the tokenization process and documents in which the tokens appear; and

using an information retrieval component to search for tokens and identify associated documents.

69. The method of claim 68, wherein the tokenized document portions comprise individual words that appear in the documents.

70. The method of claim 68, wherein the information retrieval component is configured to use regular expressions to search for tokens.

71. The method of claim 68, wherein the functional presence engine is configured to use:

one or more lexical files, individual lexical files specifying one or more matching rules and corresponding output symbols; and

a lexical analysis program which processes documents to produce tokenized document portions in accordance with the one or more matching rules specified by the one or more lexical files.

72. The method of claim 71, wherein the one or more rules are specified as regular expressions.

73. The method of claim 68, wherein the knowledge base component is associated with one or more files that define:

cases of text that can be matched to text in documents received by the functional presence engine; and

responses that are triggered in an event of a match, the files being defined in a hierarchical tag-based language.

74. The method of claim 73, wherein the hierarchical tag-based language comprises an extensible markup language.

75. The method of claim 73, wherein one tag comprises a top level tag that encapsulates all other tags.

76. The method of claim 73, wherein one tag comprises a top level tag that encapsulates all other tags, and wherein another tag comprises a tag that specifies a file to be loaded.

77. The method of claim 73, wherein one tag comprises a top level tag that encapsulates all other tags, and wherein another tag comprises a tag that specifies a file that is to be loaded and used to perform the lexical analysis.

78. The method of claim 73, wherein one tag comprises a top level tag that encapsulates all other tags, and wherein another tag comprises a tag that specifies a case whose sub-tags identify text patterns that are attempted to be matched and associated responses that should be taken in an event of a match.

79. The method of claim 73, wherein one tag comprises a top level tag that encapsulates all other tags, and wherein another tag comprises a tag that specifies a case whose sub-tags identify text patterns that are attempted to be matched and associated responses that should be taken in an event of a match, wherein text patterns are defined as regular expressions.

80. A system comprising:

one or more computer-readable media;

a dynamic agent server embodied on the one or more computer-readable media and configured to manage deployable agents;

one or more deployable agents embodied on the one or more computer-readable media and configured to be deployed by the dynamic agent server;

one or more knowledge base objects embodied on the one or more computer-readable media and associated with and useable by the one or more deployable agents, the knowledge base objects being associated with one or more files that define cases of text that can be matched by a functional presence engine configured as a probabilistic parser, to text obtained from one or more data origination entities, the one or more files being defined in a hierarchical tag-based language.

81. The system of claim 80, wherein the hierarchical tag-based language comprises an extensible markup language.

82. The system of claim 80, wherein an agent comprises a data source that provides a pipeline through which data travels and a runtime object that is configured to receive and process data in the form of text that travels through the pipeline.

83. The system of claim 80, wherein an agent comprises a data source that provides a pipeline through which data travels and a runtime object that is configured to receive and process data in the form of text that travels through the pipeline, and wherein the system comprises a plurality of agents, and wherein data sources can comprise different types of data sources.

84. The system of claim 83, wherein at least one data source comprises an IRC data source.

85. The system of claim 83, wherein at least one data source comprises a TCP/IP data source.

86. The system of claim 83, wherein at least one data source comprises a POP3 data source.

87. The system of claim 80, wherein said one or more deployable agents comprises agents of a first type and agents of a second different type.

88. The system of claim 80, wherein said one or more deployable agents comprises agents of a first type and agents of a second different type, wherein a first type of agent is a passive agent that listens to one or more data origination entities.

89. The system of claim 80, wherein said one or more deployable agents comprises agents of a first type and agents of a second different type, wherein a first type of agent is a passive agent that listens to one or more data origination entities, and wherein a second type of agent is an active agent that is configured to interact with a data origination entity.

90. A method comprising:

providing a dynamic agent server configured to manage deployable agents;

deploying one or more deployable agents that are configured to receive data;

with the one or more deployable agents, using one or more knowledge base objects to process the data using a functional presence engine configured as a probabilistic parser, the knowledge base objects being associated with one or more files that define cases of text that can be matched to text obtained from one or more data origination entities, the one or more files being defined in a hierarchical tag-based language.

91. The method of claim 90, wherein the hierarchical tag-based language comprises an extensible markup language.

92. The method of claim 90, wherein an agent comprises a data source that provides a pipeline through which data travels and a runtime object that is configured to receive and process data in the form of text that travels through the pipeline.

93. The method of claim 90, wherein an agent comprises a data source that provides a pipeline through which data travels and a runtime object that is configured to receive and process data in the form of text that travels through the pipeline, and wherein the system comprises a plurality of agents, and wherein data sources can comprise different types of data sources.

94. The method of claim 90, wherein said one or more deployable agents comprises agents of a first type and agents of a second different type.

95. The method of claim 90, wherein said one or more deployable agents comprises agents of a first type and agents of a second different type, and wherein a first type of agent is a passive agent that listens to one or more data origination entities.

96. The method of claim 90, wherein said one or more deployable agents comprises agents of a first type and agents of a second different type, and wherein a first type of agent is a passive agent that listens to one or more data origination entities, and wherein a second type of agent is an active agent that is configured to interact with a data origination entity.

97. A method comprising:

receiving text obtained from a data origination entity;

probabilistically parsing the text effective to tokenize text portions with tokens, said probabilistically parsing comprising using one or more matching rules defined as regular expressions in an attempt to match to text received from the data origination entity;

conducting one or more searches on the tokens.

98. The method of claim 97, wherein the act of probabilistically parsing comprises using one or more files defining a hierarchical tag-based structure to provide said one or more matching rules.

99. The method of claim 97, wherein the act of probabilistically parsing comprises using one or more files defining a hierarchical tag-based structure to provide said one or more matching rules, and wherein the hierarchical tag-based structure comprises a tag that specifies a file that is to be loaded and used to perform said probabilistically parsing.

100. The method of claim 97, wherein said acts of receiving and probabilistically parsing are performed, collectively, by one or more runtime objects that receive the text and one or more knowledge base objects that define how the parsing is to take place.

101. The method of claim 97, wherein said acts of receiving and probabilistically parsing are performed, collectively, by one or more runtime objects that receive the text and one or more knowledge base objects that define how the parsing is to take place, and wherein individual runtime objects can be associated with more than one knowledge base object.

102. The method of claim 97, wherein said acts of receiving and probabilistically parsing are performed, collectively, by one or more runtime objects that receive the text and one or more knowledge base objects that define how the parsing is to take place, and wherein runtime objects are configured to maintain state information associated with text that is received.

103. The method of claim 97, wherein said acts of receiving and probabilistically parsing are performed, collectively, by one or more runtime objects that receive the text and one or more knowledge base objects that define how the parsing is to take place, and wherein the knowledge base objects are configured to process text received from the runtime objects and return scores generated by processing the text, to the run time objects.