System and method for identifying word patterns in text

• US 20030167276A1
• Filed: 03/07/2001
• Published: 09/04/2003
• Est. Priority Date: 03/07/2001
• Status: Active Grant

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Simpson, Don M., Usey, Robert W. Jr.
• Granted Patent: US 7,426,505 B2
• Time in Patent Office: Days
• US Class Current: 707/102
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

A system and method for identifying word patterns in text is conducted in real time and is highly suitable for network and Internet use. The system comprises a semantic network that may be compiled on a local computer or at a remote host and a software text analysis module for receiving the text to be analyzed, parsing the text, submitting the text to the semantic network, and receiving the results. The method involves receiving a stream of text, breaking the stream of text into a plurality of threads, tokenizing the words in each thread, and comparing the words to identified words in the semantic network. Recognized, words are then examined, together with surrounding words in the text to determine whether the words are part of a word pattern. Word patterns are located at nodes in the semantic network in a hierarchical structure, and certain word patterns correspond to objects of the semantic network. When all word patterns involving a word are located, links are followed to objects corresponding to the word patterns. Several nodes may point to a single object, but each object is represented only once in the semantic network. Identified objects may thus be identified in real time, as the text streams through the text analysis module.

Claims

1. A method for identifying objects referenced in a stream of text, the method comprising:

receiving an incoming stream of text comprised of words;

consulting a semantic network to automatically identify one or more word patterns in the incoming stream of text with a single examination of each word; and

referencing a known object identified by a word pattern of the semantic network.

2. The method of claim 1, further comprising loading the semantic network substantially entirely into RAM memory of a processor and wherein the step of consulting the semantic network is conducted by consulting the semantic network within the RAM memory.

3. The method of claim 1, further comprising dividing the stream of text into a plurality of threads and conducting the step of consulting a semantic network concurrently for words in each of the plurality of threads and further comprising examining groups of words spread over adjacent threads at the boundaries of the threads for word patterns.

4. The method of claim 1, wherein consulting a semantic network further comprises consulting a semantic network of recognized words and patterns of words in a hierarchical order moving from identified nodes to related nodes linked with the identified nodes.

5. The method of claim 1, wherein consulting a semantic network further comprises examining words in the stream of text in a sequential order as the words are received and formatting the stream of text to represent identified objects without persistently storing the stream of text.

6. The method of claim 1, further comprising tokenizing the stream of text into individual words, and wherein the step of consulting a semantic network further comprises analyzing each word in order of occurrence in the stream of text by comparing the individual words to identified words in the semantic network.

7. The method of claim 6, wherein the step of consulting a semantic network further comprises finding a match between an individual word in the stream of text and an identified word in the semantic network and comparing the individual word and an adjacent word of the stream of text to a word pattern in the semantic network.

8. The method of claim 6, wherein the step of consulting a semantic network further comprises continually adding words of the stream of text to recognized word patterns and comparing the result to other word patterns in the semantic network until no more word patterns containing the individual word are located.

9. The method of claim 1, further comprising presenting the identified known objects to a user.

10. The method of claim 9, wherein presenting the identified known objects to a user further comprises providing links between identified word patterns in the stream of text and objects in a knowledge base to which the word patterns identify.

11. The method of claim 10, wherein providing links between identified word patterns in the stream of text and objects in a knowledge base comprises displaying the word patterns corresponding to an object as a URL, the URL leading to information related to the object.

12. A method for identifying objects referenced in a stream of text, the method comprising:

loading a semantic network substantially entirely into RAM memory of a processor, the semantic network comprised of recognized words and patterns of words in a hierarchical order;

receiving an incoming stream of text comprised of words;

tokenizing the stream of text into individual words;

examining the individual words in the stream of text in a sequential order as the words are received by consulting the semantic network within the RAM memory to automatically identify one or more word patterns in the incoming stream of text with a single examination of each individual word in the order that the individual words are received examining the individual words comprising;

finding a match between an individual word in the stream of text and an identified word in the semantic network and comparing the individual word and an adjacent word of the stream of text to a word pattern in the semantic network, and

continually adding words of the stream of text to recognized word patterns and comparing the result to other word patterns in the semantic network until no more word patterns containing the individual word are located;

referencing a known object identified by a word pattern of the semantic network; and

formatting the stream of text to represent identified objects without persistently storing the stream of text.

13. A system for identifying objects referenced in a stream of text, the system comprising:

an input pipeline configured to receive an incoming stream of text comprised of words;

a text analysis module configured to consult a semantic network to automatically identify one or more word patterns in the incoming stream of text with a single examination of each word; and

an object association module configured to reference a known object identified by a word pattern of the semantic network.

14. The system of claim 13, wherein the semantic network is configured to be loaded substantially entirely into RAM memory of a processor, and wherein the text analysis module is configured to consult the semantic network within the RAM memory.

15. The system of claim 13, wherein the input pipeline is further configured to divide the stream of text into a plurality of threads such that the semantic network may be consulted concurrently for words in each of the plurality of threads.

16. The system of claim 13, further comprising a word pattern comparison module configured to consult the semantic network in a hierarchical order moving from identified nodes to related nodes linked with the identified nodes.

17. The system of claim 13, wherein the text analysis module is further configured to examine words in the stream of text in a sequential order as the words are received and format the stream of text to represent identified objects without persistently storing the stream of text.

18. The system of claim 13, wherein the text analysis module is further configured to break the stream of text into individual words, and to analyze each word in an order of occurrence of the word in the stream of text by comparing the individual words to identified words in the semantic network.

19. The system of claim 18, wherein the text analysis module is configured to, upon finding a match between an individual word in the stream of text and an identified word in the semantic network, compare the individual word and an adjacent word of the stream of text to a word pattern in the semantic network.

20. The system of claim 18, wherein the text analysis module is configured to continually add words of the stream of text to recognized word patterns and compare the result to other word patterns in the semantic network until no more word patterns are located.

21. The system of claim 13, further comprising a compilation module configured to present the identified known objects to a user.

22. The system of claim 21, wherein the compilation module is configured to present the identified known objects to a user by providing links between identified word patterns in the stream of text and objects in a knowledge base to which the word patterns identify.