Method for identifying word patterns in text

• US 7,426,505 B2
• Filed: 03/07/2001
• Issued: 09/16/2008
• Est. Priority Date: 03/07/2001
• Status: Expired due to Term

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Simpson, Don M., Usey, Robert W. Jr.
• Primary Examiner(s): NGUYEN, CAM LINH T
• Application Number:
  US09/801,340
• Publication Number:
  US 20030167276A1
• Time in Patent Office: 2,750 Days
• Field of Search: 707 1- 5, 707/3, 707/6, 707/102, 706/55
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

A method for identifying word patterns in text is conducted in real time and is highly suitable for network and Internet use. The method involves receiving a stream of text, breaking the stream of text into a plurality of threads, tokenizing the words in each thread, and comparing the words to identified words in the semantic network. Recognized, words are then examined, together with surrounding words in the text to determine whether the words are part of a word pattern. Word patterns are located at nodes in the semantic network in a hierarchical structure, and certain word patterns correspond to objects of the semantic network. When all word patterns involving a word are located, links are followed to objects corresponding to the word patterns. Several nodes may point to a single object, but each object is represented only once in the semantic network. Identified objects may thus be identified in real time, as the text streams through the text analysis module.

Claims

1. A method for identifying objects referenced in a stream of text, the method comprising:

receiving an incoming stream of text;

tokenizing the stream of text into individual words;

constructing word patterns of one or more consecutive words from the stream of text;

consulting a semantic network to automatically find a match between one or more word patterns in the incoming stream of text and a word pattern in the semantic network, such that each word in the incoming stream is searched once in the semantic network; and

referencing a known object within the semantic network based on an identified word pattern from the stream of text, the known object identified by a word pattern of the semantic network.

2. The method of claim 1, further comprising loading the semantic network substantially entirely into a common RAM memory space of a processor and wherein the step of consulting the semantic network is conducted by consulting the semantic network within the RAM memory.

3. The method of claim 1, further comprising dividing the stream of text into a plurality of threads and conducting the step of consulting a semantic network concurrently for words in each of the plurality of threads and further comprising examining groups of words spread over adjacent threads at the boundaries of the threads for word patterns.

4. The method of claim 1, wherein consulting a semantic network further comprises consulting a semantic network of recognized words and patterns of words in a hierarchical order moving from identified nodes to related nodes linked with the identified nodes.

5. The method of claim 1, wherein consulting a semantic network further comprises examining words in the stream of text in a sequential order as the words are received and formatting the stream of text to represent identified objects without persistently storing the stream of text.

6. The method of claim 1, wherein the step of consulting a semantic network further comprises continually adding words of the stream of text to recognized word patterns and comparing the result to other word patterns in the semantic network until no more word patterns containing the individual word are located.

7. The method of claim 1, further comprising presenting the identified known objects to a user.

8. The method of claim 7, wherein presenting the identified known objects to a user further comprises providing links between identified word patterns in the stream of text and objects in a knowledge base to which the word patterns identify.

9. The method of claim 8, wherein providing links between identified word patterns in the stream of text and objects in a knowledge base comprises displaying the word patterns corresponding to an object as a URL, the URL leading to information related to the object.

10. A method for identifying objects referenced in a stream of text, the method comprising:

loading a semantic network substantially entirely into a common RAM memory space of a processor, the semantic network comprised of recognized words and patterns of words in a hierarchical order;

receiving an incoming stream of text comprised of words;

tokenizing the stream of text into individual words;

examining the individual words in the stream of text in a sequential order as the words are received by consulting the semantic network within the RAM memory to automatically identify one or more word patterns in the incoming stream of text, such that each word in the incoming stream is searched once in the semantic network in the order that the individual words are received, examining the individual words comprising;

finding a match between an individual word in the stream of text and an identified word in the semantic network and comparing the individual word and an adjacent word of the stream of text to a word pattern in the semantic network, and

continually adding words of the stream of text to recognized word patterns and comparing the result to other word patterns in the semantic network until no more word patterns containing the individual word are located;

referencing a known object within the semantic network, the known object identified by a word pattern of the semantic network; and

formatting the stream of text to represent identified objects without persistently storing the stream of text.