Lexicon with sectionalized data and method of using the same

• US 7,490,034 B2
• Filed: 04/30/2002
• Issued: 02/10/2009
• Est. Priority Date: 04/30/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Finnigan, James P., Huttenhower, Curtis E., Potter, Douglas W., Powell, Kevin R.
• Primary Examiner(s): Edouard; Patrick N
• Assistant Examiner(s): YEN, ERIC L
• Application Number:
  US10/137,456
• Publication Number:
  US 20030204392A1
• Time in Patent Office: 2,478 Days
• Field of Search: 382/186, 382/305, 434/169, 364/419.11, 364/900, 395/613, 704/7, 704/101, 704/1, 704/9, 704/10, 707/3, 707/532
• US Class Current: 704/10
• CPC Class Codes: G06F 40/242 Dictionaries

Abstract

A data structure for a word lexicon includes a plurality of separate data sections for storing information related to word entries. An indices section includes pointers indicating the location of the information. The location of the associated pointers for each word entry is obtained as a function of a list of the word entries.

Claims

1. A computer readable storage medium having a lexicon for storing word information and adapted for use with a text analyzer in a language processing system, wherein the lexicon is adapted to be used in a plurality of language processing tasks, the lexicon comprising:

a word list section for storing a list of words;

a set of data sections corresponding with each word on the word list, wherein the data sections store substantially different selected information about the corresponding word in the word list; and

for each word on the word list, a plurality of pointers stored in an indices table apart from the sets of data sections, each of the pointers pointing to a different data section related to different information about the corresponding word, wherein the plurality of pointers comprises a first set and a second set of the pointers, the first set used to access information related to a first natural language processing task and the second set used to access information related to a second natural language processing task, wherein the first set of the pointers is not the same as the second set of the pointers.

2. The computer readable storage medium of claim 1 wherein the word list section comprises a Trie data structure.

3. The computer readable storage medium of claim 2 wherein each pointer includes an identification as a function of nodes in the Trie data structure.

4. The computer readable storage medium of claim 2 wherein each pointer includes an identification comprising an offset value stored with the Trie data structure.

5. The computer readable storage medium of claim 1 wherein each pointer includes an identification comprising an offset value stored with the word list section.

6. The computer readable storage medium of claim 1 and further comprising an index table section for storing identifications correlated with words of the word list section, each word in the word list section having a corresponding entry in the index table section.

7. The computer readable storage medium of claim 1 wherein each of the pointers includes a first indication of which data section to access and a second indication of an offset value related to the information stored therein.

8. The computer readable storage medium of claim 7 wherein each of the pointers includes a third indication of a classification of the word.

9. The computer readable storage medium of claim 8 wherein the classification comprises part of speech (POS).

10. The computer readable storage medium of claim 8 wherein the classification comprises one of tonal and inflection features.

11. The computer readable storage medium of claim 8 wherein the pointers of each classification of each of the words are sequentially provided as a set in the indices section, and wherein each of the pointers includes a fourth indication comprising a flag indicating the last pointer of the corresponding set.

12. The computer readable storage medium of claim 11 and further comprising a string heap section for storing a plurality of strings, wherein a plurality of data locations in the set of data sections corresponding to each word reference the same string in the string heap section.

13. The computer readable storage medium of claim 11 and further comprising a header section providing information as to the size and location of the word list section, the indices sections and each of the set of data sections corresponding to each word.

14. The computer readable storage medium of claim 1 wherein one of the set of data sections corresponding to each word stores information related to spell checking.

15. The computer readable storage medium of claim 1 wherein one of the set of data sections corresponding to each word stores information related to morphology.

16. The computer readable storage medium of claim 1 wherein one of the set of data sections corresponding to each word stores information related to linguistics.

17. The computer readable storage medium of claim 1 wherein one of the set of data sections corresponding to each word stores information indicating a word belongs to a multiple word expression.

18. The computer readable storage medium of claim 1 wherein two data sections of the set of data sections corresponding to each word separately store information selected from the group consisting of spell checking information, morphology information, linguistic information and multiple word expression information.

19. The computer-readable storage medium of claim 1, wherein the language processing tasks include at least one of grammar checking and spell checking.

20. The computer readable storage medium of claim 1, wherein one of the data sections is a standard author data section.

21. The computer readable storage medium of claim 1, wherein one of the data sections is a standard linguistic data section.

22. The computer readable storage medium of claim 1, wherein one of the data sections is a parsing data section.

23. The computer readable storage medium of claim 1, wherein one of the data sections is a domain/subject data section.

24. The computer readable storage medium of claim 1, wherein one of the data sections is a spelling data section.

25. The computer readable storage medium of claim 1, wherein one of the data sections is a multiple word expression data section.

26. A computer-implemented method for obtaining word information by accessing a lexicon that is adapted for use in a plurality of different natural language processing tasks, wherein the lexicon is adapted to be used with a text analyzer in a language processing system, and wherein the lexicon stores word information pertaining to a plurality of words, the lexicon comprising:

a word list section storing the plurality of words;

sets of data sections, wherein each set of data sections corresponds with an individual word in the word list section, each data section among a set of data sections storing different selected information about the corresponding word in the word list; and

an indices section storing a plurality of pointers apart from the sets of data sections, wherein each plurality of pointers corresponds with an individual word and comprises a first set of pointers associated with a natural language processing task and a second set of pointers associated with a different natural language processing task, wherein the first set of pointers is different from the second set of pointers, each of the sets of pointers pointing to data in a data section, the method comprising;

accessing the word list section as a function of said word to ascertain a pointer identification for the indices section;

using the pointer identification to obtain one of the first or second sets of pointers in the indices section based on the natural language processing task to be performed;

using one of the first or second sets of pointers to obtain information from only some data sections of the set of data sections, the only some data sections having information about said word necessary to perform the natural language processing task.

27. The computer-implemented method of claim 26 wherein the identification is an offset value stored with the word list section.

28. The computer-implemented method of claim 26 wherein the lexicon further comprises an index table section storing each of the identifications correlated with words of the word list section, and wherein accessing the word list section as a function of said word to ascertain a pointer set identification for the indices section further includes using the word list section to find a corresponding entry in the index table section having the corresponding identification.

29. The computer-implemented method of claim 28 wherein the pointers of the pointer set pertaining to each word are grouped together as a function of a classification, and wherein the method further comprises identifying the classification and using the pointers of the identified classification.

30. The computer-implemented method of claim 29 wherein the classification is a function of part of speech (POS).

31. The computer-implemented method of claim 26, and further comprising selectively accessing the data sections based on the identified pointers, wherein selectively accessing comprises only accessing data sections associated with an identified pointer.

32. The computer-implemented method of claim 31, wherein selectively accessing comprises not accessing some data sections.

37. The computer-implemented method of claim 26 wherein the classification is a function of part of speech (POS).

33. A computer-implemented method for storing word information in a lexicon adapted for use in a plurality of natural language processing tasks, wherein the lexicon is adapted for use with a text analyzer in a language processing system, the method comprising:

storing word information in a sets of data sections, each set of data sections corresponding with an individual word in a word list, each data section of a set of data sections storing substantially different selected information about the corresponding word in the word list;

storing sets of pointers in an indices section separate from the sets of data sections, each set of pointers corresponding with an individual word in the word list, wherein some words comprise a first set of pointers associated with a natural language processing task and a second set of pointers associated with a different natural language processing task, wherein the first set of pointers is different from the second set of pointers, and wherein each of the pointers points to selected data in the sets of data sections; and

storing the word list in a word list section separate from the sets of data sections and the indices section, the word list having information for identifying the corresponding set of pointers corresponding to an individual word.

34. The computer-implemented method of claim 33 wherein the information for identifying is an offset value stored with the word list section.

35. The computer-implemented method of claim 33 and further comprising:

storing identification values in an index table section separate from the sets of data sections, each identification value corresponding to a word in the word list section and being associated with a pointer set in the indices section.

36. The computer-implemented method of claim 35 wherein the set of pointers corresponding to each word are grouped together as a function of a classification, and wherein storing sets of pointers includes assigning an indication of the classification in each of the pointers.

38. A computer-implemented method for obtaining word information from a plurality of lexicons adapted for use in a plurality of different language processing tasks, the plurality of lexicons adapted for use with a text analyzer of a language processing system, each lexicon comprising:

a word list section storing a plurality of words;

sets of data sections, wherein each set of data sections corresponds with an individual word in the word list, each data section of a set of data sections storing different selected information about the corresponding word in the word list; and

an indices section storing a plurality of pointers for each individual word in the word list, the indices section stored apart from the sets of data sections, wherein each plurality of pointers comprises a first set of pointers associated with a natural language processing task and a second set of pointers associated with a different natural language processing task, wherein the first set of pointers is different from the second set of pointers, each of the pointers pointing to data in the sets of data sections, the method comprising;

selectively accessing the sets of data sections of each of the lexicons having similar information and obtaining information therefrom using the first set or second set of pointers based on the particular language processing task performed.

39. The computer-implemented method of claim 38 wherein selectively accessing includes combining the word information from at least two data sections of each of the lexicons having similar information.

40. The computer-implemented method of claim 38 wherein selectively accessing includes obtaining the word information from a data section of a first lexicon, obtaining the word information from a data section of a second lexicon and then using only the information from the data section of the second lexicon.

41. The computer-implemented method of claim 38 wherein selectively accessing includes obtaining the word information from similar data sections of each lexicon until a stop indicator is located.

42. The computer-implemented method of claim 38 selectively accessing the data sections includes sequentially accessing the plurality of lexicons in a selected order where a first instruction is followed pertaining to whether or not to read the word information from each lexicon, and where a second instruction is followed pertaining to combine the word information of more than one of the plurality of lexicons.

43. The computer-implemented method of claim 38 wherein selectively accessing the data sections comprises:

for each lexicon that is accessed,

accessing the corresponding word list section as a function of a given word to ascertain a pointer identification for the corresponding indices section based on the particular natural language processing task;

using the pointer identification to obtain a pointer of the corresponding indices section; and

using the pointer to ascertain which corresponding data section of the plurality of data sections has information about said word and where the information is located in the corresponding data section.

44. The computer-implemented method of claim 43 selectively accessing the data sections includes sequentially accessing the lexicons in a selected order.