System and Method for Building and Retrieving a Full Text Index

• US 20080065607A1
• Filed: 09/06/2007
• Published: 03/13/2008
• Est. Priority Date: 09/08/2006
• Status: Active Grant

• Current Assignee: Open Text Holdings, Inc. (Open Text Corporation)
• Original Assignee: Guidance Software Incorporated (Open Text Corporation)
• Inventors: Weber, Dominik
• Granted Patent: US 7,752,193 B2
• Time in Patent Office: Days
• US Class Current: 707/3
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing

Abstract

An indexing engine generates a full text index of English and non-English files provided to the indexing engine. The indexing engine receives an input file for indexing, and normalizes the unique words contained in the input file. The normalizing includes stripping the words of any diacritical marks, taking into account different multilingual issues, case folding the words into lowercase, and the like. The normalized words are stored in a dictionary, and a word record is generated for each stored word. Each word record includes a flag that indicates whether one or more variations exist in the input file for the normalized word. One or more tables store information on the variations for the normalized words. When a query engine is invoked to search for an input query word, the variations are searched only if the user has set an option to consider such variations.

Claims

1. An indexing method comprising:

receiving an input file for indexing, the input file including a word including a diacritic symbol;

storing information on the diacritic symbol in a first data structure;

stripping the word of the diacritic symbol; and

storing the stripped word in a second data structure,

wherein, a query engine is configured to receive an input query word and search the second data structure for a match of the input query word.

2. The method of claim 1, wherein the input query word includes the diacritic symbol, and the query engine is further configured to:

strip the input query word of the diacritic symbol; and

search the second data structure for the stripped input query word.

3. The method of claim 2, wherein the query engine is further configured to:

search the first data structure for a match of the diacritic symbol stripped from the input query word if the input query word is associated with a command to consider diacritics.

4. An indexing method comprising:

receiving an input file for indexing;

normalizing a plurality of words contained in the input file;

storing the plurality of normalized words in a word table; and

generating a word record for each of the plurality of normalized words in the word table, each word record including one or more flags for indicating whether one or more variations exist in the file for the normalized word,

wherein, a query engine is configured to receive an input query word, normalize the input query word, and search the word table for the normalized input query word.

5. The method of claim 4, wherein normalizing one of the plurality of words includes stripping the word of a diacritic symbol.

6. The method of claim 4, wherein normalizing one of the plurality of words includes replacing a particular character of the word with a predefined character.

7. The method of claim 4 further comprising:

generating one or more tables storing information on the one or more variations for each of the normalized words, wherein, the query engine is configured to search the one or more tables for information on the one or more variations matching the input query word responsive to a command to consider the one or more variations.

8. The method of claim 7, wherein the variation of one of the normalized words includes a diacritic symbol, and the table stores information on the diacritic symbol.

9. The method of claim 7, wherein the variation of one of the normalized words includes a casing variation.

10. A method for processing a search query for a query word, the method comprising:

receiving an input query word;

normalizing the input query word;

searching a word table for the normalized input query word, the word table storing a plurality of normalized words associated with a file;

retrieving from the word table information on a word record for a normalized word matching the normalized input query word, the word record including a flag for indicating whether one or more variations exist in the file for the normalized word, wherein the one or more variations are considered for a match against the input query word responsive to a command to consider the one or more variations.

11. The method of claim 10, wherein normalizing the query word includes stripping the query word of a diacritic symbol.

12. The method of claim 10, wherein normalizing the query word includes replacing a particular character with a predefined character.

13. The method of claim 10, wherein one or more tables store information on the one or more variations of the normalized word, the method further comprising:

searching the one or more tables for information on the one or more variations matching the input query word responsive to the command to consider the one or more variations.

14. The method of claim 13, wherein one the variations of the normalized word includes a diacritic symbol, and the table stores information on the diacritic symbol.

15. The method of claim 13, wherein one of the variations of the normalized word is a casing variation.

16. An indexing system comprising:

an input file for indexing;

means for normalizing a plurality of words contained in the input file;

a word table storing the plurality of normalized words; and

a word record for each of the plurality of normalized words, each word record including a flag for indicating whether one or more variations exist in the input file for the normalized word,

wherein, a query engine is configured to receive an input query word, normalize the input query word, and search the word table for the normalized input query word.

17. The system of claim 16, wherein the means for normalizing one of the plurality of words includes stripping the word of a diacritic symbol.

18. The system of claim 16, wherein the means for normalizing one of the plurality of words includes replacing a particular character of the word with a predefined character.

19. The system of claim 16 further comprising:

one or more tables storing information on the one or more variations for each of the normalized words, wherein, the query engine is configured to search the one or more tables for information on the one or more variations matching the input query word responsive to a command to consider the one or more variations.

20. The system of claim 16, wherein the variation of one of the normalized words includes a diacritic symbol, and the table stores information on the diacritic symbol.

21. The system of claim 16, wherein the variation of one of the normalized words includes a casing variation.

22. A system for processing a search query for a query word, the system comprising:

a word table storing a plurality of normalized words associated with a file;

a word record for each normalized word stored in the word table, each word record including a flag for indicating whether one or more variations exist in the file for the normalized word;

a processor; and

a memory operably coupled to the processor and storing program instructions therein, the processor being operable to execute the program instructions, the program instructions including;

receiving an input query word;

normalizing the input query word;

searching the word table for the normalized input query word; and

retrieving from the word table information on the word record for the normalized word matching the normalized input query word, wherein the one or more variations indicated in the retrieved word record are considered for a match against the query word responsive to a command to consider the one or more variations.

23. The system of claim 22, wherein the normalizing the query word includes stripping the word of a diacritic symbol.

24. The system of claim 22 further comprising:

one or more tables storing information on the one or more variations indicated in the retrieved word record, wherein the program instructions further include searching the one or more tables for information on the one or more variations matching the input query word responsive to the command to consider the one or more variations.

25. The system of claim 24, wherein one of the variations indicated in the retrieved word record is inclusion of a diacritic symbol, and the table stores information on the diacritic symbol.