System and method for evaluating characters in an inputted search string against a character table bank comprising a predetermined number of columns that correspond to a plurality of pre-determined candidate character sets in order to provide enhanced full text search

• US 7,039,637 B2
• Filed: 08/27/1999
• Issued: 05/02/2006
• Est. Priority Date: 12/31/1998
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Murray, Brendan P., Takizawa, Kuniaki
• Primary Examiner(s): CHANNAVAJJALA, SRIRAMA T
• Application Number:
  US09/384,088
• Publication Number:
  US 20030009447A1
• Time in Patent Office: 2,440 Days
• Field of Search: 707 1- 9, 707100-1041, 707200-205, 707/501.1, 707511-515, 707531-532, 707536-535, 704 8- 10, 704/5, 704/1, 703/27, 706/11, 706/45, 706/58, 345/716, 345/741, 345/968, 345/551, 708/207, 708/671, 340/146.2, 382/229, 715/536
• US Class Current: 1/1
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 40/126 Character encoding
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

An evaluator system accepts input textual messages in unknown languages and assesses which character sets, corresponding to languages, matches that message. Textual messages whose individual characters are encoded in 16 bit Unicode or other universal format are parsed, and character sets which can express each character and the accumulated correspondence is logged. When the character sets against which the message is being tested only provide partial matches, the invention can determine which offers the best fit, including by means of a weighting function. The evaluation technology of the invention can be applied to multipart documents, and to search engines and indices. Documents can be indexed according to assigned character sets, and quary strings matched to indices according to language.

Claims

1. A computer implemented method of evaluating characters in an inputted search string to generate a search index, comprising the steps of:

a) accepting an input of the characters of the search string, wherein the characters can be represented in any of a plurality of character sets corresponding to an undetermined language;

b) evaluating the search string by comparing each of the characters of the search string to a plurality of pre-determined candidate character sets to determine one or more matches between the plurality of pre-determined candidate character sets and the search string; and

c) generating the search index by assigning character sets to a code page, wherein the character sets are assigned based on he results of the evaluation of the charcters of the search string and the plurality of pre-determined candidate character sets that correspond to the characters of the search string.

2. The method of claim 1, wherein the comparing of step (b) comprises the step of comparing each character of the search string to an entry for each of the candidate character sets if a character table bank.

3. The method of claim 2, wherein the step of comparing each character comprises the step of testing the ability of each candidate character set to express that character by performing a logical mask between a universal code for that character and an indicator in the character table bank indicating whether each of the candidate character sets contains that character.

4. The method of claim 3, wherein the universal code is Unicode.

5. The method of claim 1, further comprising the step of(d) computing a total number of characters matched to each of the candidate character sets.

6. The method of claim 1, further comprising the step of (e) selecting a best match based upon the total number of characters matched to each of the candidate character sets.

7. The method of claim 1, further comprising the step of (f) evaluating the characters of a query string.

8. The method of claim 7, further comprising the step of (g) performing a search of the query string against search indices whose character set matches the character set of the query string.

9. A computer implemented system for evaluating characters in an inputted search string to generate a search index, comprising:

an input interface to accept an input of the characters of the search string, wherein the characters can be represented in any of a plurality of character sets corresponding to an undetermined language; and

a processor unit, connected to the input interface, the processor unit evaluating the search string by comparing each of the characters of the search string to a plurality of pre-determined candidate character sets to determine one or more matches between the plurality of pre-determined candidate character sets and the search string, and generating the search index by assigning character sets to a code page, wherein the character sets are assigned based on the results of the evaluation of the characters of the search string and the plurality of pre-determined candidate character sets that correspond to the characters of the search string.

10. The system of claim 9, wherein the processor unit compares each character of the search string to an entry for each of the candidate character sets in a character table bank.

11. The system of claim 10, wherein the processor unit tests the ability of each candidate character set to express that character by performing a logical mask between a universal code for that character and an indicator in the character table bank indicating whether each of the candidate character sets contains that character.

12. The system of claim 11, wherein the universal code is Unicode.

13. The system of claim 9, wherein the processor unit computes a total number of characters matched to each of the candidate character sets.

14. The system of claim 9, wherein the processor unit selects a best match based upon the total number of characters matched to each of the candidate character sets.

15. The system of claim 9, wherein the processor unit evaluates the characters of a query string.

16. The system of claim 15, wherein the processor unit performs a search of the query string against search indices whose character set matches the character set of the query string.

17. A computer implemented system for evaluating characters in an inputted search string to generate a search index, comprising:

input interface means to accept an input of the characters of the search string, wherein the characters can be represented in any of a plurality of character sets corresponding to an undetermined language; and

processor means, connected to the input interface means, the processor means evaluating the search string by comparing each of the characters of the search string to a plurality of pre-determined candidate character sets to determine one or more matches between the plurality of pre-determined candidate character sets and the search string, and generating the search index by assigning character sets to a code page, wherein the character sets are assigned based on the results of the evaluation of the characters of the search string and the plurality of pre-determined candidate character sets that correspond to the characters of the search string.

18. The system of claim 17, wherein the processor means compares each character of the search string to an entry for each of the candidate character sets in a character table bank.

19. The system of claim 18, wherein the processor means tests the ability of each candidate character set to express that character by performing a logical mask between a universal code for that character and an indicator in the character table bank indicating whether each of the candidate character sets contains that character.

20. The system of claim 19, wherein the universal code is Unicode.

21. The system of claim 17, wherein the processor means computes a total number of characters matched to each of the candidate character sets.

22. The system of claim 17, wherein the processor means selects a best match based upon the total number of characters matched to each of the candidate character sets.

23. The system of claim 17, wherein the processor means evaluates the characters of a query string.

24. The system of claim 23, wherein the processor means performs a search of the query string against search indices whose character set matches the character set of the query string.

25. A computer implemented storage medium for storing machine readable code, the machine readable code being executable to evaluate characters in an inputted electronic search string according to the steps of:

a) accepting an input of the characters of the search string, wherein the characters can be represented in any of a plurality of character sets corresponding to an undetermined language;

b) evaluating the search string by comparing each of the characters of the search string to a plurality of pre-determined candidate character sets to determine one or more matches between the plurality of pre-determined candidate character sets and the search string; and

c) generating the search index by assigning character sets to a code page, wherein the character sets are assigned based on the results of the evaluation of the characters of the search string and the plurality of pre-determined candidate character sets that correspond to the characters of the search string.

26. The storage medium of claim 25, wherein the comparing of step (b) comprises the step of comparing each character of the search string to an entry to each of the candidate character sets in a character table bank.

27. The storage medium of claim 26, wherein the step of comparing each character comprises the step of testing the ability of each candidate character set to express that character by performing a logical mask between a universal code for that character and an indicator in the character table bank indicating whether each of the candidate character sets contains that character.

28. The storage medium of claim 27, wherein the universal code is Unicode.

29. The storage medium of claim 25, wherein the steps further comprise the step of (d) computing a total number of characters matched to each of the candidate character sets.

30. The storage medium of claim 25, wherein the steps further comprise the step of (e) selecting a best match based upon the total number of characters matched to each of the candidate character sets.

31. The storage medium of claim 25, wherein the steps further comprise the step of (f) evaluating the characters of a query string.

32. The storage medium of claim 31, wherein the steps further comprise the step of (g) performing a search of the query string against search indices whose character set matches the character set of the query string.

33. A computer implemented method of evaluating characters in an inputted search string to generate a search index, comprising the steps of:

a) accepting an input of the characters of the search string, wherein the characters can be represented in any of a plurality of character sets corresponding to an undetermined language;

b) evaluating the search string by comparing each of the characters of the search string to a plurality of pre-determined candidate character sets to determine one or more matches between the plurality of pre-determined candidate character sets and the search string, wherein each of the characters of the search string are compared to one or more character sets of a character bank by parsing the characters of the search string and identifying the one or more character sets of the character bank that express each of the characters of the search string, wherein each of the character sets represented in the character bank that correspond to each of the characters of the search string are compared to pre-selected character set indicators of a bit mask to determine a match between each of the character sets represented in the character bank that correspond to the characters of the search string and the characters set indicators of the bit mask, wherein a first column of the character bank corresponds to a first column of the bit mask, and wherein the first column of the character bank and the first column of bit mask correspond to the same character set; and

c) generating a search index based on the results of the evaluation of the search string and the plurality of pre-determined candidate character sets.

34. A computer implemented method of evaluating characters in an inputted search string against a character table bank comprising a predetermined number of columns that correspond to a plurality of pre-determined candidate character sets in order to provide enhanced full text search features, the method comprising:

accepting the inputted search string having at least one character;

comparing the at least one character of the inputted search string to the plurality of pre-determined candidate character sets;

filling the columns of the character table bank to indicate whether or not the at least one character of the string is supported by corresponding pre-determined candidate character sets;

creating a bit mask comprising columns equivalent in number to the number of columns in the character table bank;

filling the columns of the bit mask to provide an indication of the plurality of pre-determined character sets against which the filled columns of the character table bank are to be matched;

evaluating the search string by comparing the filled character table bank against the filled columns of the bit mask; and

generating a search index based on the results of the evaluation.

35. A computer implemented system for evaluating characters in an inputted search string against a character table bank comprising a predetermined number of columns that correspond to a plurality of pre-determined candidate character sets in order to provide enhanced full text search features, the system comprising:

an accepting module that accepts the inputted search string having at least one character;

a comparing module that compares the at least one character of the inputted search string to the plurality of pre-determined candidate character sets;

a bank filling module that fills the columns of the character table bank to indicate whether or not the at least one character of the string is supported by corresponding pre-determined candidate character sets;

a creating module that creates a bit mask comprising columns equivalent in number to the number of columns in the character table bank;

a mask filling module that fills the columns of the bit mask to provide an indication of the plurality of pre-determined character sets against which the filled columns of the character table bank are to be matched;

an evaluating module that evaluates the search string by comparing the filled character table bank against the filled columns of the bit mask; and

an index generating module that generates a search index based on the results of the evaluation.

36. A computer implemented system for evaluating characters in an inputted search string against a character table bank comprising a predetermined number of columns that correspond to a plurality of pre-determined candidate character sets in order to provide enhanced full text search features, the system comprising:

accepting means that accepts the inputted search string having at least one character;

comparing means that compares the at least one character of the inputted search string to the plurality of pre-determined candidate character sets;

bank filling means that fills the columns of the character table bank to indicate whether or not the at least one character of the string is supported by corresponding pre-determined candidate character sets;

creating means that creates a bit mask comprising columns equivalent in number to the number of columns in the character table bank;

mask filling means that fills the columns of the bit mask to provide an indication of the plurality of pre-determined character sets against which the filled columns of the character table bank are to be matched;

evaluating means that evaluates the search string by comparing the filled character table bank against the filled columns of the bit mask; and

index generating means that generates a search index based on the results of the evaluation.

37. A computer implemented storage medium for storing machine readable code, the machine readable code being executable to evaluate characters in an inputted search string against a character table bank comprising a predetermined number of columns that correspond to a plurality of pre-determined candidate character sets in order to provide enhanced full text search features, the method comprising:

accepting the inputted search string having at least one character;

comparing the at least one character of the inputted search string to the plurality of pre-determined candidate character sets;

filling the columns of the character table bank to indicate whether or not the at least one character of the string is supported by corresponding pre-determined candidate character sets;

creating a bit mask comprising columns equivalent in number to the number of columns in the character table bank;

filling the columns of the bit mask to provide an indication of the plurality of pre-determined character sets against which the filled columns of the character table bank are to be matched;

evaluating the search string by comparing the filled character table bank against the filled columns of the bit mask; and

generating a search index based on the results of the evaluation.