System and method for evaluating character sets of a message containing a plurality of character sets

• US 6,539,118 B1
• Filed: 08/27/1999
• Issued: 03/25/2003
• Est. Priority Date: 12/31/1998
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Murray, Brendan P., Takizawa, Kuniaki
• Primary Examiner(s): Mancuso, Joseph
• Assistant Examiner(s): Dastouri, Mehrdad
• Application Number:
  US09/384,442
• Time in Patent Office: 1,306 Days
• Field of Search: 382/182, 382/217, 382/218, 382/229, 704/1, 704/2, 704/8, 704/9, 707/530, 707/531, 707/532, 707/533, 707/536, 707/104.1, 707/513, 707/5, 707/102, 358/1.11, 358/1.13
• US Class Current: 382/229
• CPC Class Codes: G06F 40/126 Character encoding

Abstract

An evaluator system accepts input textual messages in unknown languages and assesses which character sets, corresponding to languages, matches that message. Textual messages whose individual characters are encoded in 16 bit Unicode of other universal format are parsed, and character sets which can express each character and the accumulated correspondence is logged. When the character sets against which the message is being tested only provide partial matches, the invention can determine which offers the best fit, including by way of a weighting function. The evaluation technology of the invention can be applied to multipart documents, and to search engines and indices.

Claims

1. A method of evaluating characters in a message containing a plurality of character sets, comprising the steps of:

creating a character table bank having a plurality of columns for each character row, wherein the columns correspond in number to the plurality of character sets and include an indication of whether or not an associated character set is represented in the corresponding character row;

creating a mask, the mask comprising a number of mask columns that correspond to an equivalent number of columns in the character table bank, wherein the mask columns contain an indication of the character sets against which the characters of the message are to be evaluated;

accepting an input of the characters of the message;

evaluating a portion of the message by accessing the corresponding character row of the character table bank for each of a predetermined number of the characters of the message and

performing a logical AND operation between each of the corresponding character rows and the mask;

filling a character match list with an entry for each of the character sets that result in a non-zero result after the logical AND operation; and

returning the character match list.

2. The method of claim 1, wherein the step of accessing a corresponding character row comprises the step of matching a universal code expression for each of the characters of the message to a universal code indicator in the character table bank.

3. The method of claim 2, wherein the universal code is Unicode.

4. The method of claim 1, further comprising the step of computing a total number of characters matched to each of the candidate character sets.

5. The method of claim 1, further comprising the step of repeating the step of evaluating a portion of the message for a next portion of the message.

6. The method of claim 5, further comprising the step of storing a best match for each portion of the message.

7. The method of claim 6, further comprising the step of generating a multipart text having a plurality of character sets corresponding to the best match for each portion of the text.

8. The method of claim 7, wherein the multipart text comprises a MIME document.

9. A system for evaluating characters in a message containing a plurality of character sets, comprising:

a character table bank having a plurality of columns, wherein the columns correspond in number to the plurality of character sets and include an indication of whether or not an associated character set is represented in the corresponding character row;

a mask, the mask comprising a number of mask columns that correspond to an equivalent number of columns in the character table bank, wherein the mask columns contain an indication of the character sets against which the characters in the message are to be evaluated;

an input interface to accept an input of the characters of the message;

a processor unit, connected to the input interface, the processor unit evaluating a portion of the message by accessing the corresponding character row of the character table bank for each of a predetermined number of the characters of the message and

performing a logical AND operation between each of the corresponding character rows and the mask, filling a character match list with an entry for each of the character sets that result in a non-zero result after the logical AND operation; and

returning the character match list.

10. The system of claim 9, wherein the processor unit accesses a corresponding character row by matching a universal code expression for each of the characters of the message to a universal code indicator in the character table bank.

11. The system of claim 10, wherein the universal code is Unicode.

12. The system of claim 9, wherein the processor unit computes a total number of characters matched to each of the candidate character sets.

13. The system of claim 9, wherein the processor unit repeats the evaluating for a next portion of the message.

14. The system of claim 13, wherein the processor unit stores a best match for each portion of the message.

15. The system of claim 14, wherein the processor unit generates a multipart text having plurality of character sets corresponding to the best match for each portion of the document.

16. The system of claim 15, wherein the multipart text comprises a MIME document.

17. A system for evaluating characters in a message containing a plurality of character sets, comprising:

character table bank means, the character table bank means having a plurality of columns, wherein the columns correspond in number to the plurality of character sets and include an indication of whether or not an associated character set is represented in the corresponding character row;

mask means, the mask means comprising a number of mask columns that correspond to an equivalent number of columns in the character table bank, wherein the mask columns contain an indication of the character sets against which the characters in the message are to be evaluated;

input interface means to accept an input of the characters of the message;

processor means, connected to the input interface means, the processor means evaluating a portion of the message by accessing the corresponding character row of the character table bank for each of a predetermined number of the characters of the message and

performing a logical AND operation between each of the corresponding character rows and the mask means, filling a character mach list means with an entry for each of the character sets that result in a non-zero result after the logical AND operation; and

returning the character match list means.

18. The system of claim 17, wherein the processor means accesses a corresponding character row by matching a universal code expression for each of the characters of the message to a universal code indicator in the character table bank.

19. The system of claim 18, wherein the universal code is Unicode.

20. The system of claim 17, wherein the processor means computes a total number of characters matched to each of the candidate character sets.

21. The system of claim 17, wherein the processor means repeats the evaluating for a next portion of the message.

22. The system of claim 21, wherein the processor means stores a best match for each portion of the message.

23. The system of claim 22, wherein the processor means generates a multipart text having a plurality of character sets corresponding to the best match for each portion of the document.

24. The system of 23, wherein the multipart text comprises a MIME document.

25. A storage medium for storing machine readable code, the machine readable code being executable to evaluate characters in a message containing a plurality of character sets according to the steps of:

creating a character table bank having a plurality of columns, wherein the columns correspond in number to the plurality of character sets and include an indication of whether or not an associated character set is represented in the corresponding character row;

creating a mask, the mask comprising a number of mask columns that correspond to an equivalent number of columns in the character table bank, wherein the mask columns contain an indication of the character sets against which the characters of the message are to be evaluated;

accepting an input of the characters of the message;

evaluating a portion of the message by accessing the corresponding character row of the character table bank for each of a predetermined number of the characters of the message and

performing a logical AND operation between each of the corresponding character rows and the mask;

filling a character match list with an entry for each of the character sets that result in a non-zero result after the logical AND operation; and

returning the character match list.

26. The storage medium of claim 25, wherein the step of accessing a corresponding character row comprises the step of matching a universal code expression for each of the characters of the message to a universal code indicator in the character table bank.

27. The storage medium of claim 26, wherein the universal code is Unicode.

28. The storage medium of claim 25, wherein the steps further comprise the step of computing a total number of characters matched to each of the candidate character sets.

29. The storage medium of claim 25, wherein the steps further comprise the step of repeating the step of evaluating a portion of the message for a next portion of the message.

30. The storage medium of claim 29, further comprising the step of storing a best match for each portion of the message.

31. The storage medium of claim 30, wherein the steps further comprise the step of generating a multipart text having a plurality of character sets corresponding to the best match for each portion of the text.

32. The storage medium of claim 31, wherein the multipart text comprises a MIME document.