Apparatus and method for extracting and manipulating the reading order of text to prepare a display document for analysis

• US 9,658,989 B2
• Filed: 08/25/2008
• Issued: 05/23/2017
• Est. Priority Date: 09/03/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Lee, Adrian
• Primary Examiner(s): Paula, Cesar
• Assistant Examiner(s): Blackwell, James H
• Application Number:
  US12/197,860
• Publication Number:
  US 20090063965A1
• Time in Patent Office: 3,193 Days
• Field of Search: 715255-256, 715264-265, 707705, 707736, 707E17001, 704 1- 10, 704251, 7042701, 382173-180
• CPC Class Codes: G06F 40/129 Handling non-Latin characters, e.g. kana-to-kanji conversion
• CPC Class Codes: G06F 40/263 Language identification
• CPC Class Codes: G06F 40/53 Processing of non-Latin text kana-to-kanji conversion G06F40/129; vowelisation G06F40/232
• CPC Class Codes: G06F 9/454 Multi-language systems; Localisation; Internationalisation

Abstract

A method for preparing a display document for analysis includes: extracting character data from the display document; determining a first order associated with processing of the character data and a second order associated with a logical order of the character data; determining whether the first order is different from the second order; and reversing at least a portion of the character data in response to the determination that the first order is different from the second order.

Claims

1. A method for preparing a display document for analysis comprising:

extracting character data from said display document, wherein a language of said character data in said display document is unknown when said character data is extracted;

determining a first order associated with processing of said character data and a second order associated with a logical order of said character data, including comparing said character data against a set of dictionaries to determine said second order based on a match between said character data and a word listed in a dictionary of said set of dictionaries, each dictionary corresponding to a particular language and listing words of that language, wherein comparing said character data against a set of dictionaries further comprises, if a first comparison of said character data to said dictionaries does not determine a language of said character data, reversing an order of said character data and making a second comparison of said reversed character data against said set of dictionaries;

determining whether said first order is different from said second order; and

reversing at least a portion of said character data in response to said determination that said first order is different from said second order.

2. The method of claim 1, further comprising:

determining a commonly-occurring word in said character data;

comparing said commonly-occurring word against a rule; and

in response to said comparing step, determining said second order.

3. The method of claim 1, further comprising:

selecting a dictionary from said set for comparing said character data based on a previously-determined indicator of a language of said character data to increase a speed at which a matching dictionary is identified from the set of dictionaries.

4. The method of claim 3, further comprising determining a language of said character data based on said comparison to said set of dictionaries and determining said second order based on said determined language of said character data.

5. The method of claim 1, further comprising:

identifying at least one language associated with said character data in said document.

6. The method of claim 5, further comprising analyzing at least one of:

vocabulary associated with said character data;

at least one character encoding associated with said character data and script data associated with a script of said character data.

7. The method of claim 5, further comprising using said identified language for determining a rule against which to compare said character data in said document, in order to determine said second order.

8. The method of claim 7, wherein at least one of said language identifier and said order identifier is operable to generate a confidence value associated with at least one of:

said identified language and said determined second order.

9. The method of claim 8, further comprising comparing said confidence value against a confidence threshold.

10. The method of claim 1, further comprising determining whether said character data in said document has a predominant first order.

11. The method of claim 1, further comprising:

determining whether individual subsets of said character data have a different respective first order; and

reversing at least one of said subsets according to said second order.

12. The method of claim 1, further comprising merging results from two or more different analyses to identify a language of said character data.

13. A method for preparing a display document for analysis comprising:

extracting character data from said display document, wherein a language of said character data in said display document is unknown when said character data is extracted;

determining a first order associated with processing of said character data and a second order associated with a logical order of said character data;

determining whether said first order is different from said second order; and

reversing at least a portion of said character data in response to said determination that said first order is different from said second order;

wherein determining the second order comprises identifying a punctuation character that is position dependent such that a space character will appear on only one side of the punctuation character, where the side of the punctuation character on which the space character appears depends on said second order; and

comparing characters around said punctuation character data against a rule to determine said second order.

14. The method of claim 13, wherein said second order is determined by identifying, in said character data, a full stop character and then determining on which side of said full stop character a space character appears.

15. An apparatus for preparing a display document for analysis comprising a processor implementing:

an extractor for extracting character data from said display document, wherein the character data comprises image data representing an image of a number of characters without including character codes;

an order identifier for determining a first order associated with processing of said character data and a second order associated with a logical order of said character data, and for determining whether said first order is different from said second order, wherein determining the second order comprises identifying a punctuation character that is position dependent such that a space character will appear on only one side of the punctuation character, where the side of the punctuation character on which the space character appears depends on said second order; and

a reverse component for reversing at least a portion of said character data, responsive to said order identifier determining that said first order is different from said second order.

16. The apparatus of claim 15, wherein said order identifier is configured to:

determine position dependent character data, wherein said position dependent character data comprises at least one of;

a punctuation character and presentation data;

compare said position dependent character data against a rule; and

in response to said comparison, determine said second order.

17. The apparatus of claim 15, wherein said order identifier is configured to:

determine a commonly-occurring word in said character data;

compare said commonly-occurring word against a rule; and

in response to said comparison, determine said second order.

18. The apparatus of claim 15, wherein said order identifier is configured to compare said character data against a set of dictionaries in order to determine said second order.

19. The apparatus of claim 15, further comprising a language identifier for identifying at least one language associated with said character data in said document, wherein said language identifier is configured to analyze at least one of:

vocabulary associated with said character data;

at least one character encoding associated with said character data, and script data associated with a script of said character data.

20. The apparatus of claim 19, wherein said order identifier is configured to use said identified language for determining a rule against which to compare said character data in said document, in order to determine said second order.

21. The apparatus of claim 20, wherein said language identifier and said order identifier are configured to generate a confidence value associated with at least one of:

said identified language and said determined second order.

22. A computer program product for preparing a Portable Document Format (PDF) document for analysis, the computer program product comprising:

a non-transitory computer usable storage medium having computer usable program code embodied therewith, the computer usable program code comprising;

non-transitory computer usable program code configured to extract character data from said PDF document, wherein a language of said character data in said PDF document is unknown when said character data is extracted;

non-transitory computer usable program code configured to determine a first order associated with processing of said character data and a second order associated with a logical order of said character data, including comparing said character data against a set of dictionaries to determine said second order based on a match between said character data and a word listed in a dictionary of said set of dictionaries, each dictionary corresponding to a particular language and listing words of that language, wherein comparing said character data against a set of dictionaries further comprises, if a first comparison of said character data to said dictionaries does not determine a language of said character data, reversing an order of said character data and making a second comparison of said reversed character data against said set of dictionaries;

non-transitory computer usable program code configured to determine whether said first order is different from said second order; and

non-transitory computer usable program code configured to reverse said character data in response to a determination that said first order is different from said second order.

23. An apparatus for preparing a display document for analysis comprising a processor implementing:

an extractor to extract character data from said display document, wherein a language of said character data in said display document is unknown when said character data is extracted;

an order identifier to determine a first order associated with processing of said character data and a second order associated with a logical order of said character data, including comparing said character data against a set of dictionaries to determine said second order based on a match between said character data and a word listed in a dictionary of said set of dictionaries, each dictionary corresponding to a particular language and listing words of that language, wherein comparing said character data against a set of dictionaries further comprises, if a first comparison of said character data to said dictionaries does not determine a language of said character data, reversing an order of said character data and making a second comparison of said reversed character data against said set of dictionaries; and

a reverse component for reversing at least a portion of said character data, responsive to said order identifier determining that said first order is different from said second order.