System and method for adaptive sentence boundary disambiguation

• US 8,131,546 B1
• Filed: 12/28/2007
• Issued: 03/06/2012
• Est. Priority Date: 01/03/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: Storediq Incorporated (International Business Machines Corporation)
• Inventors: Zoellner, Keith
• Primary Examiner(s): Han, Qi
• Application Number:
  US11/965,934
• Time in Patent Office: 1,530 Days
• Field of Search: 704/248, 704/253, 704/1, 704/9, 704/4
• US Class Current: 704/248
• CPC Class Codes: G06F 40/163 Handling of whitespace
• CPC Class Codes: G06F 40/177 of tables; using ruled lines
• CPC Class Codes: G06F 40/183 Tabulation, i.e. one-dimensional positioning
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars

Abstract

Embodiments disclosed herein provide a system and method useful for pre-processing non-sentence text extracted from business documents (e.g., malformed bulleted lists, runaway sentence identification, spatially separated data, etc.). One embodiment includes two heuristic algorithms: one searches for sentences in a document and another looks for non-sentences (e.g., lists, tables, tabs, names of people, addresses, etc.) in the same document. In one embodiment, when malformed text is encountered, a particular character (e.g., “?”) is inserted to signify to a natural language processing layer that this set of “words” represent a logical construct and should be evaluated independent of other sentences. Embodiments disclosed herein allow non-sentence text, which is linguistically dry but contextually rich, be included in the natural language processing. Embodiments disclosed herein also facilitate to reduce false positive concept extraction assertions by the natural language processing layer.

Claims

1. A method for adaptive sentence boundary disambiguation, comprising:

receiving, from a natural language processing system, a document containing text;

identifying, by a first heuristic algorithm, sentence text in the document;

identifying, by a second heuristic algorithm, non-sentence text in the document, wherein the second heuristic algorithm is operable to identify one of non-sentence texts in a group consisting of lists, tables, names of people, addresses, text without a sentence structure, text included as a list and spatially separated data as non-sentence text;

parsing said non-sentence text into one or more logical constructs, wherein each logical construct comprises a set of words;

inserting a disambiguator after each of said one or more logical constructs to define a sentence boundary for the logical construct based on one or more natural language structures; and

sending the disambiguated document to the natural language processing system, wherein the disambiguated document consists of disambiguated sentences, each disambiguated sentence having a defined boundary and including related contextual information, and the disambiguator is used to signal the natural language processing system the presence of a logical construct to be evaluated independently of other logical constructs.

2. The method according to claim 1, wherein metadata indicates the document includes a list.

3. The method according to claim 1, wherein the disambiguator comprises a question mark.

4. The method according to claim 1, further comprising processing said non-sentence text to extract concepts from said one or more logical constructs.

5. The method according to claim 1, further comprising determining whether a predetermined number of consecutive lines in said document contain a number of words that is equal to or less than a threshold, wherein said predetermined number and said threshold are configurable.

6. The method according to claim 1, further comprising determining whether words in said document are separated by a predetermined number of consecutive spaces or tabs, wherein said predetermined number is configurable.

7. The method according to claim 1, further comprising determining whether there are consecutive words in said document that equal to or exceed a predetermined number without a proper end-of-sentence mark, wherein said predetermined number is configurable.

8. A non-transitory computer-readable storage medium carrying computer-executable program instructions translatable to implement a method for adaptive sentence boundary disambiguation, comprising:

receiving, from a natural language processing system, a document containing text;

identifying, by a first heuristic algorithm, sentence text in the document;

identifying, by a second heuristic algorithm, non-sentence text in the document, wherein the second heuristic algorithm is operable to identify one of non-sentence texts in a group consisting of lists, tables, names of people, addresses, text without a sentence structure, text included as a list and spatially separated data as non-sentence text;

parsing said non-sentence text into one or more logical constructs, wherein each logical construct comprises a set of words;

inserting a disambiguator after each of said one or more logical constructs to define a sentence boundary for the logical construct based on one or more natural language structures, and

sending the disambiguated document to the natural language processing system, wherein the disambiguated document consists of disambiguated sentences, each disambiguated sentence having a defined boundary and including related contextual information, and the disambiguator is used to signal the natural language processing system the presence of a logical construct to be evaluated independently of other logical constructs.

9. The non-transitory computer-readable storage medium of claim 8, wherein metadata indicates the document includes a list.

10. The non-transitory computer-readable storage medium of claim 8, wherein the disambiguator comprises a question mark.

11. The non-transitory computer-readable storage medium of claim 10, wherein said computer-executable program instructions are further translatable to determine whether a predetermined number of consecutive lines in said document contain a number of words that is equal to or less than a threshold, wherein said predetermined number and said threshold are configurable.

12. The non-transitory computer-readable storage medium of claim 10, wherein said computer-executable program instructions are further translatable to determine whether words in said document are separated by a predetermined number of consecutive spaces or tabs, wherein said predetermined number is configurable.

13. The non-transitory computer-readable storage medium of claim 10, wherein said computer-executable program instructions are further translatable to determine whether there are consecutive words in said document that equal to or exceed a predetermined number without a proper end-of-sentence mark, wherein said predetermined number is configurable.

14. A system for adaptive sentence boundary disambiguation, comprising:

a processor;

a computer-readable storage medium accessible by said processor and carrying program instructions executable by said processor to;

receive, from a natural language processing system, a document containing text;

identify, by a first heuristic algorithm, sentence text in the document;

identify, by a second heuristic algorithm, non-sentence text in the document, wherein the second heuristic algorithm is operable to identify one of non-sentence texts in a group consisting of lists, tables, names of people, addresses, text without a sentence structure, text included as a list and spatially separated data as non-sentence text;

parse said non-sentence text into one or more logical constructs, wherein each logical construct comprises a set of words; and

insert a disambiguator after each of said one or more logical constructs to define a sentence boundary for the logical construct based on one or more natural language structures and

send the disambiguated document to the natural language processing system, wherein the disambiguated document consists of disambiguated sentences, each disambiguated sentence having a defined boundary and including related contextual information, and the disambiguator is used to signal the natural language processing system the presence of a logical construct to be evaluated independently of other logical constructs.

15. The system of claim 14, wherein the disambiguator comprises a question mark.

16. The system of claim 14, wherein said program instructions are further executable by said processor to determine whether a predetermined number of consecutive lines in said document contain a number of words that is equal to or less than a threshold, wherein said predetermined number and said threshold are configurable.

17. The system of claim 14, wherein said program instructions are further executable by said processor to determine whether words in said document are separated by a predetermined number of consecutive spaces or tabs, wherein said predetermined number is configurable.

18. The system of claim 14, wherein said program instructions are further executable by said processor to determine whether there are consecutive words in said document that equal to or exceed a predetermined number without a proper end-of-sentence mark, wherein said predetermined number is configurable.