Processing of String Inputs Utilizing Machine Learning

• US 20180232357A1
• Filed: 12/13/2016
• Published: 08/16/2018
• Est. Priority Date: 12/13/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Beller, Charles E., Ding, Chengmin, Ginsberg, Allen, Shek, Elinna
• Granted Patent: US 10,546,063 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 40/103 Formatting, i.e. changing of presentation of documents automatic justification G06F40/189; automatic line break hyphenation G06F40/191
• CPC Class Codes: G06F 40/151 Transformation
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/177 of tables; using ruled lines
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking

Abstract

Natural language processing of raw text data for optimal sentence boundary placement. Raw text is extracted from a document and subject to cleaning. The extracted raw text is examined to identify preliminary sentence boundaries, which are used to identify potential sentences in the raw text. One or more potential sentences are assigned a well-formedness score. A value of the score correlates to whether the potential sentence is a truncated/ill-formed sentence or a well-formed sentence. One or more preliminary sentence boundaries are optimized depending on the value of the score of the potential sentence(s). Accordingly, the processing herein is an optimization that creates a sentence boundary optimized output.

Claims

1. A computer system comprising:

a processing unit in communication with a memory;

a functional unit in communication with the processing unit having tools for natural language processing, the tools to;

determine optimal sentence boundary placement within a received string input comprising;

identify two or more preliminary sentence boundaries within the input;

identify two or more potential sentences within the input utilizing the two or more preliminary sentence boundaries;

assign a first score to each identified potential sentence wherein the assigned first score corresponds to a probability of the potential sentence being an actual sentence;

selectively compare each identified potential sentences based on a relationship to the assigned first score to a first potential sentence in a first category and a second potential sentence in a second category;

selectively categorize each compared potential sentence based on the comparison into one of the first and second categories; and

transform the input into a sentence optimized output including modify at least one potential sentence utilizing the input, categorization, and at least one preliminary sentence boundary.

2. The system of claim 1, wherein the transformation includes the tools to:

create a new potential sentence utilizing the at least one potential sentence and movement of the at least one preliminary sentence boundary.

3. The system of claim 2, further comprising the tools to:

assign a second score to the created new potential sentence;

determine the assigned second score of the created new potential sentence is greater than the assigned first score of the potential sentence utilized to create the created new potential sentence; and

replace the potential sentence utilized to create the created new potential sentence with the created new potential sentence.

4. The system of claim 2, wherein the categorization includes a third category corresponding to any potential sentence having a specified similarity to at least one of the first potential sentence and the second potential sentence.

5. The system of claim 1, wherein the transformation of a potential sentence assigned to the second category includes formatting the potential sentence assigned to the second category utilizing a formatting option selected from the group consisting of:

convert a table to a comma separated value file, convert a table to a tab separated value file, convert a figure to a text description of the figure, convert a binary image to a potential sentence, convert a list into a single line with a comma separating any value.

6. The system of claim 1, wherein the transformation of a sentence assigned to the first category includes the tools to merge the sentence assigned to the first category with an adjacent potential sentence.

7. A computer program product for natural language processing, the computer program product comprising a computer readable storage medium having program code embodied therewith, the program code executable by a processor to:

determine optimal sentence boundary placement within a received string input comprising;

identify two or more preliminary sentence boundaries within the input;

identify two or more potential sentences within the input utilizing the two or more preliminary sentence boundaries;

assign a first score to each identified potential sentence wherein the assigned first score corresponds to a probability of the potential sentence being an actual sentence;

selectively compare each identified potential sentences based on a relationship to the assigned first score to a first potential sentence in a first category and a second potential sentence in a second category;

selectively categorize each compared potential sentence based on the comparison into one of the first and second categories; and

transform the input into a sentence optimized output including modify at least one potential sentence utilizing the input, categorization, and at least one preliminary sentence boundary.

8. The computer program product of claim 7, wherein the transformation includes program code to:

create a new potential sentence utilizing the at least one potential sentence and movement of the at least one preliminary sentence boundary.

9. The computer program product of claim 8, further comprising program code to:

assign a second score to the created new potential sentence;

determine the assigned second score of the created new potential sentence is greater than the assigned first score of the potential sentence utilized to create the created new potential sentence; and

replace the potential sentence utilized to create the created new potential sentence with the created new potential sentence.

10. The computer program product of claim 8, wherein the categorization includes a third category corresponding to any potential sentence having a specified similarity to at least one of the first potential sentence and the second potential sentence.

11. The computer program product of claim 7, wherein the transformation of a potential sentence assigned to the second category includes formatting the potential sentence assigned to the second category utilizing a formatting option selected from the group consisting of:

convert a table to a comma separated value file, convert a table to a tab separated value file, convert a figure to a text description of the figure, convert a binary image to a potential sentence, convert a list into a single line with a comma separating any value.

12. The computer program product of claim 7, wherein the transformation of a sentence assigned to the first category includes program code to merge the sentence assigned to the first category with an adjacent potential sentence.

13. A method for natural language processing comprising:

determining optimal sentence boundary placement within a received string input comprising;

identifying two or more preliminary sentence boundaries within the input;

identifying two or more potential sentences within the input utilizing the two or more preliminary sentence boundaries;

assigning a first score to each identified potential sentence wherein the assigned first score corresponds to a probability of the potential sentence being an actual sentence;

selectively comparing each identified potential sentences based on a relationship to the assigned first score to a first potential sentence in a first category and a second potential sentence in a second category;

selectively categorizing each compared potential sentence based on the comparison into one of the first and second categories; and

transforming the input into a sentence optimized output including modifying at least one potential sentence utilizing the input, categorization, and at least one preliminary sentence boundary.

14. The method of claim 13, wherein the transforming includes:

creating a new potential sentence utilizing the at least one potential sentence and movement of at least one preliminary sentence boundary.

15. The method of claim 14, further comprising:

assigning a second score to the created new potential sentence;

determining the assigned second score of the created new potential sentence is greater than the assigned first score of the potential sentence utilized to create the created new potential sentence; and

replacing the potential sentence utilized to create the created new potential sentence with the created new potential sentence.

16. The method of claim 13, wherein the categorizing further comprising a third category corresponding to any potential sentence having a specified similarity to at least one of the first potential sentence and the second potential sentence.

17. The method of claim 13, wherein the transforming of a potential sentence assigned to the second category includes formatting the potential sentence assigned to the second category utilizing a formatting option selected from the group consisting of:

converting a table to a comma separated value file, converting a table to a tab separated value file, converting a figure to a text description of the figure, converting a binary image to a potential sentence, converting a list into a single line with a comma separating any value.

18. The method of claim 13, wherein the transforming of a sentence assigned to the first category includes merging the sentence assigned to the first category with an adjacent potential sentence.