Method and system for mining a document containing dirty text

• US 6,978,275 B2
• Filed: 08/31/2001
• Issued: 12/20/2005
• Est. Priority Date: 08/31/2001
• Status: Expired due to Fees

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Stinger, James R., Castellanos, Maria
• Primary Examiner(s): AMSBURY, WAYNE P
• Application Number:
  US09/944,919
• Publication Number:
  US 20030046263A1
• Time in Patent Office: 1,572 Days
• Field of Search: 707/6, 707/102, 707/2
• US Class Current: 1/1
• CPC Class Codes: G06F 16/345 Summarisation for human users
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

A method and system for mining a document containing dirty text. Dirty text is removed or replaced and the document is processed using a variety of text mining techniques. In one embodiment, dirty text removal and replacement occurs in two stages. In the first stage, a general cleaning occurs on all documents without regard to what domain they belong to or the mining task to be performed. In the second stage, document cleaning is more specific to the anomalies of the domain and the mining task to be performed. In the third stage, the document is processed using a variety of data mining techniques according to the mining task. In one embodiment, the present invention scores and ranks sentences in a document according to their relevance, extracts the highest ranked sentences, and presents a summary. The present invention allows users to leverage existing domain knowledge and can be customized according the domain and task requirements.

Claims

1. A computer-implemented method for mining a document containing dirty text comprising:

removing an instance of dirty text within said document to produce a cleaned document having a content; and

performing a data mining operation on said cleaned document thereby deriving relevant information from said cleaned document and providing a summary of the content of said document, and scoring and ranking each sentence of said document, wherein said removing further comprises removing an instance of computer code from said document, and removing a table from said document.

2. The method for mining a document containing dirty text as recited in claim 1, wherein said removing further comprises replacing an instance of dirty text with a standard term.

3. The method for mining a document containing dirty text as recited in claim 1, wherein said performing a data mining operation further comprises identifying a sentence within said cleaned document by identifying a beginning and an end of said sentence.

4. The method for mining a document containing dirty text as recited in claim 3, wherein said performing a data mining operation further comprises scoring and ranking said sentence.

5. The method for mining a document containing dirty text as recited in claim 4, wherein said method further comprises generating a summary derived from said scored and ranked sentences.

6. The method for mining a document containing dirty text as recited in claim 4, wherein said method further comprises selecting a text mining component based upon said data mining operation to be performed.

7. The method for mining a document containing dirty text as recited in claim 4, wherein said method further comprises customizing said method by adjusting a parameter value.

8. A computer system comprising:

a bus;

a memory unit coupled to said bus; and

a processor coupled to said bus, said processor for executing a method for mining a document containing dirty text comprising;

producing a cleaned document having a content comprising performing a general cleaning of said document by removing an instance of dirty text within said document including instances of misspelling and grammatical errors, and performing a domain and task specific cleaning of said document including removing instances of computer code and tables to produce a cleaned document; and

performing a data mining operation on said cleaned document including providing a summary of the content of said document including scoring and ranking each sentence.

10. The computer system as recited in claim 8, wherein said performing a data mining operation further comprises identifying a sentence within said cleaned document by identifying a beginning and an end of said sentence.

11. A computer system comprising:

a bus;

a memory unit coupled to said bus; and

a processor coupled to said bus, said processor for executing a method for mining a document containing dirty text comprising;

producing a cleaned document having a content comprising performing a general cleaning of said document by removing an instance of dirty text within said document including instances of misspelling and grammatical errors, and performing a domain and task specific cleaning of said document including removing instances of computer code and tables to produce a cleaned document; and

performing a data mining operation on said cleaned document including providing a summary of the content of said document, wherein said performing a data mining operation further comprises identifying a sentence within said cleaned document by identifying a beginning and an end of said sentence, wherein said performing a data mining operation further comprises scoring and ranking said sentence and wherein scoring said sentence further comprises;

selecting scoring techniques operable for summarizing non-narrative, grammatically incorrect text;

selecting scoring techniques operable for summarizing narrative, grammatically correct text; and

using said scoring techniques to score said sentence.

9. The computer system as recited in claim 11, wherein said removing further comprises replacing an instance of dirty text with a standard term.

12. The computer system as recited in claim 11, wherein said method further comprises generating the summary derived from said scored and ranked sentences.

13. The computer system as recited in claim 11, wherein said method further comprises selecting a text mining component based upon said data mining operation to be performed.

14. The computer system as recited in claim 11, wherein said method further comprises customizing said method by adjusting a parameter value.

15. A computer-useable medium having computer-readable program code embodied therein for causing a computer system to perform the steps of:

removing an instance of dirty text within said document to produce a cleaned document having a content; and

performing a data mining operation on said cleaned document to provide a summary of said content, removing an instance of computer code from said document and removing a table from said document, and scoring and ranking each sentence.

16. The computer-useable medium of claim 15, wherein said removing further comprises replacing an instance of dirty text with a standard term.

17. The computer-useable medium recited in claim 15, wherein said performing a data mining operation further comprises identifying a sentence within said cleaned document by identifying a beginning and an end of said sentence.

18. A computer-useable medium having computer-readable program code embodied therein for causing a computer system to perform the steps of:

removing an instance of dirty text within said document to produce a cleaned document having a content; and

performing a data mining operation on said cleaned document to provide a summary of said content, wherein said performing a data mining operation further comprises identifying a sentence within said cleaned document by identifying a beginning and an end of said sentence, wherein said performing a data mining operation further comprises, wherein scoring said sentence further comprises;

selecting scoring techniques operable for summarizing non-narrative, grammatically incorrect text;

selecting scoring techniques operable for summarizing narrative, grammatically correct text; and

using said scoring techniques to score said sentence.

19. The computer-useable medium recited in claim 18, wherein said method further comprises generating a summary derived from said scored and ranked sentences.

20. The computer-useable medium as recited in claim 18, wherein said method further comprises selecting a text mining component based upon said data mining operation to be performed.

21. The computer-useable medium as recited in claim 18, wherein said method further comprises customizing said method by adjusting a parameter value.

22. A computer-implemented method for mining a document containing dirty text comprising:

producing a cleaned document having a content comprising performing a general cleaning of said document by removing one or more instance of dirty text within said document including instances of misspelling and grammatical errors, and performing a domain and task specific cleaning of said document including removing instances of computer code and tables; and

performing a data mining operation on said cleaned document, including determining a sentence score for each sentence of said cleaned document and ranking the sentences from highest to lowest based on the sentence score;

generating a summary of the content of the document using the highest ranked sentences.

23. The method of claim 22, wherein determining a sentence score for each sentence includes applying a keyword technique to each sentence.

24. The method of claim 23, wherein determining a sentence score further comprises applying a location technique to each sentence.

25. The method of claim 23, wherein determining a sentence score further comprises applying a semantic similarity technique to each sentence.

26. The method of claim 25, wherein the semantic similarity technique comprises:

generating a vector associated with each sentence; and

comparing each vector to every other vector, including defining a cosine of an angle between two vectors and using the cosine of the angle between two vectors to determine whether sentences represented by the two vectors are semantically related.