Method and system for mining a document containing dirty text

• US 20030046263A1
• Filed: 08/31/2001
• Published: 03/06/2003
• Est. Priority Date: 08/31/2001
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Stinger, James R., Castellanos, Maria
• Granted Patent: US 6,978,275 B2
• Time in Patent Office: Days
• US Class Current: 707/1
• CPC Class Codes: G06F 16/345 Summarisation for human users
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

A method and system for mining a document containing dirty text. Dirty text is removed or replaced and the document is processed using a variety of text mining techniques. In one embodiment, dirty text removal and replacement occurs in two stages. In the first stage, a general cleaning occurs on all documents without regard to what domain they belong to or the mining task to be performed. In the second stage, document cleaning is more specific to the anomalies of the domain and the mining task to be performed. In the third stage, the document is processed using a variety of data mining techniques according to the mining task. In one embodiment, the present invention scores and ranks sentences in a document according to their relevance, extracts the highest ranked sentences, and presents a summary. The present invention allows users to leverage existing domain knowledge and can be customized according the domain and task requirements.

Claims

1. A method for mining a document containing dirty text comprising:

removing an instance of dirty text within said document to produce a cleaned document; and

performing a data mining operation on said cleaned document.

2. The method for mining a document containing dirty text as recited in claim 1, wherein said removing further comprises replacing an instance of dirty text with a standard term.

3. The method for mining a document containing dirty text as recited in claim 1, wherein said removing further comprises removing an instance of computer code from said document.

4. The method for mining a document containing dirty text as recited in claim 1, wherein said removing further comprises removing a table from said document.

5. The method for mining a document containing dirty text as recited in claim 1, wherein said performing a data mining operation further comprises identifying a sentence within said cleaned document by identifying a beginning and an end of said sentence.

6. The method for mining a document containing dirty text as recited in claim 5, wherein said performing a data mining operation further comprises scoring and ranking said sentence.

7. The method for mining a document containing dirty text as recited in claim 6, wherein scoring said sentence further comprises:

selecting scoring techniques operable for summarizing non-narrative, grammatically incorrect text;

selecting scoring techniques operable for summarizing narrative, grammatically correct text; and

using said scoring techniques to score said sentence.

8. The method for mining a document containing dirty text as recited in claim 7, wherein said method further comprises generating a summary derived from said scored and ranked sentences.

9. The method for mining a document containing dirty text as recited in claim 1, wherein said method further comprises selecting a text mining component based upon said data mining operation to be performed.

10. The method for mining a document containing dirty text as recited in claim 1, wherein said method further comprises customizing said method by adjusting a parameter value.

11. A computer system comprising:

a bus;

a memory unit coupled to said bus; and

a processor coupled to said bus, said processor for executing a method for mining a document containing dirty text comprising;

removing an instance of dirty text within said document to produce a cleaned document; and

performing a data mining operation on said cleaned document.

12. The computer system as recited in claim 11, wherein said removing further comprises replacing an instance of dirty text with a standard term.

13. The computer system as recited in claim 11, wherein said removing further comprises removing an instance of computer code from said document.

14. The computer system as recited in claim 11, wherein said removing further comprises removing a table from said document.

15. The computer system as recited in claim 11, wherein said performing a data mining operation further comprises identifying a sentence within said cleaned document by identifying a beginning and an end of said sentence.

16. The computer system as recited in claim 15, wherein said performing a data mining operation further comprises scoring and ranking said sentence.

17. The computer system as recited in claim 16, wherein scoring said sentence further comprises:

selecting scoring techniques operable for summarizing non-narrative, grammatically incorrect text;

selecting scoring techniques operable for summarizing narrative, grammatically correct text; and

using said scoring techniques to score said sentence.

18. The computer system as recited in claim 17, wherein said method further comprises generating a summary derived from said scored and ranked sentences.

19. The computer system as recited in claim 11, wherein said method further comprises selecting a text mining component based upon said data mining operation to be performed.

20. The computer system as recited in claim 11, wherein said method further comprises customizing said method by adjusting a parameter value.

21. A computer-usable medium having computer-readable program code embodied therein for causing a computer system to perform the steps of:

removing an instance of dirty text within said document to produce a cleaned document; and

performing a data mining operation on said cleaned document.

22. The computer-usable medium of claim 21, wherein said removing further comprises replacing an instance of dirty text with a standard term.

23. The computer-usable medium recited in claim 21, wherein said removing further comprises removing an instance of computer code from said document.

24. The computer-usable medium recited in claim 21, wherein said removing further comprises removing a table from said document.

25. The computer-usable medium recited in claim 21, wherein said performing a data mining operation further comprises identifying a sentence within said cleaned document by identifying a beginning and an end of said sentence.

26. The computer-usable medium recited in claim 25, wherein said performing a data mining operation further comprises scoring and ranking said sentence.

27. The computer-usable medium recited in claim 26, wherein scoring said sentence further comprises:

selecting scoring techniques operable for summarizing non-narrative, grammatically incorrect text;

selecting scoring techniques operable for summarizing narrative, grammatically correct text; and

using said scoring techniques to score said sentence.

28. The computer-usable medium recited in claim 27, wherein said method further comprises generating a summary derived from said scored and ranked sentences.

29. The computer-usable medium as recited in claim 21, wherein said method further comprises selecting a text mining component based upon said data mining operation to be performed.

30. The computer-usable medium as recited in claim 21, wherein said method further comprises customizing said method by adjusting a parameter value.