Document categorisation system

• US 20060089924A1
• Filed: 09/25/2001
• Published: 04/27/2006
• Est. Priority Date: 09/25/2000
• Status: Active Grant

• Current Assignee: Telstra Corporation Limited (Telstra Group Ltd.)
• Original Assignee: Telstra Corporation Limited (Telstra Group Ltd.)
• Inventors: Kowalczyk, Adam, Raskutti, Bhavani
• Granted Patent: US 7,971,150 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification

Abstract

A document categorisation system, including a clusterer for generating clusters of related electronic documents based on features extracted from said documents, and a filter module for generating a filter on the basis of said clusters to categorise further documents received by said system. The system may include an editor for manually browsing and modifying the clusters. The categorisation of the documents is based on n-grams, which are used to determine significant features of the documents. The system includes a trend analyzer for determining trends of changing document categories over time, and for identifying novel clusters. The system may be implemented as a plug-in module for a spreadsheet application, providing a convenient means for one-off or ongoing analysis of text entries in a worksheet.

Claims

1. A document categorisation system including:

a clusterer for generating clusters of related electronic documents based on features extracted from said documents; and

a filter module for generating a filter on the basis of said clusters to categorise further documents received by said system.

3. A document categorisation system as claimed any one of claims 1 and 2, wherein said clusterer is adapted to extract features from electronic documents, determine significant features from said extracted features, and generate clusters of said documents based on said significant features.

4. A document categorisation system as claimed in any one of the preceding claims, wherein the clusterer further includes a cluster describer module for generating text describing each cluster.

6. A document categorisation system as claimed any one of the preceding claims, wherein said features include at least one of n-grams, words and phrases.

8. A document categorisation system as claimed in claim 1 or claim 7, further including an editor for adjusting said clusters.

9. A document categorisation system as claimed in any one of the preceding claims, further including a trend analyzer for determining trends of document categories over time.

2. A document categorisation system including:

a clusterer for generating clusters of related electronic documents based on features extracted from said documents; and

an editor for browsing and modifying said clusters.

5. A document categorisation system including:

an editor for browsing and modifying clustered documents; and

a filter module for generating a filter on the basis of features of said clusters to categorise further documents received by said system.

7. A document categorisation system including:

a clusterer for generating clusters of documents by executing unsupervised learning on said documents; and

a filter module for generating a filter to categorise received documents by executing supervised learning on said clusters.

10. A method for categorising documents, including creating categories for said documents based on feature extraction, where said features include at least one of n-grams, words and phrases.

12. A method for categorising documents, as claimed in claim 10 or claim 11, including selecting features of said documents based on respective discriminating abilities of the features.

13. A method for categorising documents, as claimed in claim 12, wherein each said discriminating ability is based on similarities for said documents with and without said feature.

18. A method as claimed in any one of claims 10 to 13, wherein said features are described by an n-gram extraction process.

20. A method as claimed in any one of claims 10 to 19, include determining a trend of a document category over time.

11. A method for categorising documents, including:

creating categories for said documents, based on feature extraction; and

manually modifying said categories with a category editor.

14. A method for categorising a document, including:

creating a document filter for a pre-existing document category by analysing pre-existing documents in said category; and

applying said filter to said document in order to determine whether said document belongs in said category.

15. A method as claimed in claim 14, including generating descriptive labels for large document sets.

16. A method as claimed in claim 14, wherein said filter is also used to produce descriptive labels for large document sets.

17. A method as claimed in claim 15, wherein said descriptive labels include at least one of phrases and sentences.

19. A method as claimed in any one of claims 14 to 17, wherein said filters are generated using features which are selected using an n-gram extraction process.

21. A data categorisation module for use with a spreadsheet application, said module including:

a cluster module for generating clusters of related data from data in a document of the spreadsheet application, based on extracted features of said data; and

a training module for generating a filter on the basis of said clusters to categorise further data.

22. A data categorisation module as claimed in claim 21, including a filtering module for categorising further data on the basis of said filter.

23. A data categorisation module as claimed in claim 21, wherein said data includes a plurality of entries in the document.

24. A data categorisation module as claimed in claim 23, wherein said entries include text data to be used for categorising said plurality of entries, and structured data.

25. A data categorisation module as claimed in claim 21, wherein said cluster module is adapted to generate a cluster identifier for identifying a cluster to which an entry of said data belongs, and a cluster size value for identifying the size of said cluster.

26. A data categorisation module as claimed in claim 25, wherein said cluster module is adapted to generate at least one category descriptor for said entry.

27. A data categorisation module as claimed in claim 21, wherein said cluster module generates a worksheet column for identifying a category of each entry of said data.

28. A data categorisation module as claimed in claim 27, wherein said cluster module generates a formatted version of data of an entry for indicating a category descriptor of said entry.

29. A data categorisation module as claimed in claim 21, wherein said data categorisation module includes a module for testing said filters by categorising training data on the basis of filters generated using said training data.

30. A data categorisation module as claimed in claim 21, wherein said data categorisation module includes labelling functions for generating a category identifier for an entry of said data.

31. A data categorisation module as claimed in claim 30, wherein said document is a worksheet and said functions include a labelling function for generating a plurality of category columns of said worksheet for identifying at least one category of an entry.

32. A data categorisation module as claimed in claim 22, wherein said filtering module generates a respective score for each category of an entry.

33. A data categorisation module as claimed in claim 32, wherein said filtering module generates an error for an entry if any one of said scores is inconsistent with a respective category identifier.

34. A data categorisation module as claimed in claim 32, wherein a score indicates that the corresponding entry belongs to a respective category if said score exceeds a pre-determined value.

35. A data categorisation module as claimed in claim 34, wherein the default value of said pre-determined value is zero.

36. A data categorisation module as claimed in claim 34, wherein scores exceeding said pre-determined value are formatted differently than scores less than said value.

37. A data categorisation module as claimed in claim 32, wherein a score may be used to calculate a probability that said entry belongs to the corresponding category.

38. A data categorisation module for use with a spreadsheet application, said module including a cluster module for generating clusters of related data from data in a document of the spreadsheet application, based on extracted features of said data.

39. A method of data categorisation in a spreadsheet application, including the steps of:

a cluster module for generating clusters of related data from data in a document of the spreadsheet application, based on extracted features of said data; and

a training module for generating a filter on the basis of said clusters to categorise further data.