SYSTEM AND METHOD FOR COMPUTERIZED SEMANTIC PROCESSING OF ELECTRONIC DOCUMENTS INCLUDING THEMES

• US 20140207782A1
• Filed: 01/22/2014
• Published: 07/24/2014
• Est. Priority Date: 01/22/2013
• Status: Abandoned Application

• Current Assignee: Microsoft Israel Research And Development (2002) Ltd (Microsoft Corporation)
• Original Assignee: Equivio Ltd. (Microsoft Corporation)
• Inventors: RAVID, Yiftach
• Application Number:
  US14/161,159
• Publication Number:
  US 20140207782A1
• Time in Patent Office: Days
• US Class Current: 707/739
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/288 Entity relationship models
• CPC Class Codes: G06F 16/335 Filtering based on additional data, e.g. user or group profiles filtering in web context G06F16/9535, G06F16/9536

Abstract

System and method for computerized identification of themes in a large data set, the system comprising reducing the number of data set members in a large data set, using at least one computerized data set member pruning technique other than random selection; and using a computerized theme identification technique for identifying a plurality of themes in the reduced data set.

Claims

1. A method for computerized identification of themes in a large data set, the system comprising:

reducing the number of data set members in a large data set, using at least one computerized data set member pruning technique other than random selection; and

using a computerized theme identification technique for identifying a plurality of themes in the reduced data set.

2. A method according to claim 1 wherein said computerized data set member pruning technique comprises thinning out at least one document which passes a document similarity criterion relative to at least one other document not being thinned out, thereby to combat skewing as a result of over-influence of similar, hence over-represented, documents upon said theme identification technique.

3. A method according to claim 2 wherein said thinning out at least one document which passes a document similarity criterion comprises replacing a plurality of emails forming an email thread, with at least one inclusive email, thereby to thin out emails which are included in said inclusive email hence are deemed to pass the document similarity criterion with regard to said inclusive.

4. A method according to claim 2 wherein said thinning out at least one document which passes a document similarity criterion comprises identifying and discarding near-duplicates thereby to thin out at least one document which is deemed to pass the document similarity criterion with regard to a set of near-duplicates of said document, at least one of which is not being thinned out.

5. A method according to claim 1 wherein said computerized theme identification technique comprises topic modeling.

6. A method according to claim 5 wherein said topic modeling allows documents to have a plurality of topics.

9. A method according to claim 1 and also comprising:

facilitating theme-to-word flitting by retrieving and presenting to a user, words whose word score for at least one user-selected theme;

is high.

10. A method according to claim 1 and also comprising:

facilitating theme-to-document flitting for retrieving and presenting to a user, documents whose document score for at least one user-selected theme is high.

11. A method according to claim 1 and also comprising:

facilitating document-level browsing for retrieving and presenting to a user, documents whose distributions over the plurality of themes are similar to the distribution of a user-selected document over the plurality of themes.

12. A method according to claim 1 and also comprising:

facilitating word-level browsing for retrieving and presenting to a user, words whose distributions over the plurality of themes are similar to the distribution of a user-selected word over the plurality of themes.

13. A method according to claim 1 wherein the number of data set members in the large data set is further reduced subsequent to said using step and prior to a manual review process.

14. A method according to claim 1 wherein said reducing is effected using:

random selection; and

at least one computerized data set member pruning technique other than random selection.

15. A method according to claim 14 wherein said random selection is performed after said computerized data set member pruning technique.

16. A method according to claim 14 wherein said random selection is performed before said computerized data set member pruning technique.

17. A method according to claim 5 wherein said topic modeling which allows documents to have a plurality of topics comprises one of the following computerized techniques:

Latent Dirichlet allocation (LDA), PLSI, and Pachinko allocation.

18. A method according to claim 3 wherein said thinning out at least one document which passes a document similarity criterion comprises replacing a plurality of emails forming an email thread, with a single inclusive email.

19. A method according to claim 4 wherein said identifying and discarding near-duplicates is effected using Equivio Zoom near-duplicate functionality.

20. A method according to claim 9 and wherein said facilitating comprises retrieving and presenting to a user, only those words whose word score for at least one user-selected theme;

is high and which answer to at least one additional criterion.

7. A browsing system operative in conjunction with a stored representation of a multiplicity of electronic documents and their distribution over a plurality of themes, the system comprising:

theme-to-document flitting apparatus for retrieving and presenting to a user, documents whose document score for at least one user-selected theme;

is high; and

document-level browsing apparatus for retrieving and presenting to a user, documents whose distributions over the plurality of themes are similar to the distribution of a user-selected document over the plurality of themes.

8. A system according to claim 7 and also comprising:

theme-to-word flitting apparatus for retrieving and presenting to a user, words whose word score for at least one user-selected theme;

is high;

word-level browsing apparatus for retrieving and presenting to a user, words whose distributions over the plurality of themes are similar to the distribution of a user-selected word over the plurality of themes,

thereby to provide 3-tier browsing apparatus facilitating browsing at word, document and topic levels responsive to user-initiated flitting between the levels.