METHODS FOR OBTAINING IMPROVED TEXT SIMILARITY MEASURES

• US 20090125805A1
• Filed: 11/09/2007
• Published: 05/14/2009
• Est. Priority Date: 11/09/2007
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Ananthanarayanan, Rema, Reinwald, Berthold, Balakrishnan, Sreeram V., Lo, Yuen Y.
• Granted Patent: US 7,945,525 B2
• Time in Patent Office: Days
• US Class Current: 715/255
• CPC Class Codes: G06F 40/194 Calculation of difference between files
• CPC Class Codes: G06F 40/247 Thesauruses; Synonyms
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates

Abstract

The embodiments of the invention provide methods for obtaining improved text similarity measures. More specifically, a method of measuring similarity between at least two electronic documents begins by identifying similar terms between the electronic documents. This includes basing similarity between the similar terms on patterns, wherein the patterns can include word patterns, letter patterns, numeric patterns, and/or alphanumeric patterns. The identifying of the similar terms also includes identifying multiple pattern types between the electronic documents. Moreover, the basing of the similarity on patterns identifies terms within the electronic documents that are within a category of a hierarchy. Specifically, the identifying of the terms reviews a hierarchical data tree, wherein nodes of the tree represent terms within the electronic documents. Lower nodes of the tree have specific terms; and, wherein higher nodes of the tree have general terms.

Claims

1. A method for data integration between at least two electronic documents, said method comprising:

identifying similar terms between said electronic documents, said identifying comprising basing similarity between said similar terms on patterns;

replacing said similar terms with an equivalent term to create transformed electronic documents;

performing a similarity comparison on said transformed electronic documents, comprising identifying transformed electronic documents comprising said equivalent term; and

outputting a unified view to a user, such that said unified view comprises said transformed electronic documents comprising said equivalent term.

2. The method according to claim 1, all the limitations of which are incorporated herein by reference, wherein said identifying of said similar terms between said electronic documents comprises identifying multiple pattern types between said electronic documents.

3. The method according to claim 1, all the limitations of which are incorporated herein by reference, wherein said basing of said similarity on patterns comprises identifying terms within said electronic documents that are within a category of a hierarchy.

4. The method according to claim 3, all the limitations of which are incorporated herein by reference, wherein said replacing of said similar terms comprises replacing said terms with a generic domain name describing said category.

5. The method according to claim 3, all the limitations of which are incorporated herein by reference, further comprising maintaining an index of said terms.

6. The method according to claim 3, all the limitations of which are incorporated herein by reference, wherein said identifying of said terms comprises reviewing a hierarchical data tree, wherein nodes of said tree represent terms within said electronic documents, wherein lower nodes of said tree comprise specific terms, and wherein higher nodes of said tree comprise general terms.

7. The method according to claim 6, all the limitations of which are incorporated herein by reference, wherein said replacing of said similar terms comprises replacing a term represented by a node within said tree with a different term represented by a higher node within said tree.

8. A method for data integration between at least two electronic documents, said method comprising:

identifying similar terms between said electronic documents, said identifying comprising basing similarity between said similar terms on patterns, wherein said patterns comprise at least one of word patterns, letter patterns, numeric patterns, and alphanumeric patterns;

replacing said similar terms with an equivalent term to create transformed electronic documents;

performing a similarity comparison on said transformed electronic documents, comprising identifying transformed electronic documents comprising said equivalent term; and

outputting a unified view to a user, such that said unified view comprises said transformed electronic documents comprising said equivalent terms.

9. The method according to claim 8, all the limitations of which are incorporated herein by reference, wherein said identifying of said similar terms between said electronic documents comprises identifying multiple pattern types between said electronic documents.

10. The method according to claim 8, all the limitations of which are incorporated herein by reference, wherein said basing of said similarity on patterns comprises identifying terms within said electronic documents that are within a category of a hierarchy.

11. The method according to claim 10, all the limitations of which are incorporated herein by reference, wherein said replacing of said similar terms comprises replacing said terms with a generic domain name describing said category.

12. The method according to claim 10, all the limitations of which are incorporated herein by reference, further comprising maintaining an index of said terms.

13. The method according to claim 10, all the limitations of which are incorporated herein by reference, wherein said identifying of said terms comprises reviewing a hierarchical data tree, wherein nodes of said tree represent terms within said electronic documents, wherein lower nodes of said tree comprise specific terms, and wherein higher nodes of said tree comprise general terms.

14. The method according to claim 13, all the limitations of which are incorporated herein by reference, wherein said replacing of said similar terms comprises replacing a term represented by a node within said tree with a different term represented by a higher node within said tree.

15. A method for data integration between at least two electronic documents, said method comprising:

identifying similar terms between said electronic documents, said identifying comprising basing similarity between said similar terms on patterns, said basing of said similarity on patterns comprising identifying terms within said electronic documents that are within a category of a hierarchy, wherein said patterns comprise at least one of word patterns, letter patterns, numeric patterns, and alphanumeric patterns;

replacing said similar terms with equivalent terms to create transformed electronic documents, said replacing of said similar terms comprising replacing said terms with a generic domain name describing said category;

performing a similarity comparison on said transformed electronic documents, comprising identifying transformed electronic documents comprising said equivalent terms; and

outputting a unified view to a user, such that said unified view comprises said transformed electronic documents comprising said equivalent terms.

16. The method according to claim 15, all the limitations of which are incorporated herein by reference, wherein said identifying of said similar terms between said electronic documents comprises identifying multiple pattern types between said electronic documents.

17. The method according to claim 15, all the limitations of which are incorporated herein by reference, further comprising maintaining an index of said terms.

18. The method according to claim 15, all the limitations of which are incorporated herein by reference, wherein said identifying of said terms comprises reviewing a hierarchical data tree, wherein nodes of said tree represent terms within said electronic documents, wherein lower nodes of said tree comprise specific terms, and wherein higher nodes of said tree comprise general terms.

19. The method according to claim 18, all the limitations of which are incorporated herein by reference, wherein said replacing of said similar terms comprises replacing a term represented by a node within said tree with a different term represented by a higher node within said tree.

20. A computer program product comprising computer readable program code stored on computer readable storage medium embodied therein for performing a method of measuring similarity between at least two electronic documents, said method comprising:

identifying similar terms between said electronic documents, said identifying comprising basing similarity between said similar terms on patterns;

replacing said similar terms with an equivalent term to create transformed electronic documents;

performing a similarity comparison on said transformed electronic documents, comprising identifying transformed electronic documents comprising said equivalent term; and

outputting a unified view to a user, such that said unified view comprises said transformed electronic documents comprising said equivalent term.