Systems and methods for semantic search, content correlation and visualization

• US 9,489,350 B2
• Filed: 04/29/2011
• Issued: 11/08/2016
• Est. Priority Date: 04/30/2010
• Status: Active Grant

• Current Assignee: Contiem Incorporated
• Original Assignee: Orbis Technology
• Inventors: Crochet, Larry, Niv, Michael
• Primary Examiner(s): Sirjani, Fariba
• Application Number:
  US13/097,746
• Publication Number:
  US 20110270606A1
• Time in Patent Office: 2,020 Days
• Field of Search: 704 1- 10
• US Class Current: 1/1
• CPC Class Codes: G06F 40/194 Calculation of difference between files

Abstract

Methods and systems for searching over large (i.e., Internet scale) data to discover relevant information artifacts based on similar content and/or relationships are disclosed. Improvements over simple keyword and phrase based searching over internet scale data are shown. Search engines providing accurate and contextually relevant search results are disclosed. Users are enabled to identify related documents and information artifacts and quickly, ascertain, via visualization, which of these documents are original, which are derived (or copied) from a source document or information artifact, and which subset is independently generated (i.e., an original document or information artifact).

Claims

1. A computer-implemented method for comparing content overlap between a first electronic document and a second electronic document, comprising:

receiving, at a computer, a search request from a user for documents containing one or more keywords;

using the computer to access an electronic database and present to the user a first list of one or more documents in the electronic database based on the one or more keywords, the first list of one or more documents including a first hyperlink for the first electronic document;

receiving at the computer a request for the first electronic document via the first hyperlink;

determining, in response to the request for the first electronic document, a second list of documents in the electronic database that are similar to the first electronic document, the second list of documents including the second electronic document, and the determining comprising;

using the computer to parse a text of each of the first and second documents into constituent units;

using the computer to compute a digest of each of the first and second documents based on the constituent units;

using the computer to compare the computed digests;

using the computer to compute a proportion of common contents between the first and second documents and a proportion of distinct contents between the first and second documents based on the comparison;

using the computer to determine a date associated with the first document and a date associated with the second document; and

using the computer to determine a direction of borrowing based on the determined dates; and

using the computer to display to the user the contents of the first electronic document and a hyperlink to the second electronic document and a graphic indicating the direction of borrowing between the first document and the second document, wherein the graphic includes an arrow oriented to point in the borrowing direction showing a computed direction of flow of the information from a donor document to a borrower document, and wherein the graphic comprises a measure of relationship overlap between the first document and at least one of the second document and a selected portion of the second document.

2. The method of claim 1, wherein the graphic further comprises an indication of the proportion of common contents and an indication of the proportion of distinct contents.

20. The method of claim 1, further comprising:

using the computer to determine a number of original mentions in the first and second documents based on the comparison.

21. The method of claim 1, wherein the graphic further comprises both the measure of relationship overlap between the first document and the second document and the measure of relationship overlap between the first document and a selected portion of the second document.

22. The method of claim 21, wherein the graphic further comprises:

a measure of a size of the first document, a measure of a size of the second document; and

a measure of textual overlap between the first document and the second document.

23. The method of claim 22, wherein at least one of the measure of textual overlap between the first document and the second document, the measure of relationship overlap between the first document and the second document, and the measure of relationship overlap between the first document and the selected portion of the second document is displayed as a colored bar having a length corresponding to a magnitude of the respective measure.

24. The method of claim 1, wherein the step of using a computer to parse the text of each of the first and second documents into constituent units further comprises using the computer to parse the text of each of the first and second documents into sentences.

3. A computer-implemented method for comparing content overlap between a target electronic document and at least one additional electronic document, comprising:

receiving, at a computer, a search request from a user for documents containing one or more keywords;

using the computer to access an electronic database and present to the user a first list of one or more documents in the electronic database based on the one or more keywords, the first list of one or more documents including a first hyperlink for the first electronic document;

receiving at the computer a request for the first electronic document via the first hyperlink;

determining, in response to the request for the first electronic document, a second list of documents in the electronic database that are similar to the first electronic document, the second list of documents including the second electronic document, and the determining comprising;

using the computer to parse a text of each of the target and at least one additional document into constituent units;

using the computer to identify named entities within each of the constituent units;

using the computer to pair identified named entities which appear together within the same constituent units; and

using the computer to assess a similarity between the target document and the at least one additional document based on a result of the paired entities; and

using the computer to display to the user the contents of the first electronic document and a hyperlink to the second electronic document and a graphic indicating the similarity between the target document and the at least one additional document with an indication of similarity,

wherein the graphic comprises;

a measure of relationship overlap between the target document and the at least one additional document;

a measure of relationship overlap between the target document and a selected portion of the at least one additional document;

a measure of a size of the target document;

a measure of a size of the at least one additional document; and

a measure of a size of the textual overlap between the target document and the at least one additional document,

wherein at least two of the measure of textual overlap between the target document and the at least one additional document;

the measure of relationship overlap between the target document and the at least one additional document; and

the measure of relationship overlap between the target document and a selected portion of the at least one additional document are displayed as bars having a length corresponding to a magnitude of the respective measure.

4. The method of claim 3, further comprising using the computer to determine a date associated with the target document and a date associated with the at least one additional document, and using the computer to determine a direction of borrowing based on the determined dates.

5. The method of claim 3, wherein the step of using the computer to assess further comprises using the computer to calculate a similarity score by applying a Term Frequency-Inverse Document Frequency (TF-IDF) formula to the paired entities.

6. The method of claim 3, wherein the step of using the computer to assess further comprises using the computer to calculate a first similarity score by applying a Term Frequency-Inverse Document Frequency (TF-IDF) formula to the paired entities for all text, and using the computer to calculate a second similarity score by applying a TF-IDF formula to the paired entities only for text that is determined to be original to the target document.

26. The method of claim 3, wherein the indication of similarity comprises one or more indicators, the indicators comprising:

an indication of like semantic pairs between the target document and the at least one additional document and an indication of the presence of original material in the target document, relative to the at least one additional document, for one or more of the identified named entities and/or the pair of named entities that appear together within the same constituent unit.

27. The method of claim 26, wherein the indication of the presence of original material is displayed as an asterisk.

28. The method of claim 26, wherein the indication of like semantic pairs is displayed as a circle.

29. The method of claim 3, further comprising the steps of:

using the computer to compute a digest of each of the target document and the at least one additional document based on the constituent units;

using the computer to compare the digest corresponding to the target document to the digest corresponding to the at least one additional document; and

using a result of the comparing to determine which text in the target document is original to the target document.

30. The method of claim 29, wherein the step of using a computer to parse the text of each of the target and at least one additional documents into constituent units further comprises using the computer to parse the text of each of the target and at least one additional documents into sentences.

31. The method of claim 30, wherein the step of using the computer to compute a digest of each of the target document and the at least one additional document further comprises using the computer to determine a number of sentences contained in each of the target and at least one additional documents and determining a sentence signature for each of the sentences, and wherein the step of using the computer to compare the computed digests further comprises using the sentence signatures and the respective numbers of sentences contained in each of the target and at least one additional documents to perform the comparison.

7. A system for comparing content overlap between electronic documents, the system comprising:

a server node in electronic communication with a user interface node and one or more electronic databases, the server node being configured to;

receive a search request from a user for documents containing one or more keywords;

access the one or more electronic databases and present to the user a first list of one or more documents in the one or more electronic databases based on the one or more keywords, the first list of one or more documents including a first hyperlink for the first electronic document;

receive from the user a request for the first electronic document via the first hyperlink;

determine, in response to the request for the first electronic document, a second list of documents in the one or more electronic databases that are similar to the first electronic document, the second list of documents including the second electronic document, and wherein to determine the second list of documents the server node is configured to;

parse a text of each of the first and second documents into constituent units;

compute a digest of each of the first and second documents,

compare the computed digests;

compute a proportion of common contents between the first and second documents and a proportion of distinct contents between the first and second documents based on the comparison;

determine a date associated with the first document and a date associated with the second document;

determine a direction of borrowing based on the determined dates; and

display to the user the contents of the first electronic document and a hyperlink to the second electronic document and a graphic indicating the direction of borrowing between the first and second documents, wherein the indication of borrowing direction between the first document and the second document is displayed as an arrow oriented to point in the borrowing direction showing a computed direction of flow of the information from a donor document to a borrower document, and wherein the graphic comprises a measure of relationship overlap between the first document and at least one of the second document and a selected portion of the second document.

8. The system of claim 7, wherein the graphic further comprises an indication of the proportion of common contents and an indication of the proportion of distinct contents.

9. A system for comparing content overlap between electronic documents, comprising:

a server node in electronic communication with a user interface node and one or more electronic databases, the server node being configured to;

receive a search request from a user for documents containing one or more keywords;

access the one or more electronic databases and present to the user a first list of one or more documents in the one or more electronic databases based on the one or more keywords, the first list of one or more documents including a first hyperlink for the first electronic document;

receive from the user a request for the first electronic document via the first hyperlink;

determine, in response to the request for the first electronic document, a second list of documents in the one or more electronic databases that are similar to the first electronic document, the second list of documents including the second electronic document, and wherein to determine the second list of documents the server node is configured to;

parse a text of each of the target and at least one additional document into constituent units;

identify named entities within each of the constituent units;

pair identified named entities which appear together within the same constituent units;

assess a similarity between the target document and the at least one additional document based on a result of the paired entities; and

,

display to the user the contents of the first electronic document and a hyperlink to the second electronic document and a graphic indicating the similarity between the target document and the at least one additional document with an indication of similarity,

wherein the graphic comprises;

a measure of relationship overlap between the target document and the at least one additional document;

a measure of relationship overlap between the target document and a selected portion of the at least one additional document;

a measure of a size of the target document;

a measure of a size of the at least one additional document; and

a measure of a size of the textual overlap between the target document and the at least one additional document,

wherein at least two of the measure of textual overlap between the target document and the at least one additional document;

the measure of relationship overlap between the target document and the at least one additional document; and

the measure of relationship overlap between the target document and a selected portion of the at least one additional document are displayed as bars having a length corresponding to a magnitude of the respective measure.

10. The system of claim 9, wherein the server node is further configured to determine a date associated with the target document and a date associated with the at least one additional document, and to determine a direction of borrowing based on the determined dates.

11. The system of claim 9, wherein the server node is further configured to calculate a similarity score relating to the target document and the at least one additional document by applying a Term Frequency-Inverse Document Frequency (TF-IDF) formula to the paired entities.

12. The system of claim 9, wherein the server node is further configured to calculate a first similarity score by applying a Term Frequency-Inverse Document Frequency (TF-IDF) formula to the paired entities for all text, and to calculate a second similarity score by applying a TF-IDF formula to the paired entities only for text that is determined to be original to the target document.

13. A computer program product for comparing content overlap between a first electronic document and a second electronic document, the computer program product comprising a non-transitory computer readable medium storing computer readable program code, the computer readable program code comprising:

a set of instructions for receiving a search request from a user for documents containing one or more keywords;

a set of instructions for accessing an electronic database and presenting to the user a first list of one or more documents in the electronic database based on the one or more keywords, the first list of one or more documents including a first hyperlink for the first electronic document;

a set of instructions for receiving a request for the first electronic document via the first hyperlink;

a set of instructions for determining, in response to the request for the first electronic document, a second list of documents in the electronic database that are similar to the first electronic document, the second list of documents including the second electronic document, and the set of instructions for determining comprising;

a set of instructions for parsing a text of each of the first and second documents into constituent units;

a set of instructions for computing a digest of each of the first and second documents based on the constituent units;

a set of instructions for comparing the computed digests;

a set of instructions for computing a proportion of common contents between the first and second documents and a proportion of distinct contents between the first and second documents based on the comparison; and

a set of instructions for determining a date associated with the first document and a date associated with the second document, and a set of instructions for determining a direction of borrowing based on the determined dates; and

a set of instructions for displaying to the user the contents of the first electronic document and a hyperlink to the second electronic document and a graphic indicating the direction of borrowing between the first and second documents, wherein the indication of borrowing direction between the first document and the second document is displayed as an arrow oriented to point in the borrowing direction showing a computed direction of flow of the information from a donor document to a borrower document, and wherein the graphic comprises a measure of relationship overlap between the first document and at least one of the second document and a selected portion of the second document.

14. The computer program product of claim 13, wherein the computer readable program code further comprises a set of instructions for creating a graphic to indicate the proportion of common contents and to indicate the proportion of distinct contents.

15. The computer program product of claim 13, wherein the computer readable program code further comprises a set of instructions for creating a graphic to indicate the proportion of common contents, the proportion of distinct contents, a length of the first document, and a length of the second document.

16. A computer program product for comparing content overlap between a target electronic document and at least one additional electronic document, the computer program product comprising a non-transitory computer readable medium storing computer readable program code, the computer readable program code comprising:

a set of instructions for receiving a search request from a user for documents containing one or more keywords;

a set of instructions for accessing an electronic database and presenting to the user a first list of one or more documents in the electronic database based on the one or more keywords, the first list of one or more documents including a first hyperlink for the target document;

a set of instructions for receiving a request for the target document via the first hyperlink;

a set of instructions for determining, in response to the request for the target document, a second list of documents in the electronic database that are similar to the target document, the second list of documents including at least one additional electronic document, and the set of instructions for determining comprising;

a set of instructions for parsing a text of each of the target and the at least one additional document into constituent units;

a set of instructions for identifying named entities within each of the constituent units;

a set of instructions for pairing identified named entities which appear together within the same constituent units;

a set of instructions for assessing a similarity between the target document and the at least one additional document based on a result of the paired entities; and

,

a set of instructions for displaying to the user the contents of the first electronic document and a hyperlink to the second electronic document and a graphic indicating the similarity between the target document and the at least one additional document with an indication of similarity,

wherein the graphic comprises;

a measure of relationship overlap between the target document and the at least one additional document,

a measure of relationship overlap between the target document and a selected portion of the at least one additional document;

a measure of a size of the target document;

a measure of a size of the at least one additional document; and

a measure of a size of the textual overlap between the target document and the at least one additional document,

wherein at least two of the measure of textual overlap between the target document and the at least one additional document;

the measure of relationship overlap between the target document and the at least one additional document; and

the measure of relationship overlap between the target document and a selected portion of the at least one additional document are displayed as bars having a length corresponding to a magnitude of the respective measure.

17. The computer program product of claim 16, wherein the computer readable program code further comprises a set of instructions for determining a date associated with the target document and a date associated with the at least one additional document, and a set of instructions for determining a direction of borrowing based on the determined dates.

18. The computer program product of claim 16, wherein the computer readable program code further comprises a set of instructions for calculating a similarity score by applying a Term Frequency-Inverse Document Frequency (TF-IDF) formula to the paired entities.

19. The computer program product of claim 16, wherein the computer readable program code further comprises a set of instructions for calculating a first similarity score by applying a Term Frequency-Inverse Document Frequency (TF-IDF) formula to the paired entities for all text, and a set of instructions for calculating a second similarity score by applying a TF-IDF formula to the paired entities only for text that is determined to be original to the target document.

25. The method of claim a 24, wherein the step of using the computer to compute a digest of each of the first and second documents further comprises using the computer to determine a number of sentences contained in each of the first and second documents and determining a sentence signature for each of the sentences, and wherein the step of using the computer to compare the computed digests further comprises using the sentence signatures and the respective numbers of sentences contained in each of the first and second documents to perform the comparison.