Method and System for Detection of Authors

• US 20080256093A1
• Filed: 04/11/2007
• Published: 10/16/2008
• Est. Priority Date: 04/11/2007
• Status: Active Grant

• Current Assignee: X Corp. (f/k/a Twitter, Inc.) (X Holdings Corp.)
• Original Assignee: International Business Machines Corporation
• Inventors: Yogev, Sivan, Yom-Tov, Elad, Amitay, Einat
• Granted Patent: US 7,752,208 B2
• Time in Patent Office: Days
• US Class Current: 707/100
• CPC Class Codes: G06F 16/334 Query execution G06F16/335 takes precedence
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation

Abstract

A method and system are provided for detection of authors across different types of information sources such as across documents on the Web. The method includes obtaining a compression signature (303) for a document, and determining the similarity (304) between compression signatures of two or more documents. If the similarity is greater than a threshold measure (305), the two or more documents are considered to be by the same author. Scored pairs of documents are clustered (308) to provide a group of documents by the same author. The group of documents by the same author can be used for user profiling, noise reduction, contribution sizing, detecting fraudulent contributions, obtaining other search results by the same author, or mating a document with undisclosed authorship to a document of known author.

Claims

1. A method for detection of authors across different types of information sources, comprising:

providing a collection of documents from different types of information sources;

obtaining a compression signature for a document;

determining the similarity between compression signatures of two or more documents; and

determining that, if the similarity is greater than a threshold measure, the two or more documents are by the same author.

2. The method as claimed in claim 1, wherein determining the similarity between compression signatures uses a compression distance algorithm.

3. The method as claimed in claim 2, wherein the similarity between compression signatures of two documents is determined using a normalized compression distance which measures the improvement of compression of each document by using the information in the other document.

4. The method as claimed in claim 1, including grouping all sets of two or more documents with a similarity greater than a threshold measure into groups by author.

5. The method as claimed in claim 1, wherein the threshold is a dynamic threshold which is varied dynamically in dependence on the similarity results.

6. A method as claimed in claim 4, wherein the grouping is by clustering or chaining the documents.

7. The method as claimed in claim 6, wherein clustering the documents includes ranking documents by their similarity of compression signatures and clustering by document features.

8. The method as claimed in claim 1, including:

defining pairs of documents;

obtaining a compression signature for each document;

compressing the pair of documents together;

applying a compression distance algorithm; and

comparing the compression distance of the pair to a threshold distance.

9. The method as claimed in claim 8, wherein defining pairs of documents pairs documents with the same username in the uniform resource locator of the documents.

10. The method as claimed in claim 8, wherein defining pairs of documents pairs documents from resources with a known relationship.

11. The method as claimed in claim 1, wherein a compression signature for a document is obtained from stored cached versions of the document in a search engine.

12. The method as claimed in claim 1, wherein determining that documents are by the same author is used in one of the group of:

user profiling, noise reduction, sizing of contributions, detecting fraudulent contributions, obtaining other documents by the same author, matching a document from a known author to a document without authorship indication.

13. A computer program product stored on a computer readable storage medium for detection of document authors, comprising computer readable program code means for performing the steps of:

providing a collection of documents from different types of information sources;

obtaining a compression signature for a document;

determining the similarity between compression signatures of two or more documents; and

determining that, if the similarity is greater than a threshold measure, the two or more documents are by the same author.

14. A system for detection of authors across different types of information sources, comprising:

a collection of documents from different types of information sources;

means for obtaining a compression signature for a document;

means for determining the similarity between compression signatures of two or more documents; and

means for determining that, if the similarity is greater than a threshold measure, the two or more documents are by the same author.

15. The system as claimed in claim 14, including means for grouping all sets of two or more documents with a similarity greater than a threshold measure into groups by author.

16. The system as claimed in claim 14, wherein the means for obtaining a compression signature for a document uses a stored cached version of the document in a search engine.

17. The system as claimed in claim 14, wherein the collection of documents across different types of information sources are documents on the Web including:

user profiles, Web logs (blogs), personal tagging spaces, professional and personal Web pages, open content Web pages, forums, any other types of Web pages which an author could create or contribute to.

18. The system as claimed in claim 14, including analyzing means for analyzing a group of documents determined to be by the same author.

19. The system as claimed in claim 18, wherein the analyzing means is used for one of the group of:

user profiling, noise reduction, sizing of contributions, detecting fraudulent contributions, obtaining other documents by the same author, matching a document from a known author to a document without authorship indication.

20. The system as claimed in claim 14, wherein the system is a search engine for searching over the collection of documents, including means for providing a set of documents within a threshold compression distance of a document by a known author.