Reliability of Duplicate Document Detection Algorithms

• US 20130007026A1
• Filed: 09/13/2012
• Published: 01/03/2013
• Est. Priority Date: 02/11/2004
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: KOLCZ, Aleksander, CHOWDHURY, Abdur R., ALSPECTOR, Joshua
• Granted Patent: US 8,768,940 B2
• Time in Patent Office: Days
• US Class Current: 707/755
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

In a single-signature duplicate document system, a secondary set of attributes is used in addition to a primary set of attributes so as to improve the precision of the system. When the projection of a document onto the primary set of attributes is below a threshold, then a secondary set of attributes is used to supplement the primary lexicon so that the projection is above the threshold.

Claims

1. A method for use in detecting a duplicate document, the method comprising:

generating a primary lexicon of attributes and a secondary lexicon of attributes;

determining unique attributes in a document;

determining an intersection between the unique attributes in the document and the primary lexicon;

determining whether the intersection exceeds a threshold;

when the intersection does not exceed the threshold, adding attributes from the secondary lexicon that intersect with the unique attributes in the document to the intersection to create an augmented intersection that exceeds the threshold; and

calculating a signature for the document based on the augmented intersection.