Duplicate document detection in a web crawler system
First Claim
1. A computer-implemented method of detecting duplicate documents in a network crawling system, comprising, at a server having one or more processors and memory:
- constructing a plurality of tables, each table corresponding to a portion of a document address space, storing information identifying documents having a same document content identifier and each identified document having an associated document rank;
wherein documents having the same document content identifier have the same content and documents having different document content identifiers have different content;
receiving a newly crawled document, such document characterized by a document content identifier and a document rank;
reading information stored in the plurality of tables to identify a set of documents sharing the document content identifier of the newly crawled document, and ascertaining an original representative document for the identified set of documents;
updating the information stored in at least one of the tables in accordance with the document ranks of the identified set of documents and the newly crawled document;
determining a representative document for the newly crawled document and the identified set of documents;
indexing the representative document when the representative document is the newly crawled document; and
repeating the receiving, reading, updating, determining and indexing operations with respect to a plurality of newly crawled documents, each of which shares a respective document content identifier with a respective set of documents, such that at least some of the newly crawled documents are determined to be representative documents and are indexed.
2 Assignments
0 Petitions
Accused Products
Abstract
Duplicate documents are detected in a web crawler system. Upon receiving a newly crawled document, a set of documents, if any, sharing the same content as the newly crawled document is identified. Information identifying the newly crawled document and the selected set of documents is merged into information identifying a new set of documents. Duplicate documents are included and excluded from the new set of documents based on a query independent metric for each such document. A single representative document for the new set of documents is identified in accordance with a set of predefined conditions.
123 Citations
30 Claims
-
1. A computer-implemented method of detecting duplicate documents in a network crawling system, comprising, at a server having one or more processors and memory:
-
constructing a plurality of tables, each table corresponding to a portion of a document address space, storing information identifying documents having a same document content identifier and each identified document having an associated document rank;
wherein documents having the same document content identifier have the same content and documents having different document content identifiers have different content;receiving a newly crawled document, such document characterized by a document content identifier and a document rank; reading information stored in the plurality of tables to identify a set of documents sharing the document content identifier of the newly crawled document, and ascertaining an original representative document for the identified set of documents; updating the information stored in at least one of the tables in accordance with the document ranks of the identified set of documents and the newly crawled document; determining a representative document for the newly crawled document and the identified set of documents; indexing the representative document when the representative document is the newly crawled document; and repeating the receiving, reading, updating, determining and indexing operations with respect to a plurality of newly crawled documents, each of which shares a respective document content identifier with a respective set of documents, such that at least some of the newly crawled documents are determined to be representative documents and are indexed. - View Dependent Claims (2, 3, 4, 5, 6)
-
-
7. A computer-implemented method of detecting duplicate documents in a network crawling system, comprising, at a server having one or more processors and memory:
-
constructing a plurality of tables, each table corresponding to a segment of a document address space, storing information identifying documents having a same document content identifier and each identified document having an associated document rank, wherein the plurality of tables comprise N+1 tables where N is an integer greater than one, wherein the N+1 tables comprise N tables, each generated during a respective phase of a set of N crawling phases, and a current table generated during a current one of the N crawling phases, wherein an oldest one of the N tables was generated during a previous instance of the current crawling phase; receiving a newly crawled document, such document characterized by a document content identifier and a document rank;
wherein documents having the same document content identifier have the same content and documents having different document content identifiers have different content;reading information stored in the N+1 tables to identify a set of documents sharing the document content identifier of the newly crawled document, and ascertaining an original representative document for the identified set of documents; updating the information stored in the current table in accordance with the document rankings of the identified set of documents and the newly crawled document; determining a representative document for the newly crawled document and the identified set of documents; indexing the representative document when said representative document is the newly crawled document; repeating the receiving, reading, updating, determining and indexing operations with respect to a plurality of newly crawled documents, each of which shares a respective document content identifier with a respective set of documents, such that at least some of the newly crawled documents are determined to be representative documents and are indexed; and upon completion of the current crawling phase, retiring the oldest one of the N tables. - View Dependent Claims (8, 9)
-
-
10. A system for detecting duplicate documents during network crawling, comprising:
-
one or more central processing units for executing programs; a network interface for receiving documents; and a duplicate document detection engine executable by the one or more central processing units, the engine comprising; a plurality of tables, each table corresponding to a segment of a document address space, storing information identifying documents having a same document content identifier and each identified document having an associated document rank, wherein the plurality of tables comprise N+1 tables where N is an integer greater than one, wherein the N+1 tables comprise N tables, each generated during a respective phase of a set of N crawling phases, and a current table generated during a current one of the N crawling phases, wherein an oldest one of the N tables was generated during a previous instance of the current crawling phase; instructions for receiving a newly crawled document, such document characterized by a document content identifier and a document rank;
wherein documents having the same document content identifier have the same content and documents having different document content identifiers have different content;instructions for reading information stored in the N+1 tables to identify a set of documents, sharing the document content identifier of the newly crawled document, and ascertaining an original representative document for the identified set of documents; instructions for updating the information stored in the current table in accordance with the document rankings of the identified set of documents and the newly crawled document; instructions for determining a representative document for the newly crawled document and the identified set of documents; instructions for indexing the representative document when said representative document is the newly crawled document; instructions for repeating the receiving, reading, updating, determining and indexing operations with respect to a plurality of newly crawled documents, each of which shares a respective document content identifier with a respective set of documents, such that at least some of the newly crawled documents are determined to be representative documents and are indexed; and instructions for retiring the oldest one of the N tables upon completion of the current crawling phase. - View Dependent Claims (11, 12)
-
-
13. A computer program product for use in conjunction with a computer system, the computer program product comprising a computer readable storage medium and a computer program mechanism embedded therein, the computer program mechanism comprising:
-
instructions for constructing a plurality of data structures for storing information of documents, each document characterized by a document content identifier and a document rank, the information stored in the plurality of data structures include the document content identifier and a document rank for each document;
wherein documents having the same document content identifier have the same content and documents having different document content identifiers have different content;instructions for receiving a requesting document in association with its document content identifier and document rank; instructions for selecting from the plurality of data structures a set of documents sharing the same document content identifier as the requesting document, and ascertaining an original representative document for the identified set of documents; instructions for generating a new set of documents from the requesting document and the selected set of documents in accordance with their document rank; instructions for identifying a representative document of the new set of documents; instructions for indexing the representative document when said representative document is the requesting document; and instructions for repeating the receiving, selecting, generating, identifying, and indexing operations with respect to a plurality of requesting documents, each of which shares a respective document content identifier with a respective set of documents, such that at least some of the requesting documents are determined to be representative documents and are indexed. - View Dependent Claims (14, 15, 16, 17, 18, 19, 20, 21)
-
-
22. A computer program product for use in conjunction with a computer system, the computer program product comprising a computer readable storage medium and a computer program mechanism embedded therein, the computer program mechanism comprising:
-
instructions for constructing a plurality of tables, each table corresponding to a portion of a document address space, storing information identifying documents having a same document content identifier and each identified document having an associated document rank;
wherein documents having the same document content identifier have the same content and documents having different document content identifiers have different content;instructions for receiving a newly crawled document, such document characterized by a document content identifier and a document rank; instructions for reading information stored in the plurality of tables to identify a set of documents sharing the document content identifier of the newly crawled document, and ascertaining an original representative document for the identified set of documents; instructions for updating the information stored in at least one of the tables in accordance with the document ranks of the identified set of documents and the newly crawled document; instructions for determining a representative document for the newly crawled document and the identified set of documents; instructions for indexing the representative document when said representative document is the newly crawled document; and instructions for repeating the receiving, reading, updating, determining and indexing operations with respect to a plurality of newly crawled documents, each of which shares a respective document content identifier with a respective set of documents, such that at least some of the newly crawled documents are determined to be representative documents and are indexed. - View Dependent Claims (23, 24, 25, 26, 27)
-
-
28. A computer program product of detecting duplicate documents for use in conjunction with a computer system, the computer program product comprising a computer readable storage medium and a computer program mechanism embedded therein, the computer program mechanism comprising:
-
instructions for constructing a plurality of tables, each table corresponding to a segment of a document address space, storing information identifying documents having a same document content identifier and each identified document having an associated document rank, wherein the plurality of tables comprise N+1 tables where N is an integer greater than one, wherein the N+1 tables comprise N tables, each generated during a respective phase of a set of N crawling phases, and a current table generated during a current one of the N crawling phases, wherein an oldest one of the N tables was generated during a previous instance of the current crawling phase; instructions for receiving a newly crawled document, such document characterized by a document content identifier and a document rank;
wherein documents having the same document content identifier have the same content and documents having different document content identifiers have different content;instructions for reading information stored in the N+1 tables to identify a set of documents sharing the document content identifier of the newly crawled document, and ascertaining an original representative document for the identified set of documents; instructions for updating the information stored in the current table in accordance with the document rankings of the identified set of documents and the newly crawled document; instructions for determining a representative document for the newly crawled document and the identified set of documents; instructions for indexing the representative document when said representative document is the newly crawled document; instructions for repeating the receiving, reading, updating, determining and indexing operations with respect to a plurality of newly crawled documents, each of which shares a respective document content identifier with a respective set of documents, such that at least some of the newly crawled documents are determined to be representative documents and are indexed; and instructions for retiring the oldest one of the N tables upon completion of the current crawling phase. - View Dependent Claims (29, 30)
-
Specification