Locating potentially identical objects across multiple computers based on stochastic partitioning of workload

• US 7,272,630 B2
• Filed: 11/18/2004
• Issued: 09/18/2007
• Est. Priority Date: 06/06/2001
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Theimer, Marvin M., Bolosky, William J., Douceur, John R., Adya, Atul
• Primary Examiner(s): Hu; Jinsong
• Application Number:
  US10/991,571
• Publication Number:
  US 20050097148A1
• Time in Patent Office: 1,034 Days
• Field of Search: 709/203, 709/201, 709/219, 715/739
• US Class Current: 709/203
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure
• CPC Class Codes: Y10S 707/99946 Object-oriented database structure network
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

Potentially identical objects (e.g., files) are located across multiple computers based on stochastic partitioning of workload. For each of a plurality of objects stored on a plurality of computers in a network, a portion of object information corresponding to the object is selected. The object information can be generated in a variety of manners (e.g., based on hashing the object, based on characteristics of the object, and so forth). Any of a variety of portions of the object information can be used (e.g., the least significant bits of the object information). A stochastic partitioning process is then used to identify which of the plurality of computers to communicate the object information to for identification of potentially identical objects on the plurality of computers.

Claims

1. A method implemented in a computer, the method comprising:

generating an imprint identification code for an object stored at the computer, wherein the imprint identification code comprises a first set of bits of object information corresponding to the object, and wherein the number of bits in the first set of bits is based at least in part on a count of computers that the computer is aware of in the network;

identifying one or more additional computers each having a computer identifier that includes a second set of bits that match the imprint identification code; and

sending the object information to each of the identified one or more additional computers.

2. A method as recited in claim 1, wherein the object comprises a file and the object information comprises file information.

3. A method as recited in claim 1, wherein the first set of bits comprises the least significant bits of the object information, and wherein the second set of bits comprises the least significant bits of the computer identifier.

4. A method as recited in claim 1, wherein the object information is a semi-unique value based at least in part on the data in the object.

5. A method as recited in claim 1, wherein the sending further comprises sending an object identifier corresponding to the object along with the object information.

6. A method as recited in claim 1, further comprising storing the object information in an object information database stored at the computer.

7. A method as recited in claim 1, further comprising:

receiving object information corresponding to an object stored on another computer;

comparing the received object information to an object information database;

checking whether the received object information matches any of the object information in the database; and

determining that two potentially identical objects exist if the received object information matches any of the object information in the database.

8. A method as recited in claim 1, wherein the locations of the first set of bits correspond to the locations of the second set of bits.

9. A method as recited in claim 1, wherein the number of bits in the first set of bits is based at least in part on an average number of computers that a particular object identifier should be communicated to.

10. One or more computer storage media having stored thereon a plurality of instructions that, when executed by one or more processors of a computer that is one of a plurality of computers in a network, causes the one or more processors to perform the following acts:

generating an imprint identification code for an object stored at the computer, wherein the imprint identification code comprises a first set of bits of object information corresponding to the object, and wherein the number of bits in the first set of bits is based at least in part on a count of computers of the plurality of computers that the computer is aware of in the network;

identifying one or more additional computers of the plurality of computers each having a computer identifier that includes a second set of bits that match the imprint identification code; and

sending the object information to each of the identified one or more additional computers.

11. One or more computer storage media as recited in claim 10, wherein the object comprises a file and the object information comprises file information.

12. One or more computer storage media as recited in claim 10, the plurality of instructions further causing the one or more processors to perform the following acts:

receiving object information corresponding to an object stored on another computer of the plurality of computers;

comparing the received object information to an object information database;

checking whether the received object information matches any of the object information in the database; and

determining that two potentially identical objects exist if the received object information matches any of the object information in the database.

13. One or more computer storage media as recited in claim 10, wherein the number of bits in the first set of bits is based at least in part on an average number of computers of the plurality of computers that a particular object identifier should be communicated to.

14. A computing device comprising:

a processor;

a storage device, coupled to the processor, to store instructions that, when executed by the processor, cause the processor to;

generate an imprint identification code for an object stored at the computing device, wherein the imprint identification code comprises a first set of bits of object information corresponding to the object, and wherein the number of bits in the first set of bits is based at least in part on an average number of computing devices of the plurality of computing devices that a particular object identifier should be communicated to;

identify one or more additional computing devices each having a computing device identifier that includes a second set of bits that match the imprint identification code; and

send the object information to each of the identified one or more additional computing devices.

15. A computing device as recited in claim 14, wherein the object comprises a file and the object information comprises file information.

16. A computing device as recited in claim 14, wherein the instructions further cause the processor to:

receive object information corresponding to an object stored on another computing device;

compare the received object information to an object information database;

check whether the received object information matches any of the object information in the database; and

determine that two potentially identical objects exist if the received object information matches any of the object information in the database.

17. A computing device as recited in claim 14, wherein the number of bits in the first set of bits is based at least in part on an average number of computing devices of the plurality of computing devices that a particular object identifier should be communicated to.