Locating potentially identical objects across multiple computers based on stochastic partitioning of workload

• US 7,487,245 B2
• Filed: 05/27/2005
• Issued: 02/03/2009
• Est. Priority Date: 06/06/2001
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Theimer, Marvin M., Bolosky, William J., Douceur, John R., Adya, Atul
• Primary Examiner(s): Hu; Jinsong
• Application Number:
  US11/139,364
• Publication Number:
  US 20050222994A1
• Time in Patent Office: 1,348 Days
• Field of Search: 709/202, 709/212, 709/226, 709/228, 710/36, 710/37, 711/150
• US Class Current: 709/226
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure
• CPC Class Codes: Y10S 707/99946 Object-oriented database structure network
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

Potentially identical objects (e.g., files) are located across multiple computers based on stochastic partitioning of workload. For each of a plurality of objects stored on a plurality of computers in a network, a portion of object information corresponding to the object is selected. The object information can be generated in a variety of manners (e.g., based on hashing the object, based on characteristics of the object, and so forth). Any of a variety of portions of the object information can be used (e.g., the least significant bits of the object information). A stochastic partitioning process is then used to identify which of the plurality of computers to communicate the object information to for identification of potentially identical objects on the plurality of computers.

Claims

1. One or more computer-readable storage media having stored thereon a plurality of instructions that, when executed by one or more processors of a computer that is part of a plurality of computers in a network, cause the one or more processors to perform a method, the method comprising:

selecting a portion of file information corresponding to a file stored on one of the plurality of computers, wherein the selected portion of the file information comprises a set of least significant bits of the file information, and wherein a size of the portion of the file information is based at least in part on a count of computers that the one computer is aware of in the network;

comparing, for each of the plurality of computers, the selected portion to a portion of a computer identifier associated with the computer;

identifying which of the computer identifiers have portions matching the selected portion of the file information; and

communicating, for identification of potentially identical files stored on the plurality of computers, the file information to each of the computers associated with a computer identifier having a portion matching the selected portion of the file information.

2. One or more computer-readable storage media as recited in claim 1, wherein for each of the plurality of computers the portion of the computer identifier associated with the computer is a set of least significant bits of the computer identifier.

3. One or more computer-readable storage media as recited in claim 1, wherein the comparing comprises comparing the selected portion to a portion of a computer identifier associated with the one computer.

4. One or more computer-readable storage media as recited in claim 1, wherein the selected portion of the file information comprises a plurality of bits of the file information, wherein each portion of a computer identifier comprises a plurality of bits of the computer identifier, and wherein the locations of the plurality of bits in the file information correspond to the locations of the plurality of bits of the computer identifiers.

5. One or more computer-readable storage media as recited in claim 1, further comprising:

receiving, from one of the plurality of computers, other file information;

comparing the received file information to a file information database stored at the computer; and

determining that a file corresponding to the received file information is potentially identical to another file on another of the plurality of computers if the received file information matches file information in the database.

6. One or more computer-readable storage media as recited in claim 1, wherein the file information is a semi-unique value based at least in part on the data in the file.

7. One or more computer-readable storage media as recited in claim 6, wherein the file information includes a hash value generated by applying a hash function to the file.

8. One or more computer-readable storage media as recited in claim 1, wherein the file information is based at least in part on one or more characteristics of the file and wherein the number of least significant bits of the file information is determined by finding a largest integer that is no greater than the value of the base two logarithm of a modified computer count, the modified computer count being the total number of computers in the network divided by a system configuration parameter.

9. One or more computer-readable storage media as recited in claim 8, wherein the characteristics comprise a file size and a file type.

10. One or more computer-readable storage media as recited in claim 1, wherein the communicating further comprises communicating a file identifier corresponding to the file along with the file information.

11. One or more computer-readable storage media as recited in claim 1, wherein the size of the portion of the file information is based at least in part on an average number of computers that a particular file identifier should be communicated to.

12. One or more computer-readable storage media as recited in claim 1, further comprising:

determining a size of the portion of the file information;

wherein the size of the portion of the file information need not be the same as a size determined by others of the plurality of computers.

13. One or more computer-readable storage media as recited in claim 1, further comprising repeating the selecting, comparing, identifying, and communicating for every file stored at the one computer.

14. One or more computer-readable storage media as recited in claim 1, further comprising repeating the selecting, comparing, identifying, and communicating for every file stored at the one computer that exceeds a threshold size.

15. A system for identification of potentially identical files in the system, the system comprising:

a memory;

a processor;

an interface electronically coupled to the processor and configured to allow the system to communicate with a plurality of other computers; and

a forwarding location determination module, coupled to the interface, configured to perform a method, the method comprising;

identifying one or more of the plurality of computers to which to communicate file information corresponding to a file;

identifying a set of bits of file information associated with the file, wherein the set of bits of file information comprises a set of least significant bits of the file information, and wherein a size of the portion of the file information is based at least in part on a count of computers that the one computer is aware of in the network; and

identifying ones of the one or more computers that each have a computer identifier that has a set of bits that match the set of bits of the file information.

16. A system as recited in claim 15, wherein the number of bits in the set of bits is based at least in part on a count of computers coupled to the system that the system is aware of, and wherein the number of least significant bits of the file information is determined by finding a largest integer that is no greater than the value of the base two logarithm of a modified computer count, the modified computer count being the total number of computers in the network divided by a system configuration parameter.

17. A system as recited in claim 15, wherein the number of bits in the set of bits is based at least in part on an average number of computers that a particular file identifier should be communicated to.

18. A system as recited in claim 15, further comprising a file information comparison module coupled to a file information database, wherein the file information comparison module is configured to:

receive, from one of the plurality of computers, other file information;

compare the received file information to file information in the database; and

determine that a file corresponding to the received file information is potentially identical to another file on another of the plurality of computers if the received file information matches file information in the database.