Locating potentially identical objects across multiple computers based on stochastic partitioning of workload

• US 7,571,186 B2
• Filed: 11/18/2004
• Issued: 08/04/2009
• Est. Priority Date: 06/06/2001
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Theimer, Marvin M., Bolosky, William J., Douceur, John R., Adya, Atul
• Primary Examiner(s): Breene; John E
• Assistant Examiner(s): NGUYEN, PHONG H
• Application Number:
  US10/991,989
• Publication Number:
  US 20050071315A1
• Time in Patent Office: 1,720 Days
• Field of Search: 707/1, 707/103.X, 707/200, 707/205, 709/201, 709/203, 709/212, 709/217, 709/218, 709/219, 709/226, 709/227, 709/229
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure
• CPC Class Codes: Y10S 707/99946 Object-oriented database structure network
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users
• CPC Class Codes: Y10S 707/99953 Recoverability

Abstract

Potentially identical objects (e.g., files) are located across multiple computers based on stochastic partitioning of workload. For each of a plurality of objects stored on a plurality of computers in a network, a portion of object information corresponding to the object is selected. The object information can be generated in a variety of manners (e.g., based on hashing the object, based on characteristics of the object, and so forth). Any of a variety of portions of the object information can be used (e.g., the least significant bits of the object information). A stochastic partitioning process is then used to identify which of the plurality of computers to communicate the object information to for identification of potentially identical objects on the plurality of computers.

Claims

1. A method implemented on a processor coupled to a memory, comprising:

determining a size for an imprint identification code of an object stored at a computer, wherein the size of the imprint identification code is based at least in part on a count of other computers in a network coupled to the computer, wherein each of the other computers in the network have communicated to the said computer information indicating that each other computer is actively available in the network;

identifying a particular imprint identification code based on the size for each object stored at the computer, wherein the imprint identification code is a calculated set of bits of object information corresponding to the object and derived from the object;

in an event that the object information is based on the data in the object, the object information is a semi-unique value based at least in part on the data in the object;

in an event that the object information is based on characteristics of the object, the object information is a value based at least in part on one or more characteristics of the object;

accessing the imprint identification code to compute a mapping that maps the imprint identification code to one or more computer identifiers associated with one or more computers that are active on the network, wherein the one or more computer identifiers have the same set of bits as the imprint identification code;

identifying one or more computers to receive the object information based at least in part on the mapping; and

sending the object information to each of the identified one or more computers.

2. A method as recited in claim 1, wherein the object comprises a file and wherein the object information comprises file information.

3. A method as recited in claim 1, wherein the one or more computers includes the computer at which the object is stored.

4. A method as recited in claim 1, wherein the accessing comprises accessing a locally stored imprint to computer mapping.

5. A method as recited in claim 1, wherein the accessing comprises accessing an imprint to computer mapping stored at another computer, wherein the information communicated to the computer is object information, and wherein the information is sent to the computer when the computer joins the network.

6. A method as recited in claim 1, wherein the identifying one or more computers comprises identifying one computer to receive the object information based at least in part on the accessed mapping, wherein the object information comprises one or more imprint identification codes, and wherein the sending comprises sending the object information to the one identified computer so that the one identified computer can forward the object information to one or more other computers responsible for maintaining the object information.

7. A method as recited in claim 1, wherein the size of the imprint identification code is based at least in part on an average number of computers that a particular object identifier or imprint identification code should be communicated to.

8. The method of claim 1 wherein sending the object information to each of the identified one or more computers, causes each of the identified one or more computers to perform a method, the method comprising:

receiving object information corresponding to another object;

comparing the received object information to object information in an object information database;

checking whether the received object information matches any of the object information in the object information database; and

determining that two identical objects exist if the received object information matches any of the object information in the object information database.