System and method for clustering distributed hash table entries

• US 8,849,825 B1
• Filed: 12/23/2010
• Issued: 09/30/2014
• Est. Priority Date: 12/23/2010
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: McHugh, Jason G., Gattu, Praveen K., Ten-Pow, Michael A., Denny-Brown, Derek E. II
• Primary Examiner(s): Alam, Hosain
• Assistant Examiner(s): Johnson, Johnese
• Application Number:
  US12/978,253
• Time in Patent Office: 1,377 Days
• Field of Search: 707/737
• US Class Current: 707/737
• CPC Class Codes: G06F 16/1873 Versioning file systems, temporal file systems, e.g. file system supporting different historic versions of files
• CPC Class Codes: G06F 16/2255 Hash tables

Abstract

A distributed storage system may store data object instances in persistent storage and may store keymap information for those data object instances in a distributed hash table on multiple computing nodes. Each data object instance may include a composite key containing a user key. The keymap information for each data object instance may map the user key to a locator and the locator to the data object instance. A request to store or retrieve keymap information for a data object instance may be routed to a particular computing node based on a consistent hashing scheme in which a hash function is applied to a portion of the composite key of the data object instance. Thus, related entries may be clustered on the same computing nodes. The portion of the key to which the hash function is applied may include a pre-determined number of bits or be identified using a delimiter.

Claims

1. A method, comprising:

performing, by a computer system that stores a plurality of data objects in a distributed storage system;

receiving a request to store an instance of a data object in the distributed storage system, wherein the request comprises a composite key for the data object, and wherein the composite key comprises a sharable user key and one or more other elements usable to uniquely identify the instance of the data object;

generating keymap information for the instance of the data object that maps the user key to a locator and the locator to the instance of the data object;

applying a hash function to a portion of the composite key that is less than all of the composite key;

determining a computing node on which to store the keymap information dependent on a result of said applying the hash function, wherein the computing node is one of a plurality of computing nodes that each store a portion of a distributed hash table;

storing the keymap information on the determined computing node, wherein the determined computing node also stores keymap information for other data object instances that are related to the instance of the data object.

2. The method of claim 1, further comprising:

receiving a request to retrieve the instance of the data object, wherein the request specifies the composite key;

applying a hash function to a portion of the specified composite key that is less than all of the specified composite key;

determining the computing node on which the keymap information is stored dependent on a result of said applying the hash function to a portion of the specified composite key; and

returning keymap information for the instance of the data object.

3. The method of claim 2, further comprising retrieving the instance of the data object from the distributed storage system dependent on the returned keymap information.

4. The method of claim 1, wherein keymap information for all data object instances having the user key is stored on the same computing node.

5. The method of claim 1, wherein the portion of the key to which the hash function is applied is dependent on the position of a delimiter in the composite key.

6. The method of claim 1, wherein the portion of the composite key to which the hash function is applied comprises a pre-determined number of bits.

7. A method, comprising:

performing, by a computer system that stores a plurality of data objects in a distributed storage system;

receiving a request to store keymap information for an instance of a data object to be stored in the distributed storage system, wherein the request comprises a composite key for the data object, wherein the composite key comprises a sharable user key and one or more other elements usable to uniquely identify the instance of the data object, and wherein the keymap information maps the user key to a locator and the locator to the instance of the data object;

applying a hash function to a portion of the composite key that is less than all of the composite key;

determining a computing node on which to store the keymap information dependent on a result of said applying the hash function, wherein the computing node is one of a plurality of computing nodes that each store a portion of a distributed hash table;

storing the keymap information on the determined computing node, wherein the determined computing node also stores keymap information for other data object instances that are in a same key cluster as the data object.

8. The method of claim 7, wherein the key cluster is defined by a set of composite keys for data object instances to be stored on the same computing node.

9. The method of claim 7, further comprising:

receiving a request to retrieve the instance of the data object, wherein the request specifies the composite key;

applying a hash function to a portion of the specified composite key that is less than all of the specified composite key;

determining the computing node on which the keymap information is stored dependent on a result of said applying the hash function to a portion of the specified composite key; and

returning keymap information for the instance of the data object.

10. The method of claim 9, further comprising retrieving the instance of the data object from the distributed storage system dependent on the returned keymap information.

11. The method of claim 7, wherein keymap information for all data object instances having the user key is stored on the same computing node.

12. The method of claim 7, wherein the portion of the composite key to which the hash function is applied is dependent on the position of a delimiter in the composite key.

13. The method of claim 7, wherein the portion of the composite key to which the hash function is applied comprises a pre-determined number of bits.

14. The method of claim 7, wherein the portion of the composite key to which the hash function is applied comprises at least a portion of the user key.

15. The method of claim 7, further comprising:

receiving a request for keymap information for a data object, wherein the request includes a particular user key but does not include a version identifier;

determining the latest version of data object instances that are stored in the distributed storage system and that have the particular user key from among a plurality of stored data object instances having the particular user key, wherein determining the latest version comprises examining keymap information for two data object instances having the particular user key; and

returning keymap information for the latest version of data object instances that are stored in the distributed storage system and that have the particular user key.

16. The method of claim 15, wherein the keymap information for the two data object instances having the particular user key is stored on the same computing node.

17. The method of claim 7, wherein the distributed hash table caches keymap information for data objects stored in the distributed storage system, wherein said storing the keymap information on the determined computing node comprises storing the keymap information in the distributed hash table, and wherein the keymap information is indexed in the distributed hash table by a hash value generated as a result of said applying the hash function.

18. The method of claim 7, further comprising storing the keymap information for the instance of the data object in persistent storage.

19. A non-transitory, computer-readable storage medium storing program instructions that when executed on one or more computers cause the one or more computers to perform:

receiving a request to store keymap information for an instance of a data object to be stored in a distributed storage system, wherein the request comprises a composite key for the data object, wherein the composite key comprises a sharable user key and one or more other elements usable to uniquely identify the instance of the data object, and wherein the keymap information maps the user key to a locator and the locator to the instance of the data object;

applying a hash function to a portion of the composite key that is less than all of the composite key;

determining a computing node on which to store the keymap information dependent on a result of said applying the hash function, wherein the computing node is one of a plurality of computing nodes that each store a portion of a distributed hash table;

storing the keymap information on the determined computing node, wherein the determined computing node also stores keymap information for other data object instances that are in a same key cluster as the data object.

20. The storage medium of claim 19, wherein the key cluster is defined by a set of composite keys for data object instances to be stored on the same computing node.

21. The storage medium of claim 19, wherein when executed on the one or more computers, the program instructions further cause the one or more computers to perform:

receiving a request to retrieve the instance of the data object, wherein the request specifies the composite key;

applying a hash function to a portion of the specified composite key that is less than all of the specified composite key;

determining the computing node on which the keymap information is stored dependent on a result of said applying the hash function to a portion of the specified composite key; and

returning keymap information for the instance of the data object.

22. The storage medium of claim 21, wherein when executed on the one or more computers, the program instructions further cause the one or more computers to perform retrieving the instance of the data object from the distributed storage system dependent on the returned keymap information.

23. The storage medium of claim 19, wherein keymap information for all data object instances having the user key is stored on the same computing node.

24. The storage medium of claim 19, wherein the portion of the composite key to which the hash function is applied is dependent on the position of a delimiter in the composite key.

25. The storage medium of claim 19, wherein the portion of the composite key to which the hash function is applied comprises a pre-determined number of bits.

26. The storage medium of claim 19, wherein the portion of the composite key to which the hash function is applied comprises at least a portion of the user key.

27. The storage medium of claim 19, wherein when executed on the one or more computers, the program instructions further cause the one or more computers to perform:

receiving a request for keymap information for a data object, wherein the request includes a particular user key but does not include a version identifier;

determining the latest version of data object instances that are stored in the distributed storage system and that have the particular user key from among a plurality of stored data object instances having the particular user key, wherein determining the latest version comprises examining keymap information for two data object instances having the particular user key, and wherein the keymap information for the two data object instances having the particular user key is stored on the same computing node; and

returning keymap information for the latest version of data object instances that are stored in the distributed storage system and that have the particular user key.

28. The storage medium of claim 19, wherein the distributed hash table caches keymap information for data objects stored in the distributed storage system, wherein said storing the keymap information on the determined computing node comprises storing the keymap information in the distributed hash table, and wherein the keymap information is indexed in the distributed hash table by a hash value generated as a result of said applying the hash function.

29. A system, comprising:

a distributed hash table that is implemented on a plurality of computing nodes and that caches keymap information for a plurality of data objects stored in a persistent data store;

one or more processors; and

a memory coupled to the one or more processors and storing program instructions that when executed by the one or more processors cause the one or more processors to perform;

receiving a request to store keymap information for an instance of a data object stored in the persistent data store, wherein the request comprises a composite key for the data object, and wherein the composite key comprises a sharable user key and one or more other elements usable to uniquely identify the instance of the data object, and wherein the keymap information maps the user key to a locator and the locator to the instance of the data object;

applying a hash function to a portion of the composite key that is less than all of the composite key;

determining a computing node on which to store the keymap information in the distributed hash table dependent on a result of said applying the hash function, wherein the computing node is one of the plurality of computing nodes, and where each of the plurality of computing nodes stores a portion of the distributed hash table;

storing the keymap information on the determined computing node, wherein the determined computing node also stores keymap information for other data object instances that are related to the instance of the data object.

30. The system of claim 29, wherein the keymap information is indexed in the distributed hash table by a hash value generated as a result of said applying the hash function.

31. The system of claim 29, wherein when executed by the one or more processors, the program instructions further cause the one or more processors to perform:

receiving a request to retrieve the instance of the data object, wherein the request specifies the composite key;

applying a hash function to a portion of the specified composite key that is less than all of the specified composite key;

determining the computing node on which the keymap information is stored in the distributed hash table dependent on a result of said applying the hash function to a portion of the specified composite key; and

returning keymap information for the instance of the data object.

32. The system of claim 31, wherein when executed by the one or more processors, the program instructions further cause the one or more processors to perform retrieving the instance of the data object from the distributed storage system dependent on the returned keymap information.

33. The system of claim 29, wherein keymap information for all data object instances having the user key is stored in the distributed hash table on the same computing node.

34. The system of claim 29, wherein the portion of the composite key to which the hash function is applied is dependent on the position of a delimiter in the composite key.

35. The system of claim 29, wherein the portion of the composite key to which the hash function is applied comprises a pre-determined number of bits.