Management of class-associated object mapping information corresponding to a distributed storage system

• US 8,521,771 B1
• Filed: 07/31/2009
• Issued: 08/27/2013
• Est. Priority Date: 07/31/2009
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Sorenson, James Christopher III, Kakulapati, Gunavardhan, McHugh, Jason G.
• Primary Examiner(s): Smith, Brannon W
• Application Number:
  US12/533,945
• Time in Patent Office: 1,488 Days
• Field of Search: 707/758, 707/999.104, 707/770
• US Class Current: 707/770
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database

Abstract

Systems and methods for managing mapping information for objects maintained in a distributed storage system are provided. The distributed storage system can include a keymap subsystem that manages the mapping information according to object keys. Requests for specific object mapping information are directed to specific keymap coordinators within the keymap subsystem. Each keymap coordinator can maintain a cache for caching mapping information maintained at various information sources. To manage the cache, the keymap system can utilize generation identifiers that identify the component within the distributed storage system that last updated information sources. Additionally, the keymap subsystem can utilize multiple generation identifiers to manage different classes of data. Each keymap coordinator can determine whether cached version of a class of keymap information is valid based on an analysis of generation identifiers.

Claims

1. A method of managing keymap information associated with an object stored within a storage system, the object represented by a key, the keymap information mapping the key to one or more locations of instances of the object on the storage system, wherein at least a portion of interactions with the keymap information is facilitated by one or more keymap coordinators for coordinating interaction between clients and keymap information source, the method comprising:

obtaining, at a keymap coordinator, an identifier associated with the keymap coordinator, wherein keymap information including the identifier associated with the keymap coordinator indicates that the keymap coordinator has most recently modified the keymap information;

obtaining a read request at the keymap coordinator for keymap information associated with a first key;

obtaining cached keymap information, at the keymap coordinator, corresponding to the read request, the cached keymap information including two or more generations identifiers associated with the cached keymap information, each of the two or more generations identifiers corresponding to a component that accessed at least one class of keymap information associated with the first key at a time corresponding to creation of the generations identifier, wherein the cached keymap information is associated with a plurality of classes corresponding to a plurality of independently accessible types of keymap information, and wherein each of the plurality of classes is associated with a portion of the cached keymap information;

determining whether the identifier associated with the keymap coordinator matches the most recent generations identifier of the two or more generations identifiers;

if the identifier associated with the keymap coordinator matches the most recent generations identifier of the two or more generations identifiers, processing the obtained cached keymap information responsive to the read request; and

if the identifier associated with the keymap coordinator does not match the most recent identifier of the two or more generations identifiers;

transmitting a request from the keymap coordinator for keymap information corresponding to the at least one class of keymap information;

obtaining responsive keymap information at the keymap coordinator;

updating the cached keymap information with the obtained responsive keymap information; and

processing the updated cached keymap information responsive to the read request.

2. The method as recited in claim 1, wherein the keymap information further includes a locator mapped to the first key.

3. The method as recited in claim 1, wherein each of the two or more generations identifiers correspond to a separate class of data.

4. The method as recited in claim 1 further comprising:

responsive to obtaining the read request at the keymap coordinator, initiating a request for keymap information associated with the first key to a cache component associated with the keymap coordinator.

5. The method as recited in claim 1, wherein the most recent generations identifier matches an identifier of the component that most recently modified the keymap information.

6. A method of managing keymap information associated with an object stored within a storage system, the object represented by a key, the keymap information mapping the key to one or more locations of instances of the object on the storage system, wherein at least a portion of interactions with the keymap information is facilitated by one or more keymap coordinators, the method comprising:

obtaining, at a keymap coordinator, an identifier associated with the keymap coordinator, wherein keymap information including the identifier associated with the keymap coordinator indicates that the keymap coordinator has most recently accessed the keymap information;

obtaining a read request at the keymap coordinator for keymap information associated with a first key;

obtaining cached keymap information, at the keymap coordinator from an associated cache component corresponding to the read request, the cached keymap information including two or more generations identifiers, the two or more generations identifier identifying at least a component that last accessed the keymap information associated with the first key and associated with a class of cached keymap information, wherein the cached keymap information is associated with a plurality of independently accessible classes of keymap information, and wherein each of the plurality of classes is associated with a portion of the cached keymap information;

determining a selected generations identifier from the two or more generations identifiers based on a current class associated with the read request;

determining whether the selected generations identifier matches the identifier associated with the keymap coordinator;

if the selected generations identifier matches the identifier associated with the keymap coordinator, processing the obtained cached keymap information responsive to the read request; and

if the selected generations identifier does not match the identifier associated with the keymap coordinator, updating the cached keymap information and processing the updated cached keymap information responsive to the read request.

7. The method as recited in claim 6, wherein the keymap information further includes a locator mapped to the first key.

8. The method as recited in claim 6, wherein the at least one generations identifier comprises two or more generations identifiers.

9. The method as recited in claim 6, wherein the component that last accessed the keymap information comprises the component that last modified the keymap information.

10. The method as recited in claim 6 further comprising:

prior to obtaining cached keymap information, initiating a request to the cache component associated with the keymap coordinator for the keymap information associated with the first key.

11. The method as recited in claim 6, wherein updating the cached keymap information comprises:

transmitting a request from the keymap coordinator to a keymap information source for keymap information associated with the first key;

obtaining responsive keymap information at the keymap coordinator;

updating the cached keymap information at the cache component with the obtained responsive keymap information.

12. A method of managing keymap information associated with an object stored within a storage system, the object represented by a key, the keymap information mapping the key to one or more locations of instances of the object on the storage system, wherein at least a portion of interactions with the keymap information is facilitated by one or more callers, the method comprising:

obtaining a caller identifier, wherein keymap information including the caller identifier indicates that a caller associated with the caller identifier has most recently accessed the keymap information;

obtaining a modification request for modifying keymap information associated with a first key of a set of keymap information;

obtaining a class-based generations identifier associated with the first key and identifying a component that most recently accessed the keymap information associated with the first key of the set of keymap information, wherein the set of keymap information is associated with a plurality of independently accessible classes of keymap information, and wherein each of the plurality of independently accessible classes of keymap information is associated with a portion of the set of keymap information and with one or more class-based generations identifiers;

determining whether the class-based generations identifier associated with the first key matches the caller identifier;

if the class-based generations identifier matches the caller identifier, processing the modification request; and

if the class-based generations identifier does not match the caller identifier, generating a new class-based generations identifier for the keymap information associated with the first key, wherein the new class-based generations identifier is based on a class associated with the keymap information, and wherein the new class-based generations identifier matches the caller identifier.

13. The method as recited in claim 12, wherein the keymap information further includes a locator mapped to the first key.

14. The method as recited in claim 12, wherein the caller is a keymap coordinator.

15. The method as recited in claim 12, wherein the component that most recently accessed the keymap information comprises the component that most recently modified the keymap information.

16. The method as recited in claim 12, wherein the modification request comprises one of a write request and a delete request.

17. The method as recited in claim 12, wherein the new class-based generations identifier comprises a caller identifier corresponding to the caller.

18. The method as recited in claim 12 further comprising associating the new class-based generations identifier with the keymap information associated with the first key.

19. The method as recited in claim 12, wherein processing the modification request comprises processing the keymap information associated with the first key responsive to the modification request.

20. The method as recited in claim 19, wherein processing the keymap information comprises updating the keymap information associated with the first key responsive to the modification request.

21. A system of managing keymap information associated with an object stored within a storage system, the object represented by a key, the keymap information mapping the key to one or more locations of instances of the object on the storage system, wherein at least a portion of interactions with the keymap information is facilitated by one or more keymap coordinators, the system comprising:

one or more processors configured to implement;

a keymap coordinator associated with a keymap identifier, wherein the keymap coordinator obtains a read request associated with a first key; and

a cache component that obtains a read request from the keymap coordinator for cached keymap information associated with the first key, the cached keymap information including two or more class-based generations identifiers associated with the first key, the two or more class-based generations identifiers identifying at least a component that last accessed the keymap information associated with the first key, wherein the cached keymap information is associated with a plurality of independently accessible classes of keymap information, and wherein each of the plurality of independently accessible classes of keymap information is associated with a portion of the cached keymap information;

wherein the keymap coordinator is further operable to;

obtain the cached keymap information;

select a class-based generations identifier from the two or more class-based generations identifiers, wherein the selected class-based generations identifier identifies the component that last accessed the keymap information associated with the first key; and

determine whether the selected generations identifier matches the keymap identifier;

if the selected generations identifier matches the keymap identifier, process the obtained cached keymap information responsive to the read request; and

if the selected generations identifier does not match the keymap identifier, update the cached keymap information and process the updated cached keymap information responsive to the read request.

23. The system as recited in claim 21, wherein the caller comprises a keymap coordinator.

24. The system as recited in claim 21, wherein the independent class of keymap information corresponds to deleted keymap information.

25. The system as recited in claim 21, wherein the independent class of keymap information corresponds to keymap information having a null value.

22. A system of keymap managing information associated with an object stored within a storage system, the object represented by a key, the keymap information mapping the key to one or more locations of instances of the object on the storage system, wherein at least a portion of interactions with the keymap information is facilitated by one or more callers, the system comprising:

one or more processors configured to implement;

a caller associated with a caller identifier, wherein the caller obtains a modification request associated with a first key; and

a manager component that;

obtains the modification request associated with the first key from the caller, the modification request for modifying keymap information associated with the first key;

selects a generations identifier identifying a component that most recently accessed the keymap information associated with the first key, the selected generations identifier corresponding an independent class of keymap information relative to other classes of keymap information maintained at an information source, wherein each of the classes of keymap information is associated with a portion of a cache of keymap information;

determines whether the selected generations identifier matches the caller identifier;

if the selected generations identifier matches the caller identifier, processes the modification request; and

if the selected generations identifier does not match the caller identifier, generates a new generations identifier for the keymap information, the new generations identifier matching the caller identifier.