Consistent data storage in distributed computing systems

• US 10,592,475 B1
• Filed: 03/21/2014
• Issued: 03/17/2020
• Est. Priority Date: 12/27/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Ghidireac, Bogdan Eduard, Sirota, Peter, Leidle, Robert Frederick, Mills, Mathew Alan, McPherson, George Steven, Wu, Xing, Fritz, Jonathan Andrew
• Primary Examiner(s): Mahmoudi, Tony
• Assistant Examiner(s): Le, Michael
• Application Number:
  US14/222,103
• Time in Patent Office: 2,188 Days
• Field of Search: 707827
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: G06F 16/14 Details of searching files based on file metadata
• CPC Class Codes: G06F 16/182 Distributed file systems

Abstract

Methods and apparatus for providing consistent data storage in distributed computing systems. A consistent distributed computing file system (consistent DCFS) may be backed by an object storage service that only guarantees eventual consistency, and may leverage a data storage service (e.g., a database service) to store and maintain a file system/directory structure (a consistent DCFS directory) for the consistent DCFS that may be accessed by compute nodes for file/directory information relevant to the data objects in the consistent DCFS, rather than relying on the information maintained by the object storage service. The compute nodes may reference the consistent DCFS directory to, for example, store and retrieve strongly consistent metadata referencing data objects in the consistent DCFS. The compute nodes may, for example, retrieve metadata from consistent DCFS directory to determine whether the object storage service is presenting all of the data that it is supposed to have.

Claims

1. A distributed computing system, comprising:

one or more hardware compute nodes configured to process a data set of data objects, wherein the data objects are stored using a first hardware computer system and a second hardware computer system that are accessible over one or more networks, wherein the processing of the data set includes generating a request to modify at least one of the data objects stored using the first and second hardware computer systems;

the first hardware computer system, configured to implement a distributed computing file system (DCFS) using an unstructured object storage service, wherein the DCFS stores the data objects of the data set as files using the unstructured object storage service, wherein the unstructured object storage service implements a first client-facing interface accessible to a plurality of clients, and wherein the unstructured data storage service is not guaranteed to return a latest version of the data objects produced by the request via the first client-facing interface;

the second hardware computer system, configured to implement a DCFS directory for the DCFS using a multi-tenant database service, wherein the DCFS directory stores metadata for the data objects of the DCFS, wherein the multi-tenant database service implements a second client-facing interface accessible to the plurality of clients, and wherein the multi-tenant database service is guaranteed to return a latest version of the metadata of the data objects produced by the request via the second client-facing interface;

wherein the DCFS directory is an authoritative store and source for directory information of the DCFS during processing of the data set by the one or more hardware compute nodes.

2. The distributed computing system as recited in claim 1, wherein, to process the data set, at least one of the one or more hardware compute nodes is configured to access the DCFS directory via the second client-facing interface of the multi-tenant database service to obtain strongly consistent metadata for accessing the data objects stored in the DCFS according to the first client-facing interface of the unstructured object storage service.

3. The distributed computing system as recited in claim 2, wherein at least one of the one or more hardware compute nodes is further configured to access the DCFS directory to store or update the metadata for accessing the data objects stored in the DCFS.

4. The distributed computing system as recited in claim 1, wherein the one or more hardware compute nodes are provisioned as resource instances in a client'"'"'s private network on a provider network.

5. The distributed computing system as recited in claim 1, wherein one or both of the DCFS and the DCFS directory are implemented in a provider network, and wherein the one or more hardware compute nodes are implemented in a client network external to the provider network.

6. The distributed computing system as recited in claim 1, wherein the distributed computing system is implemented according to a distributed computing framework that enables the distributed processing of the data set across a cluster of compute nodes using a programming model of the distributed computing framework.

7. A method, comprising:

establishing a distributed computing file system (DCFS) on one or more storage devices to store data objects in an unstructured object storage service, wherein the unstructured object storage service is implemented on a first hardware computer system accessible over one or more networks and implements a first client-facing interface accessible to a plurality of clients;

establishing a DCFS directory to store metadata for the data objects in the DCFS in a multi-tenant database service different from the unstructured object storage service, wherein the multi-tenant database service is implemented on a second hardware computer system accessible over the one or more networks and implements a second client-facing interface accessible to the plurality of clients;

responsive to a request to modify a data object of the data objects, modifying the data object in the unstructured object storage service and metadata of the data object in the multi-tenant database service, wherein the unstructured data storage service is not guaranteed to return a latest version of the data object produced by the request via the first client-facing interface, and the multi-tenant database service is guaranteed to return a latest version of the metadata of the data object produced by the request via the second client-facing interface; and

accessing, by one or more compute nodes distinct from the first and second hardware computer systems, the DCFS directory to obtain the metadata for accessing the data objects stored in the DCFS.

8. The method as recited in claim 7, wherein the unstructured object storage service is eventually consistent, and wherein the metadata stored in the DCFS directory and accessed by the one or more compute nodes is strongly consistent.

9. The method as recited in claim 7, wherein the DCFS directory is an authoritative store and source for directory information of the DCFS during processing of the data objects stored in the DCFS by the one or more compute nodes.

10. The method as recited in claim 7, further comprising accessing the unstructured object storage service to retrieve respective data objects stored in the DCFS according to the metadata obtained from the DCFS directory.

11. The method as recited in claim 7, further comprising accessing the DCFS directory to update the metadata for accessing the data objects stored in the DCFS.

12. The method as recited in claim 7, further comprising:

storing, by at least one of the one or more compute nodes, one or more data objects to a cache;

checking, by at least one of the one or more compute nodes, the cache for required data objects;

if a required data object is in the cache, retrieving the data object from the cache; and

if the required data object is not in the cache, accessing the DCFS directory to obtain metadata for retrieving the data object from the DCFS.

13. The method as recited in claim 7, further comprising:

periodically checking the metadata stored in the DCFS directory against metadata of the DCFS to detect inconsistencies between the DCFS directory and the DCFS; and

upon detecting an inconsistency, resolving the inconsistency to correct the DCFS directory or the DCFS.

14. The method as recited in claim 7, further comprising at least one of the one or more compute nodes locally storing at least a portion of the DCFS directory.

15. The method as recited in claim 7, further comprising:

receiving a request to rename one or more data objects in the DCFS; and

in response to the request, modifying metadata corresponding to the one or more data objects in the DCFS directory to reflect new names for the one or more data objects.

16. The method as recited in claim 15, wherein said modifying metadata corresponding to the one or more data objects in the DCFS directory to reflect new names for the one or more data objects maintains strong consistency for the metadata in the DCFS directory.

17. The method as recited in claim 15, further comprising, subsequent to said modifying metadata corresponding to the one or more data objects in the DCFS directory to reflect new names for the one or more data objects, performing one or more object storage service operations to obtain eventual consistency of the renamed one or more data objects in the DCFS.

18. The method as recited in claim 7, wherein the database service maintains the DCFS directory as a key-value store or as a relational database.

19. The method as recited in claim 7, wherein keys in the DCFS directory are generated according to a hash technique applied to path information of the data objects in the DCFS, and wherein values in the DCFS directory store the metadata for accessing the data objects stored in the DCFS.

20. The method as recited in claim 19, wherein the hash technique is an MD5 hash technique.

21. A non-transitory computer-accessible storage medium storing program instructions computer-executable to implement:

establishing a distributed computing file system (DCFS) to store data objects via one or more application program interface calls to an eventually consistent unstructured object storage service, wherein the unstructured object storage service is implemented on a first hardware computer system accessible over one or more networks and implements a first client-facing interface accessible to a plurality of clients;

establishing a DCFS directory to store metadata for the data objects in a multi-tenant database service different from the unstructured object storage service, wherein the multi-tenant database service is implemented on a second hardware computer system accessible over the one or more networks and implements a second client-facing interface accessible to the plurality of clients;

responsive to a request to modify a data object of the data objects, modifying the data object in the unstructured object storage service and metadata of the data object in the multi-tenant database service, wherein the unstructured data storage service is not guaranteed to return a latest version of the data object produced by the request via the first client-facing interface, and the multi-tenant database service is guaranteed to return a latest version of the metadata of the data object produced by the request via the second client-facing interface; and

accessing the DCFS directory to obtain consistent metadata for accessing the data objects stored in the DCFS.

22. The non-transitory computer-accessible storage medium as recited in claim 21, wherein the program instructions are further computer-executable to implement accessing the unstructured object storage service to retrieve respective data objects stored in the DCFS according to the metadata obtained from the DCFS directory.

23. The non-transitory computer-accessible storage medium as recited in claim 21, wherein the program instructions are further computer-executable to implement accessing the DCFS directory to update the consistent metadata for accessing the data objects stored in the DCFS.

24. The non-transitory computer-accessible storage medium as recited in claim 21, wherein the DCFS directory is a key-value store, wherein keys in the DCFS directory are generated according to a hash technique applied to path information of the data objects in the DCFS, and wherein values in the DCFS directory store the consistent metadata for accessing the data objects stored in the DCFS.