Fault tolerance in a distributed file system

• US 10,133,646 B1
• Filed: 03/24/2017
• Issued: 11/20/2018
• Est. Priority Date: 06/25/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Joseph, Rejith George, Lee, Tin-Yu, Chheda, Bandish N., Le Grand, Scott Michael, Baji, Saurabh Dileep
• Primary Examiner(s): McCarthy, Christopher S
• Application Number:
  US15/468,708
• Time in Patent Office: 606 Days
• CPC Class Codes: G06F 11/14 Error detection or correction of the data by redundancy in operation G06F11/16 takes precedence
• CPC Class Codes: G06F 11/1438 Restarting or rejuvenating
• CPC Class Codes: G06F 11/1469 Backup restoration techniques
• CPC Class Codes: G06F 11/1484 involving virtual machines
• CPC Class Codes: G06F 11/2007 using redundant communication media
• CPC Class Codes: G06F 11/203 using migration
• CPC Class Codes: G06F 11/2033 switching over of hardware resources
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 2009/45579 I/O management, e.g. providing access to device drivers or storage
• CPC Class Codes: G06F 2009/45591 Monitoring or debugging support
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 9/45533 Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects

Abstract

A method for providing fault tolerance in a distributed file system of a service provider may include launching at least one data storage node on at least a first virtual machine instance (VMI) running on one or more servers of the service provider and storing file data. At least one data management node may be launched on at least a second VMI running on the one or more servers of the service provider. The at least second VMI may be associated with a dedicated IP address and the at least one data management node may store metadata information associated with the file data in a network storage attached to the at least second VMI. Upon detecting a failure of the at least second VMI, the at least one data management node may be re-launched on at least a third VMI running on the one or more servers.

Claims

1. A computer-readable storage having instructions thereon, which if performed by a computer, cause the computer to at least:

replace one or more database nodes, in a distributed file system, with a new database node using metadata stored in a network storage in response to detecting one or more failures in the one or more database nodes, wherein the new database node is a data management node within the distributed file system including a plurality of data storage nodes, and wherein the metadata is associated with file data on the plurality of data storage nodes; and

attach the network storage to the data management node so that the data management node can access the metadata.

2. The computer-readable storage of claim 1, wherein the instructions further cause the computer to at least detect a failed node, wherein the failed node is attached to the network storage.

3. The computer-readable storage of claim 1, wherein the replacement of the one or more database nodes includes instructions that cause the computer to launch a new virtual machine instance associated with the new database node.

4. A method, comprising:

detecting a failure on a database node amongst a plurality of database nodes;

replacing the failed database node with a new database node using metadata stored in a network storage;

wherein the metadata comprises;

information identifying one or more file blocks of a file; and

information identifying how many times the one or more file blocks are replicated.

5. The method of claim 4, further including attaching the network storage to the plurality of database nodes to collect metadata in case of failure.

6. The method of claim 4, after replacing the failed database node with the new database node, attaching the network storage to the new database node.

7. The method of claim 4, wherein the replacing includes directing network traffic to the new database node in place of the failed database node.

8. The method of claim 4, wherein the replacing includes relaunching the failed database node on a virtual machine instance executing on a server computer.

9. The method of claim 4, wherein the failed database node is a virtual machine instance executing on a server computer in a service provider environment.

10. The method of claim 4, wherein the metadata includes a command log file comprising at least one instruction for updating the metadata.

11. The method of claim 4, wherein the network storage stores metadata associated with the plurality of database nodes.

12. The method of claim 4, wherein the plurality of database nodes are in a distributed file system wherein files are divided into blocks with multiple copies of blocks stored across multiple nodes.

13. A system, comprising:

a plurality of host server computers coupled together through a network to form a distributed file system, wherein files are divided into blocks with multiple copies of blocks stored across multiple nodes; and

a fault tolerance server operable to;

replace one or more database nodes with a new database node using metadata stored in a network storage within the distributed file system in response to detecting one or more failures in the one or more database nodes;

wherein the metadata comprises;

information identifying one or more file blocks of the files; and

information identifying how many times the one or more file blocks are replicated.

14. The system of claim 13, wherein the replacement of the one or more database nodes includes directing network traffic to the new database node in place of a failed database node.

15. The system of claim 13, wherein the fault tolerance service associates an IP address of a failed node with an IP address of the new database node.

16. The system of claim 13, wherein the failed database node is a virtual machine instance executing on a server computer in a service provider environment.