Optimized file system layout for distributed consensus protocol

• US 10,229,009 B2
• Filed: 12/16/2015
• Issued: 03/12/2019
• Est. Priority Date: 12/16/2015
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Muth, John, Gomez, Juan C.
• Primary Examiner(s): Vo, Truong V
• Application Number:
  US14/971,807
• Publication Number:
  US 20170177448A1
• Time in Patent Office: 1,182 Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1448 Management of the data involved in backup or backup restore
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/1471 involving logging of persistent data for recovery
• CPC Class Codes: G06F 11/187 Voting techniques
• CPC Class Codes: G06F 16/128 Details of file system snapshots on the file-level, e.g. snapshot creation, administration, deletion use of snapshots for error detection or correction G06F11/14, G06F11/16
• CPC Class Codes: G06F 16/1865 Transactional file systems
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0652 Erasing, e.g. deleting, data cleaning, moving of data to a wastebasket
• CPC Class Codes: G06F 3/0688 Non-volatile semiconductor memory arrays

Abstract

A layout of a file system is optimized to meet storage requirements of a distributed consensus protocol implemented on a plurality of nodes of a cluster. Illustratively, the file system has an on-disk layout representation that enables efficient input/output (I/O) operation performance. The on-disk layout of the file system embodies a plurality of file system objects: membership, snapshot and log objects. Each object is allocated an area or region (e.g., a contiguous storage space) on-disk. In addition, each object has two or more storage container instances, e.g., files. In the case of snapshot and membership objects, the two files of each object are used in a circular (alternating) fashion so that write operations directed to a first file storing a current copy of the snapshot/membership content may be performed without destroying a previous committed copy of the snapshot/membership content stored in a second file.

Claims

1. A system comprising:

a cluster of storage systems coupled to a network;

a memory coupled to a processor included in each storage system, the memory configured to store a file system executable by the processor that records configuration changes to the cluster via a distributed consensus protocol; and

a storage device connected to each storage system of the cluster and configured to store an on-disk layout of the file system having a plurality of file system objects, each file system object allocated a region on the storage device, each file system object further having two or more storage container instances, wherein the storage container instances for at least one object are used in an alternating arrangement so that write operations directed to a first storage container instance storing a current copy of content is performed without destroying a previous committed copy of the content stored in a second storage container instance.

2. The system of claim 1 wherein the storage container instances are files.

3. The system of claim 1 wherein the storage container instances are files and wherein each file contains a header that describes the file to enable the file system to perform a multi-block read operation to an end of the file instead of having to serially read all blocks using a multitude of operations.

4. The system of claim 1 wherein the plurality of file system objects include membership, snapshot and log objects.

5. The system of claim 1 wherein the plurality of file system object includes a log object written as a whole atomically.

6. The system of claim 1 wherein the at least one object is one of a snapshot and a membership objects, and wherein the content is one of snapshot and membership content.

7. The system of claim 1 wherein at least one file object is protected with an error correcting code that enables the distributed consensus protocol executed by the processor to detect and recover from partial write operations by falling back to the previous committed copy.

8. The system of claim 1 wherein a log region of the on-disk layout is configured to store sequential logs representing a state of the cluster as provided by the distributed consensus protocol.

9. The system of claim 1 wherein the distributed consensus protocol is one of Paxos and Raft.

10. The system of claim 1 wherein configuration updates for the distributed consensus protocol are received at the memory, organized as log records, and persistently stored to a log file of a log region as a sequential log.

11. The system of claim 1 wherein configuration updates for the distributed consensus protocol are stored as a compressed sequential log having a current state of the configuration updates as a snapshot representing a point-in-time image of the log.

12. The system of claim 1 wherein a current state of configuration changes are stored as a snapshot in a snapshot region of the on-disk layout.

13. The system of claim 12 wherein, in response to the current state of the configuration changes stored as a snapshot on the storage device, a storage space of a log file storing the configuration changes prior to the snapshot is reclaimed.

14. A method comprising:

connecting a storage array to a storage system of a cluster of storage systems, the storage system having a processor configured to store a file system executable by the processor to record configuration changes to the cluster via a distributed consensus protocol;

storing an on-disk layout of the file system on the storage array, the on-disk layout of the file system having a plurality of file system objects;

allocating a region on disk for each file system object, each file system object further having two or more files; and

employing the files for at least one object in an alternating arrangement so that write operations directed to a first file storing a current copy of content is performed without destroying a previous committed copy of the content stored in a second file.

15. The method of claim 14 further comprising:

describing each file with a header that enables the file system to perform a multi-block read operation to an end of the file instead of having to serially read all blocks using a multitude of operations.

16. The method of claim 14 wherein the plurality of file system objects include membership, snapshot and log objects.

17. The method of claim 14 further comprising:

protecting at least one file with an error correcting code that enables the distributed consensus protocol to detect and recover from partial write operations by falling back to the previous committed copy.

18. The method of claim 14 further comprising:

storing sequential logs in a log file of a log region of the on-disk layout, the sequential logs representing a state of the cluster as provided by the distributed consensus protocol.

19. The method of claim 14 wherein a log file storing configuration changes is rolled back with a single write operation through use of log entry information maintained in a memory of the storage system.

20. A non-transitory computer readable medium including program instructions for execution on a processor of a storage system of a cluster, the program instructions configured to:

record changes to a cluster configuration via a distributed consensus protocol;

store an on-disk layout of a file system on disks of a storage array connected to the storage system, the on-disk layout of the file system having a plurality of file system objects;

allocate a region on disk for each file system object having two or more files; and

employ the files for at least one object in an alternating arrangement so that write operations directed to a first file storing a current copy of the cluster configuration is performed without destroying a previous committed copy of the cluster configuration stored in a second file.