BUFFERING AND REPLICATING DATA WRITTEN TO A DISTRIBUTED STORAGE SYSTEM

• US 20180089222A1
• Filed: 12/04/2017
• Published: 03/29/2018
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Hildebrand, Dean, Schmuck, Frank B.
• Granted Patent: US 10,909,084 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/1805 Append-only file systems, e.g. using logs or journals to store data

Abstract

Embodiments of the invention relate to small write performance enhancements for parallel file systems. One embodiment includes flushing a received write transaction from a first memory device to a second memory device on a node in a file system based on one or more byte-ranges within a file system block. It is determined to flush the received write transaction to a recovery log that is stored in a non-volatile storage layer of the file system. If it is determined to flush the received write transaction to the recovery log: appending data associated with the received write transaction in the recovery log of the file system, replicating the data associated with the received write transaction in another non-volatile storage layer of another node if required, and marking the one or more byte-ranges as committed.

Claims

1. A method comprising:

storing one or more data updates in a non-volatile storage layer of a file system;

initiating replication of the one or more data updates on one or more servers;

gathering the one or more data updates into one or more memory chunks in a buffer of the file system; and

flushing the buffer by writing the one or more memory chunks in the buffer to permanent storage.

2. The method of claim 1, further comprising:

clearing the one or more data updates from the non-volatile storage layer of the file system in response to flushing the buffer.

3. The method of claim 1, wherein each data updates is one of an application update or a file system metadata update.

4. The method of claim 1, wherein the replication and the buffer reduces latency of each data update.

5. The method of claim 1, wherein the one or more memory chunks comprise one or more contiguous memory blocks.

6. The method of claim 1, further comprising:

flushing one or more replicated copies of the one or more data updates from the one or more server to the permanent storage as a recovery mechanism in response to a node failure of the file system.

7. A system comprising a computer processor, a computer-readable hardware storage device, and program code embodied with the computer-readable hardware storage device for execution by the computer processor to implement a method comprising:

storing one or more data updates in a non-volatile storage layer of a file system;

initiating replication of the one or more data updates on one or more servers;

gathering the one or more data updates into one or more memory chunks in a buffer of the file system; and

flushing the buffer by writing the one or more memory chunks in the buffer to permanent storage.

8. The system of claim 7, wherein the method further comprises:

clearing the one or more data updates from the non-volatile storage layer of the file system in response to flushing the buffer.

9. The system of claim 7, wherein each data updates is one of an application update or a file system metadata update.

10. The system of claim 7, wherein the replication and the buffer reduces latency of each data update.

11. The system of claim 7, wherein the one or more memory chunks comprise one or more contiguous memory blocks.

12. The system of claim 7, wherein the method further comprises:

flushing one or more replicated copies of the one or more data updates from the one or more server to the permanent storage as a recovery mechanism in response to a node failure of the file system.

13. A computer program product comprising a computer-readable hardware storage device having program code embodied therewith, the program code being executable by a computer to implement a method comprising:

storing one or more data updates in a non-volatile storage layer of a file system;

initiating replication of the one or more data updates on one or more servers;

gathering the one or more data updates into one or more memory chunks in a buffer of the file system; and

flushing the buffer by writing the one or more memory chunks in the buffer to permanent storage.

14. The computer program product of claim 13, wherein the method further comprises:

clearing the one or more data updates from the non-volatile storage layer of the file computer program product in response to flushing the buffer.

15. The computer program product of claim 13, wherein each data updates is one of an application update or a file computer program product metadata update.

16. The computer program product of claim 13, wherein the replication and the buffer reduces latency of each data update.

17. The computer program product of claim 13, wherein the one or more memory chunks comprise one or more contiguous memory blocks.

18. The computer program product of claim 13, wherein the method further comprises:

flushing one or more replicated copies of the one or more data updates from the one or more server to the permanent storage as a recovery mechanism in response to a node failure of the file computer program product.