Coalescing Metadata for Mirroring to a Remote Storage Node in a Cluster Storage System

• US 20140214772A1
• Filed: 01/28/2013
• Published: 07/31/2014
• Est. Priority Date: 01/28/2013
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Kadayam, Harihara S.
• Granted Patent: US 9,805,052 B2
• Time in Patent Office: Days
• US Class Current: 707/655
• CPC Class Codes: G06F 11/2066 Optimisation of the communication load
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Described herein are a system and method for remote mirroring/copying data and metadata sets from a local node to a remote node that reduces the number of metadata sets that are mirrored. In some embodiments, the local node may coalesce metadata sets into metadata chains, each metadata chain comprising a grouping of two or more metadata sets. In some instances, a “representative” metadata set of a metadata chain may be selected for sending to the remote node for storing, wherein the other metadata sets of the metadata chain are not sent to the remote node. In these embodiments, the selected metadata set may represent all the metadata sets in the chain and be the only metadata set in the chain that is transmitted and stored to the remote node. As such, the network congestion between the local and remote nodes may be reduced.

Claims

1. A system for storing data and metadata from a local node to a remote node, the system comprising:

the local node comprising a mirroring engine configured for;

receiving a plurality of data and metadata sets from a client, each received data or metadata set comprising a data or metadata request from the client to store the data or metadata set to the remote node;

producing a plurality of metadata chains, each metadata chain comprising a grouping of at least two received metadata requests;

storing each received metadata request to a metadata request (MR) data structure according to the time order the requests are received, the MR data structure for managing metadata chains, each metadata chain comprising a head metadata request comprising the earliest received metadata request in the metadata chain and a tail metadata request comprising the last received metadata request in the metadata chain; and

transmitting a representative metadata request in the metadata chain to the remote node for storage if the representative metadata request comprises a tail metadata request in the metadata chain.

2. The system of claim 1, wherein the mirroring engine is further configured for:

transmitting each received data request to the remote node for storage.

3. The system of claim 1, wherein:

only the representative metadata request in the metadata chain is transmitted to the remote node for storage and the other metadata requests in the metadata chain are not transmitted to the remote node for storage.

4. The system of claim 1, wherein:

the metadata sets specify a total accumulated number of data sets produced by the client and sent to the mirroring engine; and

the data contained in the representative metadata request encompasses the data contained in the other metadata sets of the metadata chain.

5. The system of claim 1, wherein:

a new received metadata request is grouped with a previous received metadata request to produce a metadata chain if both metadata requests are to be stored to a same storage address location at the remote node; and

the previous metadata request has not yet been transmitted to the remote node.

6. The system of claim 1, wherein:

a new received metadata request is grouped with a previous received metadata request to produce a metadata chain if both metadata requests have the same data size; and

the previous metadata request has not yet been transmitted to the remote node.

7. The system of claim 1, wherein:

a new received metadata request is grouped with a previous received metadata request to produce a metadata chain if both metadata requests are to be stored to a same storage address location at the remote node;

both metadata requests have the same data size; and

the previous metadata request has not yet been transmitted to the remote node.

8. A method for storing data and metadata from a local node to a remote node, the method comprising:

providing the local node comprising a mirroring engine configured for;

receiving a plurality of data and metadata sets from a client, each received data or metadata set comprising a data or metadata request from the client to store the data or metadata set to the remote node;

producing a plurality of metadata chains, each metadata chain comprising a grouping of at least two received metadata requests;

storing each received metadata request to a metadata request (MR) data structure according to the time order the requests are received, the MR data structure for managing metadata chains, each metadata chain comprising a head metadata request comprising the earliest received metadata request in the metadata chain and a tail metadata request comprising the last received metadata request in the metadata chain; and

transmitting a representative metadata request in the metadata chain to the remote node for storage if the representative metadata request comprises a tail metadata request in the metadata chain.

9. The method of claim 8, wherein the mirroring engine is further configured for:

transmitting each received data request to the remote node for storage.

10. The method of claim 8, wherein:

only the representative metadata request in the metadata chain is transmitted to the remote node for storage and the other metadata requests in the metadata chain are not transmitted to the remote node for storage.

11. The method of claim 8, wherein:

the metadata sets specify a total accumulated number of data sets produced by the client and sent to the mirroring engine; and

the data contained in the representative metadata request encompasses the data contained in the other metadata sets of the metadata chain.

12. The method of claim 8, wherein:

a new received metadata request is grouped with a previous received metadata request to produce a metadata chain if both metadata requests are to be stored to a same storage address location at the remote node; and

the previous metadata request has not yet been transmitted to the remote node.

13. The method of claim 8, wherein:

a new received metadata request is grouped with a previous received metadata request to produce a metadata chain if both metadata requests have the same data size; and

the previous metadata request has not yet been transmitted to the remote node.

14. The method of claim 8, wherein:

a new received metadata request is grouped with a previous received metadata request to produce a metadata chain if both metadata requests are to be stored to a same storage address location at the remote node;

both metadata requests have the same data size; and

the previous metadata request has not yet been transmitted to the remote node.

15. A non-transitory computer readable medium having instructions stored thereon when executed by a processor, store data and metadata from a local node to a remote node, the non-transitory computer readable medium comprising instructions for:

on the local node, configuring a mirroring engine for;

receiving a plurality of data and metadata sets from a client, each received data or metadata set comprising a data or metadata request from the client to store the data or metadata set to the remote node;

producing a plurality of metadata chains, each metadata chain comprising a grouping of at least two received metadata requests;

storing each received metadata request to a metadata request (MR) data structure according to the time order the requests are received, the MR data structure for managing metadata chains, each metadata chain comprising a head metadata request comprising the earliest received metadata request in the metadata chain and a tail metadata request comprising the last received metadata request in the metadata chain; and

transmitting a representative metadata request in the metadata chain to the remote node for storage if the representative metadata request comprises a tail metadata request in the metadata chain.

16. The non-transitory computer readable medium of claim 15, wherein the mirroring engine is further configured for:

transmitting each received data request to the remote node for storage.

17. The non-transitory computer readable medium of claim 15, wherein:

only the representative metadata request in the metadata chain is transmitted to the remote node for storage and the other metadata requests in the metadata chain are not transmitted to the remote node for storage.

18. The non-transitory computer readable medium of claim 15, wherein:

the metadata sets specify a total accumulated number of data sets produced by the client and sent to the mirroring engine; and

the data contained in the representative metadata request encompasses the data contained in the other metadata sets of the metadata chain.

19. The non-transitory computer readable medium of claim 15, wherein:

a new received metadata request is grouped with a previous received metadata request to produce a metadata chain if both metadata requests are to be stored to a same storage address location at the remote node; and

the previous metadata request has not yet been transmitted to the remote node.

20. The non-transitory computer readable medium of claim 15, wherein:

a new received metadata request is grouped with a previous received metadata request to produce a metadata chain if both metadata requests have the same data size; and

the previous metadata request has not yet been transmitted to the remote node.

21. The non-transitory computer readable medium of claim 15, wherein:

a new received metadata request is grouped with a previous received metadata request to produce a metadata chain if both metadata requests are to be stored to a same storage address location at the remote node;

both metadata requests have the same data size; and

the previous metadata request has not yet been transmitted to the remote node.