PUSH-BASED PIGGYBACK SYSTEM FOR SOURCE-DRIVEN LOGICAL REPLICATION IN A STORAGE ENVIRONMENT

• US 20150347046A1
• Filed: 12/31/2014
• Published: 12/03/2015
• Est. Priority Date: 12/14/2012
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Wang, Xin, Deshmukh, Vijay, Komatsu, Kiyoshi, Curley, Jonathan, Beegle, Laura, Simharajan, Srishylam, Singh, Gurmeet
• Granted Patent: US 9,916,100 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/184 implemented as replicated file system
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

The disclosed techniques enable push-based piggybacking of a source-driven logical replication system. Logical replication of a data set (e.g., a snapshot) from a source node to a destination node can be achieved from a source-driven system while preserving the effects of storage efficiency operations (deduplication) applied at the source node. However, if missing data extents are detected at the destination, the destination has an extent pulling problem as the destination may not have knowledge of the physical layout on the source-side and/or mechanisms for requesting extents. The techniques overcome the extent pulling problem in a source-driven replication system by introducing specific protocols for obtaining missing extents within an existing replication environment by piggybacking data pushes from the source.

Claims

1. A non-transitory computer readable storage medium having instructions stored thereon, which when executed by one or more processors of a destination node of a source-driven replication system, cause the destination node to:

identify a missing extent of a plurality of extents associated with a replicated data set,

wherein when the replicated data set is reconstructed at the destination node with the same logical layout but a different physical layout than the original data set at a source node; and

in response to a push inquiry, send, for delivery to the source node, a push inquiry response indicating the missing extent;

process a data stream pushed from the source node, the data stream including the missing extent,

wherein the push inquiry response causes the source node to initiate a push of the missing extent.

2. The non-transitory computer readable storage medium of claim 1 wherein the instructions, when executed by the one or more processors of the destination node, further cause the destination node to:

reconstruct the replicated data set from the plurality of extents in response to reception of the missing extent.

3. The non-transitory computer readable storage medium of claim 1, wherein the data stream further includes a piggyback flag indicating that the missing extent was indicated as missing in the push inquiry response.

4. The non-transitory computer readable storage medium of claim 1 wherein the instructions, when executed by the one or more processors of the destination node, further cause the destination node to:

determine if the missing extent has already been requested via the push inquiry response by looking up the missing extent in a data structure of missing extents,

wherein the data structure includes missing extent identifiers and an indication as to whether the missing extents have been indicated as missing in a push inquiry response.

5. The non-transitory computer readable storage medium of claim 4 wherein the instructions, when executed by the one or more processors of the destination node, further cause the destination node to:

add the missing extent to the data structure of missing extents if the missing extent has not been requested via the push inquiry response.

6. The non-transitory computer readable storage medium of claim 4, wherein the missing extent identifiers are sorted in the data structure for searching in O(log N).

7. The non-transitory computer readable storage medium of claim 1, wherein the replicated data set is deduplicated at a source node of the source-driven replication system and, when the replicated data set is reconstructed at the destination node, the replicated data set maintains the deduplication.

8. The non-transitory computer readable storage medium of claim 1 wherein the instructions, when executed by the one or more processors of the destination node, further cause the destination node to:

receiving, at the destination node, a request for the missing block from a destination module.

9. The non-transitory computer readable storage medium of claim 1, wherein the data set comprises a point-in-time image or point-in-time image update.

10. A method comprising:

sending, by a source node of a source-driven replication system for delivery to a destination node, a first data stream including a plurality of extents associated with a data set,

wherein the data set is deduplicated at the source node of the source-driven replication system and, when the data set is reconstructed at the destination node, the data set maintains the deduplication;

sending, by the source node for delivery to the destination node, a push inquiry to identify if an extent of the plurality of extents associated with the data set is missing during logical reconstruction at the destination node, wherein when reconstructed at the destination node, the data set has a same logical layout but a different physical layout than the data set at the source node;

in response to sending the push inquiry, receiving, by the source node, a push inquiry response initiated by the destination node, the push inquiry response identifying the missing extent,

in response to the push inquiry response identifying the missing extent, sending, by the source node for delivery to the destination node, a second data stream including the missing extent.

11. The method of claim 10, wherein the second data stream includes a piggyback flag indicating that the missing extent was indicated as missing in the push inquiry response.

12. The method of claim 10, wherein a data structure of missing extents is maintained at the destination node, wherein the data structure includes missing extent identifiers and an indication as to whether the missing extents have been indicated as missing in a push inquiry response.

13. The method of claim 12, further comprising:

prior to sending the second data stream, sorting the missing extents in response to the push inquiry response identifying the missing extent.

14. A source node of a source-driven replication system, the node comprising:

one or more processors;

a storage interface, operatively coupled to the one or more processors, through which to access a plurality of mass storage devices;

a communication interface, operatively coupled to the one or more processors, through which to communicate with a destination node;

a source replication module operatively coupled to the one or more processors and configured to send a first data stream including a plurality of extents associated with a data set, wherein the data set is deduplicated at the source node of the source-driven replication system and, when the data set is reconstructed at the destination node, the data set maintains the deduplication; and

a source piggyback module operatively coupled to the one or more processors and the source replication module and configured to;

send a push inquiry to identify if an extent of the plurality of extents associated with the data set is missing during logical reconstruction at the destination node, wherein when reconstructed at the destination node, the data set has a same logical layout but a different physical layout than the data set at the source node,

in response to sending the push inquiry, receive a push inquiry response initiated by the destination node, the push inquiry response identifying the missing extent, and

in response to the push inquiry response identifying the missing extent, send, for delivery to the destination node, a second data stream including the missing extent.

15. The source node of claim 14, wherein source piggyback destination replication module is further configured to sort the extents in response to the push inquiry response identifying the missing extent.

16. The source node of claim 14, wherein the second data stream includes a piggyback flag indicating that the missing extent was indicated as missing in the push inquiry response.

17. The source node of claim 14, wherein a destination piggyback module at the destination node determines if the missing extent has already been requested via the push inquiry response by looking up the missing extent in a data structure of missing extents, wherein the data structure includes missing extent identifiers and an indication as to whether the missing extents have been indicated as missing in a push inquiry response, and add the missing extent to the data structure of missing extents if the missing extent has not been requested via the push inquiry response.

18. The source node of claim 17, wherein the missing extent identifiers are sorted in O(N*log N) in the data structure for searching in O(log N).

19. The source node of claim 14, wherein the source piggyback module is further configured to read the missing extent from a source file system in response to receiving the push inquiry response.

20. The source node of claim 14, wherein the source replication module is further configured to deduplicate the data set and, when the replicated data set is reconstructed at the destination node, the replicated data set maintains the deduplication.