Maintaining write order fidelity on a multi-writer system

• US 7,475,207 B2
• Filed: 07/14/2006
• Issued: 01/06/2009
• Est. Priority Date: 07/14/2005
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Van der Goot, Roel, Bromling, Steve, Hayward, Geoff, Hagglund, Dale, Karpoff, Wayne
• Primary Examiner(s): Peugh; Brian R
• Application Number:
  US11/486,754
• Publication Number:
  US 20070022264A1
• Time in Patent Office: 907 Days
• Field of Search: 711/162, 711/148, 709/213, 709/214, 709/217
• US Class Current: 711/162
• CPC Class Codes: G06F 11/2058 using more than 2 mirrored copies
• CPC Class Codes: G06F 11/2064 while ensuring consistency
• CPC Class Codes: G06F 11/2071 using a plurality of controllers
• CPC Class Codes: G06F 11/2074 Asynchronous techniques
• CPC Class Codes: G06F 11/2079 Bidirectional techniques
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/568 Storing data temporarily at an intermediate stage, e.g. caching

Abstract

Write order fidelity (WOF) is maintained for totally-active implementations wherein a plurality of access nodes at geographically separated sites can concurrently read and/or write data in a “totally active” fashion on a distributed data system. From the hosts'"'"' perspective at diverse geographic locations, a synchronous, cache-coherent view of data is provided. Data transfer is asynchronous. A time ordered data image is created and maintained so operations can be restarted after a partial system failure that causes loss of data not yet asynchronously transferred across the network, but that has been write-acknowledged to the originating host. Time ordered asynchronous data transfer is implemented as a pipeline of changes that reflect contributions from all nodes. WOF also improves network performance and lowers bandwidth consumption. Extensions can provide, in a totally-active context, features such as point-in-time snapshots, time firewalls, on-demand backend storage allocation, synchronous/asynchronous distribution of data, and continuous data protection.

Claims

1. A method of providing write order fidelity in a distributed set of data access nodes in a network, comprising the steps of:

storing a write request to a first cache for an open delta, the first cache corresponding to a first node receiving the write request;

transmitting a message, in response to a triggering event, to each node in the set of data access nodes to close the open delta;

for each node having a write request for the open delta, completing any pending write requests for the open delta and closing the open delta;

exchanging write request information among nodes so that each node is associated with a complete copy of the write request information for the closed delta;

writing each complete copy to persistent storage;

storing each complete copy of the closed delta to back-end storage for each node; and

grouping front-end volumes into write-order fidelity (WOF) groups, each WOF group including at least one site across the network.

2. A method according to claim 1, wherein:

each site includes at least one of said nodes, wherein all nodes are operable to read and write concurrently.

3. A method according to claim 1, wherein:

storing writes for each WOF group in a cache for the open delta for that WOF group.

4. A method according to claim 1, further comprising:

executing the triggering event.

5. A method according to claim 1, further comprising:

opening a new delta to receive new writes upon closing the open delta.

6. A method according to claim 1, further comprising:

providing localized cached access to remote data for geographically separated nodes.

7. A method according to claim 1, wherein:

writing each complete copy to persistent storage includes writing a metadata update entry to a recovery log, when writing a data update to a database when the metadata update entry writing is completed.

8. A method according to claim 1, further comprising:

reordering write requests in the open delta before closing the open delta.

9. A method according to claim 1, further comprising:

generating the triggering event using a mechanism selected from the group consisting of;

timed intervals, intervals based on the number of transactions, intervals based on the number of changes/writes in a delta set or partial delta, application triggers, operator induced triggers, triggers induced by other subsystems, triggers induced by error conditions, and combinations of the above.

10. A method according to claim 1, wherein:

the network is selected from the group consisting of a storage area network (SAN), a local area network (LAN), a wide are network (WAN), and a metropolitan area network (MAN).

11. A method according to claim 1, wherein:

transmitting a message includes broadcasting the message to each node in the set of data access nodes.

12. A method of providing write order fidelity in a distributed set of data access nodes in a network, comprising the steps of:

storing a write request to a first cache for an open delta, the first cache corresponding to a first node receiving the write request;

transmitting a message, in response to a triggering event, to each node in the set of data access nodes to close the open delta;

for each node having a write request for the open delta, completing any pending write requests for the open delta and closing the open delta;

exchanging write request information among nodes so that each node is associated with a complete copy of the write request information for the closed delta;

writing each complete copy to persistent storage;

storing each complete copy of the closed delta to back-end storage for each node; and

triggering a data snapshot corresponding to the closing of the open delta.

13. A system for providing write order fidelity in a distributed set of data access nodes in a network, comprising:

a storage system for storing data; and

a plurality of access nodes configured to access data in the storage system,

wherein each node in the plurality of access nodes is operable to store a write request to a cache for an open delta, the cache corresponding to the node receiving the write request, each node being further operable to transmit a message, in response to a triggering event, to said plurality of access nodes in the data storage network to complete any write requests and close the open delta, each node being further operable to exchange write request information so that each node is associated with a complete copy of the write request information for the closed delta, each node being further operable to write each complete copy to persistent storage then apply each persistent copy of the closed delta to back-end storage for that node, and the storage system is further operable to create a snapshot of any open deltas at any point in time.

14. A system according to claim 13, wherein:

at least one of the nodes is geographically remote from the other nodes.

15. A system according to claim 13, wherein:

at least one of the access nodes is operable to mirror data to a remote location.

16. A system according to claim 13, further comprising:

a plurality of sites, each site including at least one of said nodes.

17. A system according to claim 13, wherein:

the storage system is further operable to determine when one of the sites is not writing data to the storage system, and operable to set that site as a passive site until that passive site needs to write.

18. A system according to claim 17, wherein:

the storage system is further operable to broadcast a message to the other sites indicating the status of the passive site.

19. A system according to claim 13, wherein:

the storage system is operable to execute asynchronous and synchronous data transfer.

20. A system according to claim 13, wherein:

each node is further operable to exchange write request information by exchanging write request information with a first subset of the plurality of access nodes, whereby the first subset of nodes exchanges the write request information with a second subset of the plurality of access nodes.

21. A system according to claim 13, wherein:

each node is operable to replicate writes to a plurality of caches.

22. A system according to claim 13, wherein:

the data storage system maintains three deltas, each delta representing one of a point in time that data is finally committed to the disk, a point in time that data is about to be committed to the disk, and a point in time at the beginning of the exchange.

23. A system according to claim 13, wherein:

the storage system is further operable to merge deltas over time.

24. A system according to claim 13, wherein:

the storage system is further operable to close delta seta using a mechanism selected from the group consisting of;

timed intervals, intervals based on a number of transactions, intervals based on a number of writes in a delta set or partial delta, application triggers, operator induced triggers, triggers induced by other subsystems, triggers induced by error conditions, and combinations thereof.

25. A system according to claim 13, wherein:

the network is selected from the group consisting of a storage area network (SAN), a local area network (LAN), a wide are network (WAN), and a metropolitan area network (MAN).

26. A system according to claim 13, wherein:

each node is further operable to transmit a message to said plurality of access nodes by broadcasting the message to each node in the set of data access nodes.

27. A system for providing write order fidelity in a distributed set of data access nodes in a network, comprising:

a storage system for storing data;

a plurality of access nodes configured to access data in the storage system; and

a plurality of sites, each site including at least one of said nodes;

wherein each node in the plurality of access nodes is operable to store a write request to a cache for an open delta, the cache corresponding to the node receiving the write request, each node being further operable to transmit a message, in response to a triggering event, to said plurality of access nodes in the data storage network to complete any write requests and close the open delta, each node being further operable to exchange write request information so that each node is associated with a complete copy of the write request information for the closed delta, each node being further operable to write each complete copy to persistent storage then apply each persistent copy of the closed delta to back-end storage for that node; and

wherein the storage system is further operable to determine when only one site is writing data to the storage system, whereby exchanging write request information is suspended until multiple sites are writing to the storage system.

28. A computer program product embedded in a computer readable medium for providing write order fidelity in a distributed set of data access nodes in a network, comprising:

computer program code for storing a write request to a first cache for an open delta, the first cache corresponding to a first node receiving the write request;

computer program code for transmitting a message, in response to a triggering event, to each node in the set of data access nodes to close the open delta;

computer program code for completing any pending write requests for the open delta and closing the open delta for each node having a write request for the open delta;

computer program code for exchanging write request information among nodes so that each node is associated with a complete copy of the write request information for the closed delta;

computer program code for writing each complete copy to persistent storage;

computer program code for storing each complete copy of the closed delta to back-end storage for each node; and

computer program code for grouping front-end volumes into write-order fidelity (WOF) groups, each WOF group including at least one site across the network.

29. A computer program product according to claim 28, further comprising:

computer program code for storing writes for each WOF group in a cache for the open delta for that WOF group.

30. A computer program product according to claim 28, further comprising:

computer program code for providing localized cached access to remote data for geographically separated nodes.

31. A method of providing write order fidelity in a distributed set of data access nodes in a network, comprising the steps of:

providing a plurality of write-order fidelity (WOF) groups, each WOF group including at least one of said data access nodes;

storing a write request to a first cache corresponding to a first node in a first WOF group receiving the write request;

in response to a triggering event, exchanging write request information among the nodes in the first WOF group so that each node is associated with a complete copy of the write request information;

writing each complete copy to persistent storage;

storing each complete copy to back-end storage for each node; and

triggering a data snapshot corresponding to a state of the cached write request for the first WOF group.

32. A method according to claim 31, further comprising:

providing localized cached access to remote data for geographically separated nodes.

33. A method according to claim 31, wherein:

writing each complete copy to persistent storage includes writing a metadata update entry to a recovery log, when writing a data update to a database when the metadata update entry writing is completed.

34. A method of providing write order fidelity in a distributed set of data access nodes in a network, comprising the steps of:

providing a plurality of write-order fidelity (WOF) groups, each WOF group including at least one of said data access nodes;

storing a write request to a cache corresponding to one of the plurality of WOF groups, each WOF group associated with a cache and operable to receive write requests from a plurality of request writers having access to at least one node in the WOF group;

in response to a triggering event for a WOF group, exchanging write request information among the nodes in the WOF group so that each node is associated with a complete copy of the write request information;

storing each complete copy to persistent storage;

storing each complete copy to back-end storage for each node; and

triggering a data snapshot corresponding to a state of the cached write request for the WOF group.

35. A computer program product embedded in a computer readable medium for providing write order fidelity in a distributed set of data access nodes in a network, comprising:

computer program code for providing a plurality of write-order fidelity (WOF) groups, each WOF group including at least one of said data access nodes;

computer program code for storing a write request to a cache corresponding to one of the plurality of WOF groups, each WOF group associated with a cache and operable to receive write requests from a plurality of request writers each having access to at least one node in the WOF group;

computer program code for in response to a triggering event for a WOF group, exchanging write request information among the nodes in the WOF group so that each node is associated with a complete copy of the write request information;

computer program code for storing each complete copy to persistent storage;

computer program code for storing each complete copy to back-end storage for each node; and

computer program code for triggering a data snapshot corresponding to a state of the cached write request for the WOF group.