Replication of remote copy data for internet protocol (IP) transmission

• US 20030217119A1
• Filed: 05/16/2002
• Published: 11/20/2003
• Est. Priority Date: 05/16/2002
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Bergant, Milena, Bono, Jean-Pierre, Gupta, Uday K., Armangau, Philippe, Raman, Suchitra, Angelone, Raymond A., Vahalia, Uresh
• Granted Patent: US 7,546,364 B2
• Time in Patent Office: Days
• US Class Current: 709/219
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]

Abstract

Consistent updates are made automatically over a wide-area IP network, concurrently with read-only access to the remote copies. A replication control protocol (RCP) is layered over TCP/IP providing the capability for a remote site to replicate and rebroadcast blocks of the remote copy data to specified groups of destinations, as configured in a routing table. A volume multicast layer over RCP provides for multicasting to specified volume extents of the blocks. The blocks are copied at the logical level, so that it does not matter what physical structure is used for storing the remote copies. Save volumes buffer the remote copy data transmitted between the primary or secondary file system volume and the IP network, in order to ensure independence between the replication process, the IP transport method, and the primary file system being replicated.

Claims

1. In a data processing system having a plurality of host computers linked by an Internet Protocol (IP) network to a plurality of data storage systems, each of the data storage systems having data storage and at least one data mover computer for moving data between the data storage and the IP network, a method of distributing remote copy data over the IP network from a primary one of the data mover computers to a plurality of secondary ones of the data mover computers, wherein the method comprises:

the primary data mover computer sending the remote copy data over the IP network to at least one forwarder data mover computer, and the forwarder data mover computer routing the remote copy data over the IP network to the plurality of secondary data mover computers.

2. The method as claimed in claim 1, wherein the forwarder data mover computer is programmed with a TCP/IP layer and a replication control protocol layer over the TCP/IP layer, and the replication control protocol layer produces an inbound session and an outbound session for referencing a routing table identifying destinations in the IP network for the secondary data mover computers and for retransmitting the remote copy data over the IP network from the forwarder data mover computer to the secondary data mover computers.

3. The method as claimed in claim 2, which further includes the forwarder data mover computer temporarily storing the remote copy data from the TCP/IP layer in a buffer, and the inbound session passing pointers to the remote copy data in the buffer to the outbound session for the retransmission of the remote copy data in the buffer over the IP network from the forwarder data mover computer to the secondary data mover computers.

4. The method as claimed in claim 1, wherein the primary data mover computer manages a primary file system in the data storage of the storage system including the primary data mover computer, each secondary data mover computer manages a secondary file system in the data storage of the data storage system including said each secondary data mover computer, and each secondary file system is maintained as a remote copy of the primary file system.

5. The method as claimed in claim 4, wherein the secondary data mover computers provide read-only access of at least some of the host computers to consistent views of the secondary file systems when the primary data mover computer provides concurrent write access to the primary file system.

6. The method as claimed in claim 4, wherein at least one save volume is used to buffer the remote copy data transmitted from the primary data mover computer to the forwarder data mover computer.

7. The method as claimed in claim 6, wherein the save volume stores a record of progress of the transmission of the remote copy data over the IP network from the primary data mover computer to the forwarder data mover computer in order to maintain consistency of the replication process upon panic, reboot, and recovery.

8. The method as claimed in claim 6, wherein the process of transporting the remote copy data over the IP network from the primary data mover computer to the forwarder data mover computer does not depend on any replication information in memory other than the information in the save volume, so as to permit the replication process to be started or terminated on any data mover computer for load shifting or load balancing.

9. The method as claimed in claim 6, wherein a timeout parameter and a high water mark parameter are used as triggers for sending sets of the remote copy data from the save volume over the IP network to the forwarder data mover computer.

10. The method as claimed in claim 4, which includes operating a replication service that transmits modifications of the primary file system to the secondary file systems, and then creating a new secondary file system by copying the primary file system to the new secondary file system concurrent with the operation of the replication service, and after the primary file system has been copied to the new secondary file system, updating the new secondary file system with modifications transmitted by the replication service from the primary file system.

11. The method as claimed in claim 10, wherein the replication service transmits modifications from the primary file system to a save volume during the copying of the primary file system to the new secondary file system, and upon completion of the copying of the primary file system to the new secondary file system, the modifications are copied from the save volume to the new secondary file system.

12. The method as claimed in claim 11, wherein the new secondary file system is at a remote site and the save volume is at the remote site, and the method includes transmitting at least a portion of the modifications of the primary file system to the save volume concurrently with the copying of the primary file system to the new secondary file system.

13. The method as claimed in claim 11, wherein the transmission of modifications of the primary file system to the new secondary file system includes transmitting delta sets of the modifications of the primary file system, and

the copying of the primary file system to the new secondary file system includes creating a snapshot copy of the primary file system when a new delta set begins, and creating the secondary file system from the snapshot copy of the primary file system, and

upon completion of the copying of the primary file system to the new secondary file system, the modifications are copied from the save volume to the new secondary file system beginning with the new delta set.

14. The method as claimed in claim 4, wherein the remote copy is a copy by extent, so that there is not a remote copy of data blocks that are not used in the primary file system.

15. The method as claimed in claim 14, wherein the data blocks are remote copied at a logical level, so that it does not matter what physical structure is used for storing the data blocks in the secondary file systems.

16. The method as claimed in claim 1, wherein only one TCP port to the IP network is used in the forwarder data mover computer for receiving and transmitting the remote copy data to and from the network, and the one TCP port is shared with HTTP connections.

17. The method as claimed in claim 1, wherein a TCP port to the IP network is used in the forwarder data mover computer for receiving the remote copy data from the network, the TCP port is shared with HTTP connections, a level 5 filter in the forwarder data mover computer passes IP packets of HTTP connections from the TCP port to an HTTP collector service in the forwarder data mover computer, and the level 5 filter passes IP packets of the remote copy data from the TCP port to a replication collector service in the forwarder data mover computer.

18. A data processing system comprising:

a plurality of data storage systems linked by an Internet Protocol (IP) network for access by a plurality of host computers, each of the storage systems having data storage and at least one data mover computer for moving data between the data storage and the IP network, the data mover computers including means for distributing remote copy data over the IP network from a primary one of the data mover computers to a plurality of secondary ones of the data mover computers by the primary data mover computer sending the remote copy data over the IP network to at least one forwarder data mover computer, and the forwarder data mover computer routing the remote copy data over the IP network to the plurality of secondary data mover computers.

19. The data processing system as claimed in claim 18, wherein the forwarder data mover computer is programmed with a TCP/IP layer and a replication control protocol layer over the TCP/IP layer, and the replication control protocol produces an inbound session and an outbound session for referencing a routing table identifying destinations in the IP network for the secondary data mover computers and for retransmitting the remote copy data over the IP network from the forwarder data mover computer to the secondary data mover computers.

20. The data processing system as claimed in claim 19, which further includes a buffer for temporarily storing the remote copy data from the TCP/IP layer, and the inbound session passes pointers to the remote copy data in the buffer to the outbound session for the retransmission of the remote copy data in the buffer over the IP network from the forwarder data mover computer to the secondary data mover computers.

21. The data processing system as claimed in claim 18, wherein the primary data mover computer includes means for managing a primary file system in the data storage of the storage system including the primary data mover computer, and each secondary data mover computer includes means for managing a secondary file system in the data storage of the data storage system including said each secondary data mover computer by maintaining each secondary file system as a remote copy of the primary file system.

22. The data processing system as claimed in claim 21, wherein the secondary data mover computers include means for providing read-only access of at least some of the host computers to consistent views of the secondary file systems when the primary data mover computer provides concurrent write access to the primary file system.

23. The data processing system as claimed in claim 21, which includes at least one save volume for buffering the remote copy data transmitted from the primary data mover computer to the forwarder data mover computer.

24. The data processing system as claimed in claim 23, which includes means for recording, in the save volume, a record of progress of the transmission of the remote copy data over the IP network from the primary data mover computer to the forwarder data mover computer in order to maintain consistency of the replication process upon panic, reboot, and recovery.

25. The data processing system as claimed in claim 23, which includes means for transporting the remote copy data over the IP network from the primary data mover computer to the forwarder data mover computer without dependence on any replication information in memory other than the information in the save volume, so as to permit the replication process to be started or terminated on any data mover computer for load shifting or load balancing.

26. The data processing system as claimed in claim 23, which includes means for using a timeout parameter and a high water mark parameter as triggers for sending sets of the remote copy data from the save volume over the IP network to the forwarder data mover computer.

27. The data processing system as claimed in claim 21, which includes a replication service for transmitting modifications of the primary file system to the secondary file systems, and means for creating a new secondary file system by copying the primary file system to the new secondary file system concurrent with the operation of the replication service, and after the primary file system has been copied to the new secondary file system, updating the new secondary file system with modifications transmitted by the replication service from the primary file system.

28. The data processing system as claimed in claim 27, wherein the replication service transmits modifications from the primary file system to a save volume during the copying of the primary file system to the new secondary file system, and upon completion of the copying of the primary file system to the new secondary file system, the modifications are copied from the save volume to the new secondary file system.

29. The data processing system as claimed in claim 28, wherein the new secondary file system is at a remote site and the save volume is at the remote site, and the replication service transmits at least a portion of the modifications of the primary file system to the save volume concurrently with the copying of the primary file system to the new secondary file system.

30. The data processing system as claimed in claim 28, wherein the transmission of modifications of the primary file system to the new secondary file system includes transmitting delta sets of the modifications of the primary file system, and

the copying of the primary file system to the new secondary file system includes creating a snapshot copy of the primary file system when a new delta set begins, and creating the secondary file system from the snapshot copy of the primary file system, and

upon completion of the copying of the primary file system to the new secondary file system, the modifications are copied from the save volume to the new secondary file system beginning with the new delta set.

31. The data processing system as claimed in claim 21, wherein the remote copy is a copy by extent, so that there is not a remote copy of data blocks that are not used in the primary file system.

32. The data processing system as claimed in claim 31, wherein the data blocks are remote copied at a logical level, so that it does not matter what physical structure is used for storing the data blocks in the secondary file systems.

33. The data processing system as claimed in claim 18, wherein the forwarder data mover computer includes only one TCP port to the IP network for receiving and transmitting the remote copy data to and from the network, the one TCP port also servicing HTTP connections.

34. The data processing system as claimed in claim 18, wherein the forwarder data mover computer further includes an HTTP collector service, a replication collector service, a TCP port to the IP network for receiving IP packets of the remote copy data from the network and for receiving IP packets of HTTP connections, and a level 5 filter for passing the IP packets of the HTTP connections from the TCP port to the HTTP collector service and passing the IP packets of the remote copy data from the TCP port to the replication collector service.

35. A server for an Internet Protocol (IP) network, the server being programmed with a routing table, a TCP/IP layer, and a replication control protocol (RCP) session layer over the TCP/IP layer, the routing table identifying destinations in the network for remote copy data, and replication control protocol session layer being programmed to produce an inbound session in response to the file server receiving remote copy data from a source in the IP network, and an outbound session for transmitting the remote copy data to a plurality of destinations identified in the routing table as destinations for the remote copy data from the source.

36. The server as claimed in claim 35, which includes data storage for storing a file of the remote copy data, the server being programmed as a network file server for responding to file access requests from host computers in the IP network.

37. The server as claimed in claim 35, which further includes a buffer for temporarily storing the remote copy data from the TCP/IP layer in a buffer, and the inbound session passing pointers to the remote copy data in the buffer for the retransmission of the remote copy data in the buffer over the IP network from the forwarder data mover computer to the secondary data mover computers.

38. The server as claimed in claim 35, which further includes only one TCP port to the IP network for receiving and transmitting the remote copy data to and from the network, the one TCP port also servicing HTTP connections.

39. The server as claimed in claim 35, which further includes an HTTP collector service, a TCP port to the IP network for receiving IP packets of the remote copy data from the network and for receiving IP packets of HTTP connections, and a level 5 filter for passing the IP packets of the HTTP connections from the TCP port to the HTTP collector service and passing the IP packets of the remote copy data from the TCP port to the RCP layer.

40. A primary data storage system for distributing remote copy data over an Internet Protocol (IP) network to at least one secondary data storage system in the IP network, the primary data storage system including data storage and a data mover computer for moving data between the IP network and the data storage, wherein the data storage includes a primary volume including a primary copy of the remote copy data, and a save volume used as a buffer between the primary volume and the IP network, and the data mover computer is programmed with a TCP/IP layer, a replication control protocol (RCP) layer over the TCP/IP layer for transmitting blocks of data from the save volume over the IP network, and a replication module for writing modified blocks of the primary volume to the save volume.

41. The primary data storage system as claimed in claim 40, wherein the data mover computer is programmed with a volume multicast layer over the RCP layer for transmitting volume extents of the data blocks from the save volume to the RCP layer.

42. The primary data storage system as claimed in claim 40, wherein the data mover computer is programmed with a copy send-thread for copying blocks from the primary volume to the RCP layer.

43. The primary data storage system as claimed in claim 42, wherein the data mover computer is programmed with a volume multicast layer over the RCP layer for transmitting volume extents of the data blocks from the copy-send thread to the RCP layer.

44. The primary data storage system as claimed in claim 42, wherein the copy send-thread maintains a bit map of copied blocks, and the replication module maintains a log of copied blocks that been modified since they have been copied by the copy send-thread.

45. The primary data storage system as claimed in claim 40, wherein the replication module maintains a log indicating modified blocks of the primary volume, and repetitively accesses the log to copy the modified blocks from the primary volume to the save volume.

46. The primary data storage system as claimed in claim 45, wherein the replication module copies the modified blocks from the primary volume to the save volume when a timeout interval expires and when the number of modified blocks not yet copied from the primary volume to the save volume exceeds a threshold.

47. The primary data storage system as claimed in claim 40, wherein the RCP layer includes:

a session manager for managing a session queue;

collector and worker threads; and

a single-thread daemon for access to and from a TCP port.

48. The primary data storage system as claimed in claim 40, wherein the RCP layer accesses only one TCP port to the IP network for transmitting the remote copy data to the network, the one TCP port also servicing HTTP connections.

49. The primary data storage system as claimed in claim 48, which further includes an HTTP collector service, a TCP port to the IP network for receiving IP packets of replication connections and for receiving IP packets of HTTP connections, and a level 5 filter for passing the IP packets of the HTTP connections from the TCP port to the HTTP collector service and passing the IP packets of the replication connections from the TCP port to the RCP layer.

50. A secondary data storage system for receiving remote copy data distributed over an Internet Protocol (IP) network from a primary data storage system, the remote copy data including modified blocks of a primary volume in the primary data storage system, the secondary data storage system including data storage and a data mover computer for moving data between the IP network and the data storage, wherein the data storage includes a secondary volume including a secondary copy of the primary volume, and a save volume used as a buffer between the IP network and the secondary volume for buffering the modified blocks in the remote copy data, and wherein the data mover computer is programmed with a TCP/IP layer, a replication control protocol (RCP) layer over the TCP/IP layer for transmitting the modified blocks of remote copy data from the IP network to the save volume, and a playback module for writing the modified blocks of the remote copy data to the save volume.

51. The secondary data storage system as claimed in claim 50, wherein the data mover computer is programmed with a volume multicast layer over the RCP layer for transmitting volume extents of the modified blocks in the remote copy data from the RCP layer to the save volume.

52. The secondary data storage system as claimed in claim 50, wherein the data mover computer is programmed with a volume multicast layer over the RCP layer for transmitting volume extents of the data blocks from the RCP layer to the secondary volume during a remote copy from the primary volume to the secondary volume before transmission of the modified blocks of remote copy data from the primary storage system to the secondary storage system.

53. The secondary data storage system as claimed in claim 50, wherein the RCP layer includes:

a session manager for managing a session queue;

collector and worker threads; and

a single-thread daemon for access to and from a TCP port.

54. The secondary data storage system as claimed in claim 50, wherein the RCP layer accesses only one TCP port to the IP network for receiving the remote copy data from the network, the one TCP port also servicing HTTP connections.

55. The secondary data storage system as claimed in claim 50, which further includes an HTTP collector service, a TCP port to the IP network for receiving IP packets of replication connections and for receiving IP packets of HTTP connections, and a level 5 filter for passing the IP packets of the HTTP connections from the TCP port to the HTTP collector service and passing the IP packets of the replication connections from the TCP port to the RCP layer.

56. A network file server for use in an Internet Protocol (IP) network, the network file server having data storage including a file system volume for storing a file system, and a TCP port for connection to the IP network to permit access from the IP network to the file system, the network file server being programmed with a series of protocol layers including:

a TCP/IP layer for access to the IP network through the TCP port in accordance with the standard Transmission Control Protocol;

a replication control protocol (RCP) session layer over the TCP/IP layer for transmission, forwarding, and reception of blocks of remote copy data in accordance with a replication control protocol in which the blocks of remote copy data are transmitted and forwarded to specified groups of destinations in the IP network, the network file server having a routing table configured with the groups of destinations, and the RCP layer accessing the routing table to determine the destinations in the specified groups for transmission or forwarding; and

a volume multicast layer over the RCP layer for transmission or reception of specified volume extents of blocks between the file system volume and the IP network.

57. The network file server as claimed in claim 56, wherein the network file server is further programmed to perform a remote volume copy operation by copying volume extents of the file system between the file system volume and the volume multicast layer.

58. The network file server as claimed in claim 56, wherein the network file server further includes a save volume, and the network file server is further programmed to operate in a remote replication mode by transferring modifications of the file system between the file system volume and the volume multicast layer, wherein the modifications are temporarily stored in the save volume to buffer the modifications as the modifications are transferred between the file system volume and the multicast layer.

59. The network file server as claimed in claim 56, wherein the network file server includes more than one data mover computer for moving data from the data storage to the IP network, each data mover computer manages access to a respective remote copy of a file system in the data storage, and the network file server is programmed to perform local replication of file system modifications received from the IP network by updating each remote copy of the file system with the file system modifications.

60. The network file server as claimed in claim 56, wherein the RCP layer includes:

a session manager for managing a session queue;

collector and worker threads; and

a single-thread daemon for access to and from a TCP port.

61. The network file server as claimed in claim 56, wherein the RCP layer accesses only one TCP port to the IP network for receiving the remote copy data from the network, the one TCP port also servicing HTTP connections.

62. The network file server as claimed in claim 56, which further includes an HTTP collector service, a TCP port to the IP network for receiving IP packets of replication connections and for receiving IP packets of HTTP connections, and a level 5 filter for passing the IP packets of the HTTP connections from the TCP port to the HTTP collector service and passing the IP packets of the replication connections from the TCP port to the RCP layer.