Remote copy control method, storage sub-system with the method, and large area data storage system using them

• US 20030051111A1
• Filed: 03/08/2002
• Published: 03/13/2003
• Est. Priority Date: 08/08/2001
• Status: Active Grant

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Okami, Yoshinori, Nakano, Toshio, Higaki, Seiichi, Nakamura, Katsunori, Ogata, Mikito
• Granted Patent: US 7,082,506 B2
• Time in Patent Office: Days
• US Class Current: 711/162
• CPC Class Codes: G06F 11/2058 using more than 2 mirrored copies
• CPC Class Codes: G06F 11/2064 while ensuring consistency
• CPC Class Codes: G06F 11/2071 using a plurality of controllers
• CPC Class Codes: G06F 11/2074 Asynchronous techniques
• CPC Class Codes: G06F 11/2076 Synchronous techniques
• CPC Class Codes: G06F 11/2079 Bidirectional techniques
• CPC Class Codes: G06F 11/2082 Data synchronisation
• CPC Class Codes: Y10S 707/99953 Recoverability
• CPC Class Codes: Y10S 707/99955 Archiving or backup

Abstract

Two data centers located in the vicinity are connected using a synchronous transfer copy function, and one of the data centers is coupled with a third data center disposed at a remote location by an asynchronous remote copying function. The order whereat a storage sub-system located in the vicinity has received data from a host is consistently guaranteed, and the third data center holds the data. Further, each storage sub-system includes a function whereby, during normal operation, data can be exchanged and the data update state can be obtained by the storage sub-systems located in the two data centers that do not directly engage in data transmission.

Claims

1. A remote copy control method, for a storage sub-system which includes a control memory for storing control data, a cache memory for temporarily storing data, a micro processor for controlling said control memory and said cache memory, N transmission destinations for synchronously receiving said data from said micro processor, and M transmission destinations for asynchronously receiving said data from said microprocessor, said remote copy control method comprising:

a first step of storing, in said control memory, transfer states/bit maps corresponding to said N+M data destinations;

a second step of another storage sub-system, which includes a transfer state/bit map that corresponds to one of said transfer states/bit maps, issuing to a storage sub-system that does not directly perform data transmission a command for transmitting an inquiry for a data update process state; and

a third step of, upon receiving a response to said command at said second step, updating said transfer states/bit maps.

2. A remote copy control method according to claim 1, wherein the updating of said transfer states/bit maps at said third step includes the updating of a counter value that is included in said command for transmitting an inquiry for said data update process state, and that is used to count the times data in one part of a data block are updated.

3. A storage sub-system, which includes a control memory for storing control data, a cache memory for temporarily storing data, a micro processor for controlling said control memory and said cache memory, a transmission destination for synchronously receiving said data from said micro processor and a transmission destination for asynchronously receiving said data from said microprocessor, comprising:

a function for issuing an inquiry for a data update process state of another storage sub-system that does not directly relate to data transmission.

4. A storage sub-system according to claim 3, wherein one transfer state/bit map set is held by said storage sub-system and another storage sub-system that does not directly perform data transmission, and wherein said transfer bit/bit map set is updated by said function for transmitting an inquiry for a data update process state.

5. A storage sub-system according to claim 4, wherein said transfer state/bit map set has an area for holding a counter value that represents the times data in one part of a data block is updated.

6. A large area data storage system comprising a first storage sub-system located in a first data center, a second storage sub-system located in a second data center and a third sub-system located in a third data center,

wherein data is synchronously transmitted between said first storage sub-system and said second storage sub-system;

wherein data is asynchronously transmitted between said first storage sub-system and said third storage sub-system, and said third storage sub-system consistently maintains the order of said data that has been transmitted.

7. A large area data storage system according to claim 6, wherein said first storage sub-system receives data from said second storage sub-system through synchronous transfer, and asynchronous transfer is used to transmit said data to said third storage sub-system.

8. A large area data storage system according to claim 6, wherein said first storage sub-system uses synchronous transfer to transmit data received from a host to said second storage sub-system and also uses asynchronous transfer to transmit said data to said third storage sub-system.

9. A large area data storage system according to claim 6, wherein said first data center and said second data center are located in the vicinity, and said first data center and said third data center are situated at remote locations.

10. A remote copy control method performed among three or more data centers, for a large data storage system which includes a first storage sub-system located in a first data center, a second storage sub-system located in a second data center and a third sub-system located in a third data center, said remote copy control method comprising:

a first step of said first storage sub-system receiving data from a host;

a second step of said first storage sub-system synchronously transmitting, to said second storage sub-system, said data received from said host;

a third step of said first storage sub-system asynchronously transmitting, to said third storage sub-system, said data received from said host; and

a fourth step of said second storage sub-system transmitting a inquiry to said third storage sub-system to determine whether said data from said host has arrived at said third storage sub-system.

11. A remote copy control method according to claim 10, further comprising:

a fifth step of halting the function of said first data center; and

a sixth step of said second storage sub-system transmitting, to said third storage sub-system, a part of the data stored in said second storage sub-system.

12. A remote copy control method, performed among three or more data centers, for a large data storage system that includes a first storage sub-system located in a first data center, a second storage sub-system located in a second data center and a third sub-system located in a third data center, said remote copy control method comprising:

a first step of said first storage sub-system receiving data from a host;

a second step of said first storage sub-system synchronously transmitting, to said second storage sub-system, said data received from said host;

a third step of said first storage sub-system asynchronously transmitting, to said third storage sub-system, said data received from said host through synchronous transfer; and

a fourth step of said first storage sub-system issuing an inquiry to said third storage sub-system to determine whether said data from said host has arrived at said third storage sub-system.

13. A remote copy control method according to claim 12, further comprising:

a fifth step of halting the function of said second data center; and

a sixth step of said first storage sub-system transmitting, to said third storage sub-system, one part of data stored in said first storage sub-system.

14. A remote copy control method, for a storage sub-system that includes means for writing data to a storage resource and that is connected to a plurality of transmission destinations to which data stored in said storage resource is to be transmitted, comprising the steps of:

said storage sub-system writing data to said storage resource;

said storage sub-system storing a correlation between write position information, which specifies a position in said storage resource whereat said data has been written, and a sequence number that is provided in the data writing order;

said storage sub-system transmitting, to one of said transmission destinations, said data that has been written and said correlation;

another storage sub-system for receiving said sequence number from said transmission destination; and

said storage sub-system employing a correlation stored therein and said received sequence number to obtain written data that has not yet been reflected to said transmission destination.

30. A storage sub-system, which includes means for writing data to a storage resource, which is connected to a plurality of transmission destinations to which data stored in said storage resource is to be transmitted and which is used for a remote copy control method according to claim 14, comprising:

means for writing data to said storage resource;

means for storing a correlation between write position information, which specifies a position in said storage resource whereat said data has been written, and a sequence number that is provided in the data writing order;

means for transmitting, to one of said transmission destinations, said data that has been written and said correlation;

means for receiving said sequence number from said transmission destination; and

means for employing a correlation stored therein and said received sequence number to obtain written data that has not yet been reflected to said transmission destination.

15. A remote copy control method, for a large data storage system including a first storage sub-system that has means for writing data to a first storage resource and that is located in a first data center, a host connected to said first storage sub-system, a second storage sub-system that includes means for writing data to a second storage resource and that is located in a second data center, and a third sub-system that includes means for writing data to a third storage resource and that is located in a third data center, said remote copy control method comprising the steps of:

said first storage sub-system writing data to said first storage resource in accordance with an instruction from said host;

said first storage sub-system storing a correlation between write position information, which specifies a position in said first storage resource whereat said data has been written, and a sequence number that is provided in the data writing order;

said first storage sub-system transmitting, to said second storage sub-system, said data that has been written and said correlation;

said second storage sub-system storing, in said second storage resource, said data and said correlation that have been received, and transmitting said data and said correlation to said third storage sub-system;

said third storage sub-system receiving said data and said correlation, storing said data in said third storage resource and transmitting, to said first storage sub-system, said sequence number in accordance with said correlation; and

said first storage sub-system employing said received sequence and a correlation stored therein to obtain data that has not yet been reflected to said third storage resource.

16. A remote copy control method according to claim 15, further comprising the steps of:

when said second storage sub-system breaks down due to a blockage, said first storage sub-system transmitting to said third storage sub-system differential data that has been obtained based on said received sequence number and said stored correlation and that has not yet been reflected to said third storage resource, and write position information for said differential data;

said third storage sub-system receiving said differential data and said write position information, and storing said differential data in said third storage resource based on said write position information in order to synchronize the contents of said first storage resource and the contents of said third storage resource;

when data is written to said first storage resource in accordance with an instruction from said host, said first storage sub-system transmitting, to said third storage sub-system, said written data and write position information that specifies a position in said first storage resource whereat said data has been written; and

said third storage sub-system receiving said data and said write position information, and storing said data at a location in said third storage resource that is specified by said write position information.

17. A remote copy control method according to claim 16, further comprising the steps of:

when said second storage sub-system has been recovered and is again available, said first storage sub-system transmitting all the data stored in said first storage resource to said second storage resource;

said first storage sub-system writing data to said first storage resource in accordance with an instruction from said host, and transmitting said data and said sequence number to said second and third storage sub-systems;

said second storage sub-system receiving said data and said sequence number, and storing said data in said second storage resource;

said second storage sub-system storing a correlation between write position information, which specifies a location in said second storage resource whereat said data has been written, and said sequence number that is provided in the data writing order;

said third storage sub-system receiving said data and said sequence number from said first storage sub-system, storing said data in said third storage resource, and transmitting to said second storage sub-system said sequence number according to said correlation;

halting data transmission by said first storage sub-system to said second storage sub-system;

said second storage sub-system receiving said sequence number from said third storage sub-system, and employing said sequence number, said sequence number stored therein and corresponding write position information to obtain data that has not yet been reflected to said third storage resource;

said second storage sub-system transmitting to said third storage sub-system the obtained differential data that has not been reflected to said third storage resource, and write position information; and

said third storage sub-system receiving said differential data and said write position information, and storing said differential data in said third storage resource based on said write position information so as to obtain synchronization between the contents of said first storage resource and the contents of said third storage resource.

22. A remote copy control method according to one of claim 15, wherein synchronous transfer is employed for the operation of said first storage sub-system and said second storage sub-system, and wherein asynchronous transfer is employed for the operation of said first storage sub-system and said third storage sub-system.

23. A remote copy control method according to one of claim 15, wherein said second storage sub-system or said third storage sub-system stores data at a storage location in said second storage resource or said third storage resource that corresponds to said write position information stored in said first storage sub-system.

31. A storage sub-system for a large data storage system including a first storage sub-system that includes means for writing data to a first storage resource and that is located in a first data center, a host connected to said first storage sub-system, a second storage sub-system that includes means for writing data to a second storage resource and that is located in a second data center, and a third sub-system that includes means for writing data to a third storage resource and that is located in a third data center, said storage sub-system that serves as said first storage sub-system used for a remote copy control method according to claim 15 comprising:

means for writing data to said first storage resource in accordance with an instruction from said host;

means for storing a correlation between write position information, which specifies a position in said first storage resource whereat said data has been written, and a sequence number that is provided in the data writing order;

means for transmitting, to said second storage sub-system, said data that has been written and said correlation;

means for storing, in said second storage resource, said data and said correlation that have been received, and transmitting said data and said correlation to said third storage sub-system;

means for receiving said data and said correlation, storing said data in said third storage resource and transmitting, to said first storage sub-system, said sequence number in accordance with said correlation; and

means for employing said received sequence and a correlation stored therein to obtain data that has not yet been reflected to said third storage resource.

18. A remote copy control method, for a storage sub-system which includes means for writing data to a storage resource and which is connected to a plurality of transmission destinations to which data stored in said storage resource is to be transmitted, comprising the steps of:

said storage sub-system writing data to said storage resource;

said storage sub-system generating a correlation between write position information, which specifies a position in said storage resource whereat said data has been written, and a sequence number that is provided in the data writing order;

said storage sub-system transmitting to a destination A, which is one of said transmission destinations, said data that has been written and said correlation;

said destination A storing said correlation received from said storage sub-system;

said destination A receiving a sequence number from a destination B that is another one of said transmission destinations;

another storage sub-system receiving said sequence number from said transmission destination; and

said destination A employing a correlation stored therein and said sequence number received from said destination B to obtain written data that has not yet been reflected to said transmission destination.

32. A storage sub-system for a storage sub-system A that includes means for writing data to a storage resource and that is connected to a plurality of transmission destinations to which data stored in said storage resource is to be transmitted, said storage sub-system that serves as one of said transmission destinations used for a remote copy control method according to claim 18 comprising:

means for storing data that is received from said storage sub-system A and is written to said storage resource, and a correlation between write position information, which specifies a position in said storage resource whereat said data has been written, and a sequence number that is provided in the data writing order;

means for receiving a sequence number from a destination B that is another one of said transmission destinations;

means for employing a correlation stored therein and said sequence number received from said destination B to obtain written data that has not yet been reflected to said transmission destination.

19. A remote copy control method, for a large data storage system including a first storage sub-system that includes means for writing data to a first storage resource and that is located in a first data center, a second storage sub-system that includes means for writing data to a second storage resource and that is located in a second data center, a host connected to said second storage sub-system, and a third sub-system that includes means for writing data to a third storage resource and that is located in a third data center, said remote copy control method comprising the steps of:

said second storage sub-system writing data to said second storage resource in accordance with an instruction from said host;

said second storage sub-system transmitting, to said first storage sub-system, said data that has been written, write position information that specifies a position whereat said data has been written, and a sequence number that is provided in the data writing order;

said first storage sub-system storing in said first storage resource said data and said sequence number that have been received, and storing said sequence number in correlation with said write position information that specifies a storage position in said first storage resource whereat said data has been written;

said second storage sub-system transmitting said data and said sequence number to said third storage sub-system;

said third storage sub-system receiving said data and said sequence number, storing said data in said third storage resource, and transmitting, to said first storage sub-system, said sequence number that is paired with said data; and

said first storage sub-system employing said received sequence and a correlation stored therein to obtain data that has not yet been reflected to said third storage resource.

20. A remote copy control method according to claim 19, further comprising the steps of:

when said second storage sub-system breaks down due to a blockage, said first storage sub-system transmitting, to said third storage sub-system, differential data that has been obtained based on said received sequence number and said stored correlation and that has not yet been reflected to said third storage resource, and write position information for said differential data;

said third storage sub-system receiving said differential data and said write position information, and storing said differential data in said third storage resource, based on said write position information, to synchronize the contents of said first storage resource and the contents of said third storage resource;

when data is written to said first storage resource in accordance with an instruction from said host, said first storage sub-system transmitting, to said third storage sub-system, said written data and write position information that specifies a location in said first storage resource whereat said data has been written; and

said third storage sub-system receiving said data and said write position information, and storing said data at a position in said third storage resource that is specified by said write position information.

21. A remote copy control method according to claim 20, further comprising the steps of:

when said second storage sub-system is recovered and is again available, said first storage sub-system transmitting to said second storage resource all the data that has been written by said host and is stored in said first storage resource;

said first storage sub-system writing data to said first storage resource in accordance with an instruction from said host;

said first storage sub-system transmitting said data and said sequence number to said second and third storage sub-systems;

said second storage sub-system receiving said data and said sequence number, and storing said data in said second storage resource at an appropriate time;

said second storage sub-system storing a correlation between write position information, which specifies a position in said second storage resource whereat said data has been written, and said sequence number that is provided in the data writing order;

said third storage sub-system receiving said data and said sequence number, storing said data in said third storage resource, and transmitting to said second storage sub-system said sequence number according to said correlation;

halting data transmission by said first storage sub-system to said second storage sub-system;

said second storage sub-system receiving said sequence number, and employing said sequence number and a correlation stored therein to obtain data that has not yet been reflected to said third storage resource;

said second storage sub-system transmitting to said third storage sub-system differential data that has been obtained based on said sequence number and said correlation and that has not yet been reflected to said third storage resource, and write position information for said differential data; and

said third storage sub-system receiving said differential data and said write position information, and storing said differential data in said third storage resource based on said write position information to synchronize the contents of said first storage resource and the contents of said third storage resource.

33. A storage sub-system for a large data storage system including a first storage sub-system that includes means for writing data to a first storage resource and that is located in a first data center, a second storage sub-system that includes means for writing data to a second storage resource and that is located in a second data center, a host connected to said second storage sub-system, and a third sub-system that includes means for writing data to a third storage resource and that is located in a third data center, said storage sub-system that serves as said first storage sub-system used for a remote copy control method according to claim 19 comprising:

means for receiving, from said second storage sub-system, data that is written in accordance with an instruction from said host, and a correlation between write position information, which specifies a position in said storage resource whereat said data has been written, and a sequence number that is provided in the data writing order, and for storing said data and said correlation in said first storage resource;

means for receiving a sequence number from said third storage sub-system, and for employing said sequence number and said correlation stored therein to obtain written data that has not yet been reflected to said third storage sub-system.

24. A remote copy control method, for a large area data storage system having a storage sub-system A connected to a host and a plurality of other storage sub-systems B connected to said storage sub-system through communication means, comprising the steps of:

said storage sub-systems B detecting a blockage that has occurred at one, at the least, of said host, said storage sub-system A and said communication means;

when said storage sub-systems B have detected said blockage, selecting a storage sub-system b from among said storage sub-systems B as a proxy for preforming the process for said storage sub-system A;

matching the data contents of said selected storage sub-system b and the data contents of other storage sub-systems B; and

said storage sub-systems transferring the operation of said host from a sub-host that is connected to said storage sub-system b.

25. A remote copy control method according to claim 24, wherein one or more of said storage sub-systems B perform said step of:

detecting the occurrence of said blockage, and said step of selecting said storage sub-system b as a proxy for performing the process of said storage sub-system A.

26. A remote copy control method according to claim 24, wherein said step of detecting the occurrence of said blockage is a step of:

identifying that a blockage has occurred when a heart beat message from said storage sub-system A is not received at a predetermined time by said storage sub-systems B.

27. A remote copy control method according to claim 24, wherein said step, upon detecting said blockage, of selecting said storage sub-system b from among said storage sub-systems B as a proxy for performing the process of said storage sub-system A is a step of:

comparing the latest sequence numbers among the sequence numbers that represent the order for the updating of data stored in said storage sub-systems B, and selecting a storage sub-system B that stores the latest sequence number as said storage sub-system b that serves as a proxy for performing the process of said storage sub-system A.

28. A remote copy control method according to claim 24, wherein, when of said sequence numbers indicating the order for the updating of data stored in each of said storage sub-systems B one sequence number is omitted, the latest of the sequential sequence numbers is employed for the comparison.

29. A remote copy control method according to claim 24, wherein said step of matching the data contents of said selected storage sub-system b and the data contents of the other storage sub-systems B is a step of:

copying all the data between said storage sub-system b and the other storage sub-systems B, or copying differential data stored in said storage sub-systems, so that the contents of said storage sub-system b match the contents of said other storage sub-systems B.

34. A storage sub-system for a large area data storage system having a storage sub-system A connected to a host and a plurality of other storage sub-systems B connected to said storage sub-system through communication means, said storage sub-system that serves as said storage sub-system B used for a remote copy control method according to claim 24 comprising:

means for detecting a blockage that has occurred at one, at the least, of said host, said storage sub-system A and said communication means;

means for, when said storage sub-systems B have detected said blockage, selecting a storage sub-system b from among said storage sub-systems B as a proxy for preforming the process for said storage sub-system A;

means for matching the data contents of said selected storage sub-system b and the data contents of other storage sub-systems B; and

means for transferring the operation of said host from a sub-host that is connected to said storage sub-system b.