Remote copy system

• US 7,293,050 B2
• Filed: 12/03/2004
• Issued: 11/06/2007
• Est. Priority Date: 12/03/2003
• Status: Expired due to Fees

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Ninose, Kenta, Arakawa, Hiroshi, Hirakawa, Yusuke, Asaka, Yoshihiro
• Primary Examiner(s): Wu; Yicun
• Application Number:
  US11/002,105
• Publication Number:
  US 20050125618A1
• Time in Patent Office: 1,068 Days
• Field of Search: 707 1- 10, 707200-204, 711112-114, 711147-154, 714/6, 714/13
• US Class Current: 1/1
• CPC Class Codes: G06F 11/2058 using more than 2 mirrored copies
• CPC Class Codes: G06F 11/2064 while ensuring consistency
• CPC Class Codes: G06F 11/2071 using a plurality of controllers
• CPC Class Codes: G06F 11/2074 Asynchronous techniques
• CPC Class Codes: G06F 11/2082 Data synchronisation
• CPC Class Codes: G06F 2201/82 Solving problems relating to consistency
• CPC Class Codes: G06F 2201/835 Timestamp
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users
• CPC Class Codes: Y10S 707/99953 Recoverability
• CPC Class Codes: Y10S 707/99955 Archiving or backup

Abstract

In a system in which data employed by a computer is stored in a storage system, the storage system transfers this data to another storage system and a copy of the data is maintained in the other storage system, consistency of the copy is maintained even in the case of data written to the storage system by a computer, to which a write time is not applied.

A source storage system, when a write time is applied to a write request, records the write time and applies this write time to the received write data and, when no write time is applied, applies the recorded write time to the received write data and transfers the write data with this write time applied thereto, to another storage system. The target storage system stores the write data in a logical volume in the target storage system in accordance with the write time.

Claims

1. A remote copy system comprising:

a management software stored on a computer readable storage medium and implemented in one or more computers;

a plurality of first storage systems, each comprising a first control section to be coupled to the computers, at least one first logical volume for storing data received from the computers, and a first storage area for storing data to be transmitted to a second storage system; and

a plurality of second storage systems, each coupled to a first storage system and comprising a second control section, at least one second logical volume for storing a replica of data stored in the first logical volume of at least one of the plurality of first storage systems, and a second storage area for storing data received from the at least one of the plurality of first storage systems,

wherein first control sections of the plurality of first storage systems receive a plurality of first write requests from the computers and store a plurality of first write data, which relate to the plurality of first write requests and are stored in first logical volumes of the plurality of first storage systems, in first storage areas of the plurality of first storage systems,

wherein the first control sections of the plurality of first storage systems defer operations for executing a plurality of second write requests received from the computers, and store a plurality of identification information, each corresponding to a marker and received from the management software, and transmit completion reports to the management software after storing the plurality of identification information,

wherein the management software sends instructions, for restarting the operations for executing the plurality of second write requests, to the plurality of first storage systems after the management software receives the completion reports,

wherein the first control sections of the plurality of first storage systems transfer the plurality of first write data and the plurality of identification information to the plurality of second storage systems,

wherein second control sections of the plurality of second storage systems receive the plurality of first write data and the plurality of identification information from the plurality of first storage systems, and

wherein the second control sections of the plurality of second storage systems control operations according to a plurality of instructions sent from at least one of the management software and another management software, stored on other computer readable storage medium and implemented in one or more other computers coupled to the plurality of second storage systems, such that the plurality of first write data received from the plurality of first storage systems are stored in second logical volumes of the plurality of second storage systems when the plurality of identification information are designated by the plurality of instructions sent from the at least one of the management software and the another management software.

2. A remote copy system according to claim 1, wherein first control sections of the plurality of first storage systems receive a plurality of new first write requests from the computers and store a plurality of new first write data, which relates to the plurality of new first write requests and are stored in first logical volumes of the plurality of first storage systems, in first storage areas of the plurality of first storage systems,

wherein the first control sections of the plurality of first storage systems defer operations for executing a plurality of new second write requests received from the computers, and store a plurality of new identification information, each corresponding to a new marker and received from the management software, and transmit new completion reports to the management software after storing the plurality of new identification information,

wherein the management software sends instructions, for restarting the operations for executing the plurality of new second write requests, to the plurality of first storage systems after the management software receives the new completion reports,

wherein the first control sections of the plurality of first storage systems transfer the plurality of new first write data and the plurality of new identification information to the plurality of second storage systems, and

wherein second control sections of the plurality of second storage systems receive the plurality of new first write data and the plurality of new identification information from the plurality of first storage systems.

3. A remote copy system according to claim 1, wherein first control sections of the plurality of first storage systems repeatedly receive a plurality of new first write requests from the computers and repeatedly store a plurality of new first write data, which relates to the plurality of new first write requests and are stored in first logical volumes of the plurality of first storage systems, in first storage areas of the plurality of first storage systems,

wherein the first control sections of the plurality of first storage systems repeatedly defer operations for executing a plurality of new second write requests newly received from the computers, and repeatedly store a plurality of new identification information, each corresponding to a new marker and newly received from the management software, and repeatedly transmit new completion reports to the management software after storing the plurality of new identification information,

wherein the management software repeatedly sends instructions, for restarting the operations for executing the plurality of new second write requests, to the plurality of first storage systems after the management software receives the new completion reports,

wherein the first control sections of the plurality of first storage systems repeatedly transfer the plurality of new first write data and the plurality of new identification information to the plurality of second storage systems, and

wherein second control sections of the plurality of second storage systems repeatedly receive the plurality of new first write data and the plurality of new identification information from the plurality of first storage systems.

4. A remote copy system according to claim 1,

(A) wherein first control sections of the plurality of first storage systems receive a plurality of new first write requests from the computers and store a plurality of new first write data, which relates to the plurality of new first write requests and are stored in first logical volumes of the plurality of first storage systems, in first storage areas of the plurality of first storage systems;

(B) wherein the first control sections of the plurality of first storage systems defer operations for executing a plurality of new second write requests received from the computers, and store a plurality of new identification information, each corresponding to a new marker and received from the management software, and transmit new completion reports to the management software after storing the plurality of new identification information;

(C) wherein the management software sends instructions, for restarting the operations for executing the plurality of new second write requests, to the plurality of first storage systems after the management software receives the new completion reports;

(D) wherein the first control sections of the plurality of first storage systems transfer the plurality of new first write data and the plurality of new identification information to the plurality of second storage systems;

(E) wherein second control sections of the plurality of second storage systems receive the plurality of new first write data and the plurality of new identification information from the plurality of first storage systems; and

wherein the wherein clauses (A) to (E) are serially performed repeatedly.

5. A remote copy system according to claim 1, wherein each of the plurality of identification information is a marker number.

6. A remote copy system according to claim 2, wherein a value of each of the plurality of identification information is different from a value of each of the plurality of new identification information.

7. A remote copy system according to claim 1, wherein the plurality of identification information include same information.

8. A remote copy system according to claim 6, wherein the plurality of identification information include same information.

9. A remote copy system according to claim 1, wherein the first control section of each of the plurality of first storage systems receives a plurality of third write requests from the computers and stores a plurality of third write data, which relates to the plurality of third write requests and are stored in the first logical volume of each of the plurality of first storage systems, in the first storage area of each of the plurality of first storage systems,

wherein the first control section of each of the plurality of first storage systems defers operations for executing a plurality of fourth write requests received from the computers, and stores an identification information, corresponding to a marker and received from the management software, and transmits a completion report to the management software after storing the identification information,

wherein the management software sends instructions, for restarting the operations for executing the plurality of fourth write requests, to each of the plurality of first storage systems after the management software receives the completion report from the first control section of each of the plurality of first storage systems,

wherein the first control section of each of the plurality of first storage systems transfers the plurality of third write data and the identification information to at least one of the plurality of second storage systems,

wherein the second control section of each of the plurality of second storage systems receives the plurality of third write data and the identification information from at least one of the plurality of first storage systems, and

wherein the second control section of each of the plurality of second storage systems controls operation according to an instruction sent from the at least one of the management software and the another management software such that the plurality of third write data received from the at least one of the plurality of first storage systems are stored in the second logical volumes of the at least one of the plurality of second storage systems when the identification information is designated by the instruction sent from the at least one of the management software and the another management software.

10. A remote copy system according to claim 1,

(A) wherein the first control section of each of the plurality of first storage systems receives a plurality of third write requests from the computers and stores a plurality of third write data, which relates to the plurality of third write requests and are stored in the first logical volume of each of the plurality of first storage systems, in the first storage area of each of the plurality of first storage systems;

(B) wherein the first control section of each of the plurality of first storage systems defers operations for executing a plurality of fourth write requests received from the computers, and stores an identification information, corresponding to a marker and received from the management software, and transmits a completion report to the management software after storing the identification information;

(C) wherein the management software sends instructions, for restarting the operations for executing the plurality of fourth write requests, to each of the plurality of first storage systems after the management software receives the completion report from the first control section of each of the plurality of first storage systems;

(D) wherein the first control section of each of the plurality of first storage systems transfers the plurality of third write data and the identification information to at least one of the plurality of second storage systems;

(E) wherein the second control section of each of the plurality of second storage systems receives the plurality of third write data and the identification information from at least one of the plurality of first storage systems; and

wherein the wherein clauses (A) to (E) are serially performed repeatedly according to an asynchronous remote copy process.

11. A remote copy system according to claim 1, wherein the first control sections of the plurality of first storage systems store the plurality of identification information in memories of the plurality of first storage systems during deferring the operations for executing the plurality of second write requests.

12. A remote copy system according to claim 10, wherein the first control section of each of the plurality of first storage systems stores the identification information in a memory during deferring the operations for executing the plurality of fourth write requests.

13. A remote copy system according to claim 1, wherein the plurality of identification information are used for establishing consistency between a plurality of replica data stored in the second logical volumes of the plurality of second storage systems.

14. A remote copy system according to claim 1, wherein the second control sections of the plurality of second storage systems control to store the plurality of first write data, received from the plurality of first storage systems, in second storage areas of the plurality of second storage systems, and

wherein the second control sections of the plurality of second storage systems control to store the plurality of first write data, stored in the second storage areas of the plurality of second storage systems, in the second logical volumes of the plurality of second storage systems when the plurality of identification information are designated by the plurality of instructions sent from the at least one of the management software and the another management software.

15. A remote copy system according to claim 1, wherein the plurality of first storage systems and the plurality of second storage systems configure a plurality of pairs, each including the first logical volumes of one of the plurality of first storage systems and the second logical volumes of one of the plurality of second storage systems.

16. A remote copy system according to claim 1, wherein the management software and/or the another management software reads the plurality of identification information from the plurality of second storage systems and determines to designate the plurality of identification information.

17. A remote copy system comprising:

a plurality of first storage systems, each comprising a first control section to be coupled to one or more computers, at least one first logical volume, and a first storage area for storing data to be transmitted to a second storage system; and

a plurality of second storage systems, each coupled to a first storage system and comprising a second control section, at least one second logical volume for storing replicated data corresponding to data to be stored in the first logical volume of at least one of the plurality of first storage systems, and a second storage area for storing data received from the at least one of the plurality of first storage systems,

wherein each of first control sections of the plurality of first storage systems receives a plurality of first write requests from the computers and stores a plurality of first write data, which correspond to a plurality of write data to be stored in the first logical volume according to the plurality of first write requests, in the first storage area,

wherein each of the first control sections records an identification information received from a management software in the computers and transmits a completion report to the management software after recording the identification information,

wherein each of the plurality of first storage systems receives an instruction, for cancellation of a deferment process, from the management software after each of the plurality of first storage systems sends the completion report to the management software, the deferment process being a process in which write processes of a plurality of second write requests received from the computers are deferred,

wherein each of the first control sections transfers the plurality of first write data and the identification information to at least one of the plurality of second storage systems according to an asynchronous remote copy process,

wherein each of second control sections of the plurality of second storage systems receives the plurality of first write data and the identification information from at least one of the plurality of first storage systems, and

wherein each of the second control sections controls operation according to another instruction sent from at least one of the management software and another management software in one or more other computers coupled to the plurality of second storage systems such that the plurality of first write data received from at least one of the plurality of first storage systems are stored in the second logical volume of at least one of the plurality of second storage systems if information corresponding to the identification information is designated by the another instruction sent from the at least one of the management software and the another management software.

18. A remote copy system according to claim 17, wherein each of the first control sections receives a plurality of new first write requests from the computers and stores a plurality of new first write data, which correspond to a plurality of new write data to be stored in the first logical volume according to the plurality of new first write requests, in the first storage area,

wherein each of the first control sections records a new identification information received from the management software and transmits new completion report to the management software after recording the new identification information,

wherein each of the plurality of first storage systems receives new instruction, for cancellation of a new deferment process, from the management software after each of the plurality of first storage systems sends the new completion report to the management software, the new deferment process being a process in which write processes of a plurality of new second write requests received from the computers are deferred,

wherein each of the first control sections transfers the plurality of new first write data and the new identification information to at least one of the plurality of second storage systems according to the asynchronous remote copy process, and

wherein each of the second control sections receives the plurality of new first write data and the new identification information from at least one of the plurality of first storage systems.

19. A remote copy system according to claim 17, wherein each of the first control sections repeatedly receives a plurality of new first write requests from the computers and repeatedly stores a plurality of new first write data, which correspond to a plurality of new write data to be stored in the first logical volume according to the plurality of new first write requests, in the first storage area,

wherein each of the first control sections repeatedly records a new identification information received from the management software and repeatedly transmits new completion report to the management software after recording the new identification information,

wherein each of the plurality of first storage systems repeatedly receives new instruction, for cancellation of a new deferment process, from the management software after each of the plurality of first storage systems sends the new completion report to the management software, the new deferment process being a process in which write processes of a plurality of new second write requests received from the computers are deferred,

wherein each of the first control sections repeatedly transfers the plurality of new first write data and the new identification information to at least one of the plurality of second storage systems according to the asynchronous remote copy process, and

wherein each of the second control sections repeatedly receives the plurality of new first write data and the new identification information from at least one of the plurality of first storage systems.

20. A remote copy system according to claim 17,

(A) wherein each of the first control sections receives a plurality of new first write requests from the computers and stores a plurality of new first write data, which correspond to a plurality of new write data to be stored in the first logical volume according to the plurality of new first write requests, in the first storage area;

(B) wherein each of the first control sections records a new identification information received from the management software and transmits new completion report to the management software after recording the new identification information;

(C) wherein each of the plurality of first storage systems receives new instruction, for cancellation of a new deferment process, from the management software after each of the plurality of first storage systems sends the new completion report to the management software, the new deferment process being a process in which write processes a plurality of new second write requests received from the computers are deferred;

(D) wherein each of the first control sections transfers the plurality of new first write data and the new identification information to at least one of the plurality of second storage systems according to the asynchronous remote copy process;

(E) wherein each of the second control sections receives the plurality of new first write data and the new identification information from at least one of the plurality of first storage systems; and

wherein the wherein clauses (A) to (E) are serially performed repeatedly.

21. A remote copy system according to claim 17, wherein the identification information is a marker number.

22. A remote copy system according to claim 18, wherein a value of the identification information is different from a value of the new identification information.

23. A remote copy system according to claim 17, wherein the identification information recorded by each of the first control sections includes same information.

24. A remote copy system according to claim 22, wherein the identification information recorded by each of the first control sections includes same information.

25. A remote copy system according to claim 17, wherein each of the first control sections stores the identification information in a memory of each of the plurality of first storage systems during the deferment process.

26. A remote copy system according to claim 24, wherein each of the first control sections stores the identification information in a memory of each of the plurality of first storage systems during the deferment process.

27. A remote copy system according to claim 17, wherein the identification information is used for establishing consistency between a plurality of replica data stored in the second logical volumes of the plurality of second storage systems.

28. A remote copy system according to claim 17, wherein each of the second control sections controls to store the plurality of first write data, received from the at least one of the plurality of first storage systems, in the second storage area, and

wherein each of the second control sections controls to store the plurality of first write data, stored in the second storage area, in the second logical volume if information corresponding to the identification information is designated by the another instruction sent from the at least one of the management software and the another management software.

29. A remote copy system according to claim 17, wherein the plurality of first storage systems and the plurality of second storage systems configure a plurality of pairs, each including the first logical volume of one of the plurality of first storage systems and the second logical volume of one of the plurality of second storage systems.

30. A remote copy system according to claim 17, wherein the at least one of the management software and the another management software reads the identification information from each of the plurality of second storage systems and determines to designate the identification information.

31. A remote copy system according to claim 17, wherein the identification information relates to a marker.

32. A remote copy system comprising:

a plurality of first storage systems, each comprising a first control section to be coupled to one or more computers, at least one first logical volume, and a first storage area for storing data to be transmitted to a second storage system; and

a plurality of second storage systems, each coupled to a first storage system and comprising a second control section, at least one second logical volume, and a second storage area for storing replicated data, corresponding to data to be stored in at least one of first logical volumes of the plurality of first storage systems, and received from at least one of the plurality of first storage systems,

wherein first control sections of the plurality of first storage systems receive a plurality of first write requests from the computers and stores a plurality of first write data, which correspond to a plurality of write data to be stored in first logical volumes of the first storage systems according to the plurality of first write requests, in first storage areas of the plurality of first storage systems,

wherein the first control sections record a plurality of identification information received from a management software in the computers and transmit a plurality of acknowledges to the management software after recording the plurality of identification information,

wherein the plurality of first storage systems receive a plurality of instructions, for cancellation of a status in which a plurality of second write requests subsequent to the plurality of first write requests are not processed, from the management software after the plurality of first storage systems send the plurality of acknowledges to the management software,

wherein the first control sections transfer the plurality of first write data and the plurality of identification information to the plurality of second storage systems,

wherein second control sections of the plurality of second storage systems receive the plurality of first write data and the plurality of identification information from the plurality of first storage systems,

wherein the second control sections control operation according to a plurality of another instructions sent from at least one of the management software and another management software in one or more other computers coupled to the plurality of second storage systems such that the plurality of first write data received from the plurality of first storage systems are stored in second logical volumes of the plurality of second storage systems if the plurality of identification information are designated by the plurality of another instructions sent from the at least one of the management software and the another management software.

33. A remote copy system according to claim 32

(A) wherein the first control section of one of the plurality of first storage systems receives a plurality of third write requests, which include at least one of the first write requests and are sent from the computers, and stores a plurality of third write data, which correspond to a plurality of write data to be stored in the first logical volumes according to the plurality of third write requests, in the first storage areas;

(B) wherein the first control section of the one of the plurality of first storage systems records one of the plurality of identification information received from the management software and transmits one of the plurality of acknowledges to the management software after recording the one of the plurality of identification information;

(C) wherein the one of the plurality of first storage systems receives one of the plurality of instructions from the management software after the plurality of first storage systems send the plurality of acknowledges to the management software;

(D) wherein the first control section of the one of the plurality of first storage systems transfers the plurality of third write data and the one of the plurality of identification information to at least one of the plurality of second storage systems;

(E) wherein the second control section of one of the plurality of second storage systems receives the plurality of third write data and the one of the plurality of identification information from at least one of the plurality of first storage systems; and

wherein the wherein clauses (A) to (E) are serially performed repeatedly.

34. A remote copy system according to claim 33, wherein the first control section of one of the plurality of first storage systems receives a plurality of new third write requests sent from the computers and stores a plurality of new third write data, which correspond to a plurality of new write data to be stored in the first logical volumes according to the plurality of new third write requests, in the first storage areas,

wherein the first control section of the one of the plurality of first storage systems records one of a plurality of new identification information received from the management software and transmits one of the plurality of new acknowledges to the management software after recording the one of the plurality of new identification information,

wherein the one of the plurality of first storage systems receives one of a plurality of new instructions, for cancellation of a status in which a plurality of new second write requests subsequent to the plurality of new third write requests are not processed, from the management software after the plurality of first storage systems send the plurality of new acknowledges to the management software,

wherein the first control section of the one of the plurality of first storage systems transfers the plurality of new third write data and the one of the plurality of new identification information to at least one of the plurality of second storage systems, and

wherein the second control section of one of the plurality of second storage systems receives the plurality of new third write data and the one of the plurality of new identification information from at least one of the plurality of first storage systems.

35. A remote copy system according to claim 32:

(A) wherein the first control sections receive a plurality of new first write requests from the computers and store a plurality of new first write data, which correspond to a plurality of new write data to be stored in the first logical volume according to the plurality of new first write requests, in the first storage areas;

(B) wherein the first control sections record a plurality of new identification information received from the management software and transmit a plurality of new acknowledges to the management software after recording the plurality of new identification information;

(C) wherein the plurality of first storage systems receive a plurality of new instruction, for cancellation of a status in which a plurality of new second write requests subsequent to the plurality of new first write requests are not processed, from the management software after the plurality of first storage systems send the plurality of new acknowledges to the management software;

(D) wherein the first control sections transfer the plurality of new first write data and the plurality of new identification information to the plurality of second storage systems;

(E) wherein the second control sections receive the plurality of new first write data and the plurality of new identification information from the plurality of first storage systems; and

wherein the wherein clauses (A) to (E) are serially performed repeatedly.

36. A remote copy system according to claim 32, wherein the plurality of identification information are a plurality of tag numbers.

37. A remote copy system according to claim 35, wherein a value of one of the plurality of identification information is different from a value of one of the plurality of new identification information.

38. A remote copy system according to claim 32, wherein each of the plurality of identification information recorded by the first control sections include same information.

39. A remote copy system according to claim 37, wherein each of the plurality of identification information recorded by the first control sections include same information.

40. A remote copy system according to claim 32, wherein the first control sections store the plurality of identification information in a plurality of memories of the plurality of first storage systems during the status in which the plurality of second write requests are not processed.

41. A remote copy system according to claim 39, wherein the first control sections store the plurality of identification information in a plurality of memories of the plurality of first storage systems during the status in which the plurality of second write requests are not processed.

42. A remote copy system according to claim 32, wherein the plurality of identification information are used for establishing consistency between

a plurality of replicated data stored in the second logical volumes of the plurality of second storage systems.

43. A remote copy system according to claim 32, wherein the second control sections control to store the plurality of first write data, received from the at least one of the plurality of first storage systems, in second storage areas of the plurality of second storage systems, and

wherein the second control sections control to store the plurality of first write data, stored in the second storage areas, in the second logical volumes if the plurality of identification information are designated by the plurality of another instructions sent from the at least one of the management software and the another management software.

44. A remote copy system according to claim 32, wherein the plurality of first storage systems and the plurality of second storage systems configure a plurality of pairs, each including the first logical volume of one of the plurality of first storage systems and the second logical volume of one of the plurality of second storage systems.

45. A remote copy system according to claim 32, wherein the at least one of the management software and the another management software reads the plurality of identification information from the plurality of second storage systems and determines to designate the plurality of identification information.

46. A remote copy system according to claim 32, wherein each of the plurality of identification information relates to a tag.

47. A remote copy system according to claim 32, wherein the status, in which the plurality of second write requests are not processed, is a status in which write processes of the plurality of second write requests are prevented.

48. A remote copy system according to claim 32, wherein the status, in which the plurality of second write requests are not processed, is a status in which write processes of the plurality of second write requests are not processed;

and

wherein the first control sections start the status and transmit the plurality of acknowledges to the management software after starting the status.

49. A remote copy system according to claim 32, wherein the status, in which the plurality of second write requests are not processed, is a status in which write processes of the plurality of second write requests are deferred.

50. At least one storage system adaptive to be included in a plurality of first storage systems, each of the plurality of first storage systems comprising:

a first control section to be coupled to one or more computers,

one or more disk drives related to at least one first logical volume, and

a first storage area for storing data to be transmitted to a second storage system included in a plurality of second storage systems, each of the second storage systems comprising a second control section, one or more second disk drives related to at least one second logical volume, and a second storage area for storing data received from at least one of the plurality of first storage systems,

wherein first control sections of the plurality of first storage systems receive a plurality of first write requests from the computers and store a plurality of first write data, which correspond to a plurality of write data to be stored in first logical volumes of the plurality of first storage systems according to the plurality of first write requests, in first storage areas of the first storage systems,

wherein the first control sections of the plurality of first storage systems record a plurality of identification information received from a management software in the one or more computers and transmit a plurality of reports to the management software after recording the plurality of identification information,

wherein the first control sections of the plurality of first storage systems receive a plurality of instructions, for cancellation of a status in which a plurality of write processes of a plurality of second write requests subsequent to the plurality of first write requests are not processed, from the management software after the first control sections of the plurality of first storage systems send the plurality of reports to the management software, and

wherein the first control sections of the plurality of first storage systems transfer the plurality of first write data and the plurality of identification information to the plurality of second storage systems, so that second control sections of the plurality of second storage systems receive the plurality of first write data and the plurality of identification information and control operation according to a plurality of another instructions sent from at least one of the management software and another management software in one or more other computers coupled to the plurality of second storage systems such that the plurality of first write data are stored in second logical volumes of the plurality of second storage systems if the plurality of identification information are designated by the plurality of another instructions.

51. At least one storage system according to claim 50:

(A) wherein the first control section of one of the plurality of first storage systems receives a plurality of third write requests, which include at least one of the plurality of first write requests and are sent from the computers, and stores a plurality of third write data, which correspond to a plurality of write data to be stored in the first logical volume of one of the plurality of first storage systems according to the plurality of third write requests, in the first storage area of one of the first storage systems;

(B) wherein the first control section of one of the plurality of first storage systems records one of the plurality of identification information received from the management software and transmits one of the plurality of reports to the management software after recording the one of the plurality of identification information;

(C) wherein the first control section of one of the plurality of first storage systems receives one of the plurality of instructions from the management software after first control sections of the plurality of first storage systems send the plurality of reports to the management software; and

(D) wherein the first control section of one of the plurality of first storage systems transfers the plurality of third write data and the one of the plurality of identification information to at least one of the plurality of second storage systems according to an asynchronous remote copy process; and

wherein the wherein clauses (A) to (D) are serially performed repeatedly.

52. At least one storage system according to claim 51, wherein the first control section of one of the plurality of first storage systems receives a plurality of new third write requests from the computers and stores a plurality of new third write data, which correspond to a plurality of write data to be stored in the first logical volume of one of the plurality of first storage systems according to the plurality of third new write requests, in the first storage area of one of the first storage systems,

wherein the first control section of one of the plurality of first storage systems records one of a plurality of new identification information received from the management software and transmits one of the plurality of new reports to the management software after recording the one of the plurality of new identification information,

wherein the first control section of one of the plurality of first storage systems receives one of a plurality of new instructions from the management software after first control sections of the plurality of first storage systems send the plurality of new reports to the management software, and

wherein the first control section of one of the plurality of first storage systems transfers the plurality of new third write data and the one of the plurality of new identification information to at least one of the plurality of second storage systems according to an asynchronous remote copy process.

53. At least one storage system according to claim 50:

(A) wherein the first control sections of the plurality of first storage systems receive a plurality of new first write requests from the one or more computers and store a plurality of new first write data, which correspond to a plurality of new write data to be stored in the first logical volumes of the plurality of first storage systems according to the plurality of new first write requests, in the first storage areas of the first storage systems;

(B) wherein the first control sections of the plurality of first storage systems record a plurality of new identification information received from the management software and transmit a plurality of new reports to the management software after recording the plurality of new identification information;

(C) wherein the first control sections of the plurality of first storage systems receive a plurality of new instructions, for cancellation of a status in which a plurality of write processes of a plurality of new second write requests subsequent to the plurality of new first write requests are not processed, from the management software after the first control sections of the plurality of first storage systems send the plurality of new reports to the management software;

(D) wherein the first control sections of the plurality of first storage systems transfer the plurality of new first write data and the plurality of new identification information to the plurality of second storage systems; and

wherein the wherein clauses (A) to (D) are serially performed repeatedly.

54. At least one storage system according to claim 50, wherein each of the plurality of identification information is a marker number.

55. At least one storage system according to claim 53, wherein a value of one of the plurality of identification information is different from a value of one of the plurality of new identification information.

56. At least one storage system according to claim 50, wherein each of the plurality of identification information recorded by the first control sections of the plurality of first storage systems includes same information.

57. At least one storage system according to claim 55, wherein each of the plurality of identification information recorded by the first control sections of the plurality of first storage systems includes same information.

58. At least one storage system according to claim 50, wherein the first control sections of the plurality of first storage systems store the plurality of identification information in a plurality of memories of the plurality of first storage systems during the status in which the plurality of write processes of the plurality of second write requests are not processed.

59. At least one storage system according to claim 57, wherein the first control sections of the plurality of first storage systems store the plurality of identification information in a plurality of memories of the plurality of first storage systems during the status in which the plurality of write processes of the plurality of second write requests are not processed.

60. At least one storage system according to claim 50, wherein the plurality of identification information is used for establishing consistency between a plurality of replicated data stored in the second logical volumes of the plurality of second storage systems.

61. At least one storage system according to claim 51, wherein the second control sections of the plurality of second storage systems control to store the plurality of first write data, received from the plurality of first storage systems, in second storage areas of the second storage systems, and

wherein the second control sections of the plurality of second storage systems control to store the plurality of first write data, stored in the second storage areas, in the second logical volumes of the second storage if the plurality of identification information are designated by the plurality of another instructions.

62. At least one storage system according to claim 50, wherein the plurality of first storage systems and the plurality of second storage systems configure a plurality of pairs, each including the first logical volume of one of the plurality of first storage systems and the second logical volume of one of the plurality of second storage systems.

63. At least one storage system according to claim 50, wherein the at least one of the management software and the another management software reads the plurality of identification information from the plurality of second storage systems and determines to designate the plurality of identification information.

64. At least one storage system according to claim 50, wherein the plurality of identification information relate to a plurality of markers.

65. At least one storage system according to claim 50, wherein the status, in which the plurality of write processes of the plurality of second write requests are not processed, is a status in which the plurality of write processes of the plurality of second write requests are prevented.

66. At least one storage system according to claim 50, wherein the status, in which the plurality of write processes of the plurality of second write requests are not processed, is a status in which the plurality of write processes of the plurality of second write requests are not processed;

and

wherein the first control sections of the plurality of first storage systems start the status and transmit the plurality of reports to the management software after starting the status.

67. At least one storage system according to claim 50, wherein the status, in which the plurality of write processes of the plurality of second write requests are not processed, is a status in which the plurality of write processes of the plurality of second write requests are deferred.

68. A computer program stored on a computer readable storage medium and implemented in at least one first computer, the first computer adaptive to be coupled to a plurality of first storage systems each comprising a first control section and one or more disk drives related to at least one first logical volume, the plurality of first storage systems coupled to a plurality of second storage systems, the plurality of second storage systems each comprising a second control section and one or more second disk drives related to at least one second logical volume storing replicated data of data stored in the first logical volume of at least one of the plurality of first storage systems, the computer program comprising:

code controlling to send a plurality of identification information to the plurality of first storage systems;

code controlling to receive a plurality of reports from the plurality of first storage systems after sending the plurality of identification information; and

code controlling to send a plurality of instructions to the plurality of first storage systems after receiving the plurality of reports from the plurality of first storage systems, the plurality of instructions instructing to cancel a plurality of deferment processes in which write processes of a plurality of first write requests are deferred,

wherein the plurality of identification information are related to a plurality of write data by the plurality of first storage systems, the plurality of write data correspond to a plurality of second write requests received from at least one of the first computer and one or more second computers coupled to the plurality of first storage systems before receiving the plurality of first write request and are stored in first logical volumes of the plurality of first storage systems,

wherein the plurality of identification information and the plurality of write data are transferred from the plurality of first storage systems to the plurality of second storage systems, and

wherein the plurality of identification information are used for storing the plurality of write data in second logical volumes of the plurality of second storage systems, if the plurality of identification information are designated by a plurality of another instructions sent from at least one the computer program and another computer program stored on a computer readable storage medium and implemented in at least one third computer, the third computer adaptive to be coupled to the plurality of second storage systems.

69. A computer program according to claim 68, wherein the code, controlling to send the plurality of identification information to the plurality of first storage systems, controls to send a plurality of new identification information to the plurality of first storage systems,

wherein the code, controlling to receive the plurality of reports from the plurality of first storage systems after sending the plurality of identification information, controls to receive a plurality of new reports from the plurality of first storage systems after sending the plurality of new identification information, and

wherein the code, controlling to send the plurality of instructions to the plurality of first storage systems after receiving the plurality of reports from the plurality of first storage systems, controls to send a plurality of new instructions to the plurality of first storage systems after receiving the plurality of new reports from the plurality of first storage systems, the plurality of new instructions instructing to cancel a plurality of deferment processes in which write processes of a plurality of new first write requests are deferred.

70. A computer program according to claim 68, wherein the code, controlling to send the plurality of identification information to the plurality of first storage systems, (A) controls to send a plurality of new identification information to the plurality of first storage systems,

wherein the code, controlling to receive the plurality of reports from the plurality of first storage systems after sending the plurality of identification information, (B) controls to receive a plurality of new reports from the plurality of first storage systems after sending the plurality of new identification information,

wherein the code, controlling to send the plurality of instructions to the plurality of first storage systems after receiving the plurality of reports from the plurality of first storage systems, (C) controls to send a plurality of new instructions to the plurality of first storage systems after receiving the plurality of new reports from the plurality of first storage systems, the plurality of new instructions instructing to cancel a plurality of deferment processes in which write processes of a plurality of new first write requests are deferred, and

wherein functions (A) to (C) of the wherein clauses are serially performed repeatedly.

71. A computer program according to claim 68, wherein the code, controlling to send the plurality of identification information to the plurality of first storage systems, controls to send one of the plurality of identification information to one of the plurality of first storage systems,

wherein the code, controlling to receive the plurality of reports from the plurality of first storage systems after sending the plurality of identification information, controls to receive one of the plurality of reports from the one of the plurality of first storage systems after sending the one of the plurality of identification information,

wherein the code, controlling to send the plurality of instructions to the plurality of first storage systems after receiving the plurality of reports from the plurality of first storage systems, controls to send one of the plurality of instructions to the one of the plurality of first storage systems after receiving the plurality of reports from the plurality of first storage systems,

wherein the one of the plurality of identification information is related to a set of write data by the one of the plurality of first storage systems, the set of write data includes one of the plurality of write data correspond to one of the plurality of second write requests,

wherein the one of the plurality of identification information and the set of write data are transferred from the one of the plurality of first storage systems to one of the plurality of second storage systems, and

wherein the one of the plurality of identification information is used for storing the set of write data in the second logical volume of the one of the plurality of second storage systems, if the one of the plurality of identification information is designated by one of the plurality of another instructions.

72. A computer program according to claim 68, wherein each of the plurality of identification information are a marker number.

73. A computer program according to claim 69, wherein a value of one of the plurality of identification information is different from a value of one of the plurality of new identification information.

74. A computer program according to claim 68, wherein each of the plurality of identification information sent to the plurality of first storage systems includes same information.

75. A computer program according to claim 73, wherein each of the plurality of identification information sent to the plurality of first storage systems includes same information.

76. A computer program according to claim 68, wherein the plurality of identification information are stored in a plurality of memories of the plurality of first storage systems during the plurality of deferment processes.

77. A computer program according to claim 75, wherein one of the plurality of identification information is stored in a memory of one of the plurality of first storage systems during one of the plurality of deferment processes.

78. A computer program according to claim 68, wherein the plurality of identification information are used for establishing consistency between a plurality of replicated data stored in the second logical volumes of the plurality of second storage systems.

79. A computer program according to claim 68, wherein the plurality of identification information are used for storing the plurality of write data, stored in second storage areas of the plurality of second storage systems, in the second logical volumes of the plurality of second storage systems, if the plurality of identification information are designated by the plurality of another instructions.

80. A computer program according to claim 68, wherein the plurality of first storage systems and the plurality of second storage systems configure a plurality of pairs, each including the first logical volume of one of the plurality of first storage systems and the second logical volume of one of the plurality of second storage systems.

81. A computer program according to claim 68, further comprising:

code controlling to read the plurality of identification information from the plurality of second storage systems; and

code controlling to determine to designate the plurality of identification information.

82. A computer program according to claim 68, wherein one of the plurality of identification information relate to a marker.

83. A computer program stored on a computer readable storage medium and implemented in at least one first storage system, the at least one first storage system adaptive to be included in a plurality of first storage systems, the plurality of first storage systems each comprising a first control section for receiving data from one or more computers, one or more disk drives related to at least one first logical volume, and a first storage area for storing data to be transmitted to at least one of a plurality of second storage systems, the plurality of second storage systems each comprising a second control section, one or more second disk drives related to at least one second logical volume, and a second storage area for storing data received from at least one of the plurality of first storage systems, the computer program comprising:

code controlling to forming at least one of a plurality of pairs by using the first logical volume of the at least one first storage system and the second logical volume of the at least one of the plurality of second storage systems;

code controlling to receive a plurality of first write requests from the one or more computers and to store a plurality of first write data, which are received according to the plurality of first write requests and are stored in the first logical volume of the at least one of the plurality of first storage systems, in the first storage area of the at least one of the plurality of first storage systems;

code controlling to record an identification information received from a management software in the one or more computers and to transmit a report to the management software after recording the identification information;

code controlling to receive an instruction, for cancellation of a deferment process, from the management software after the plurality of first storage systems send a plurality of reports including the report to the management software, the deferment process being a process in which a plurality of second write requests received from the one or more computers are deferred; and

code controlling to transfer the plurality of first write data and the identification information to the at least one of the plurality of second storage systems, so that the plurality of second storage systems receive a plurality of write data including the plurality of first write data and a plurality of identification information including the identification information and control operations according to a plurality of another instructions sent from at least one of the management software and another management software in one or more other computers coupled to the plurality of second storage systems such that the plurality of write data are stored in second logical volumes of the plurality of second storage systems, if the plurality of identification information are designated by the plurality of another instructions.

84. A computer program according to claim 83, wherein code, controlling to receive the plurality of first write requests from the one or more computers and to store a plurality of first write data in the first storage area of the at least one of the plurality of first storage systems, controls to receive a plurality of new first write requests from the one or more computers and to store a plurality of new first write data, which are received according to the plurality of new first write requests and are stored in the first logical volume of the at least one of the plurality of first storage systems, in the first storage area of the at least one of the plurality of first storage systems,

wherein code, controlling to record the identification information received from the management software in the one or more computers and to transmit the report to the management software after recording the identification information, controls to record a new identification information received from the management software in the one or more computers and to transmit a new report to the management software after recording the new identification information,

wherein code, controlling to receive the instruction, for cancellation of the deferment process, from the management software, controls to receive a new instruction, for cancellation of a new deferment process, from the management software after the plurality of first storage systems send a plurality of new reports including the new report to the management software, the new deferment process being a process in which a plurality of new second write requests received from the one or more computers are deferred, and

wherein code, controlling to transfer the plurality of first write data and the identification information to the at least one of the plurality of second storage systems, controls to transfer the plurality of new first write data and the new identification information to the at least one of the plurality of second storage systems, so that the plurality of second storage systems receive a plurality of new write data including the plurality of new first write data and a plurality of new identification information including the new identification information.

85. A computer program according to claim 83, wherein code, controlling to receive the plurality of first write requests from the one or more computers and to store a plurality of first write data in the first storage area of the at least one of the plurality of first storage systems, (A) controls to receive a plurality of new first write requests from the one or more computers and to store a plurality of new first write data, which are received according to the plurality of new first write requests and are stored in the first logical volume of the at least one of the plurality of first storage systems, in the first storage area of the at least one of the plurality of first storage systems;

wherein code, controlling to record the identification information received from the management software in the one or more computers and to transmit the report to the management software after recording the identification information, (B) controls to record a new identification information received from the management software in the one or more computers and to transmit a new report to the management software after recording the new identification information;

wherein code, controlling to receive the instruction, for cancellation of the deferment process, from the management software, (C) controls to receive a new instruction, for cancellation of a new deferment process, from the management software after the plurality of first storage systems send a plurality of new reports including the new report to the management software, the new deferment process being a process in which a plurality of new second write requests received from the one or more computers are deferred;

wherein code, controlling to transfer the plurality of first write data and the identification information to the at least one of the plurality of second storage systems, (D) controls to transfer the plurality of new first write data and the new identification information to the at least one of the plurality of second storage systems, so that the plurality of second storage systems receive a plurality of new write data including the plurality of new first write data and a plurality of new identification information including the new identification information; and

wherein functions (A) to (D) of the wherein clauses are performed repeatedly.

86. A computer program according to claim 83, wherein each of the plurality of identification information are a marker number.

87. A computer program according to claim 84, wherein a value of one of the plurality of identification information is different from a value of one of the plurality of new identification information.

88. A computer program according to claim 83, wherein each of the plurality of identification information sent to the plurality of first storage systems includes same information.

89. A computer program according to claim 83, wherein each of the plurality of identification information sent to the plurality of first storage systems includes same information.

90. A computer program according to claim 83, wherein the identification information is stored in a memory of the at least one of the plurality of first storage systems during the deferment process.

91. A computer program according to claim 83, wherein the plurality of identification information are used for establishing consistency between a plurality of replicated data stored in the second logical volumes of the plurality of second storage systems.

92. A computer program according to claim 83, wherein the plurality of identification information are used for storing the plurality of write data, stored in second storage areas of the plurality of second storage systems, in the second logical volumes of the plurality of second storage systems, if the plurality of identification information are designated by the plurality of another instructions.

93. A computer program according to claim 83, wherein the identification information relates to a marker.

94. A remote copy system comprising:

a plurality of first storage systems, each comprising a first control section to be coupled to at least a computer and at least one first logical volume; and

a plurality of second storage systems, each coupled to a first storage system and comprising a second control section, at least one second logical volume for storing data sent from said at least first storage system,

wherein each of first control sections of the plurality of first storage systems stores a plurality of first write data from the computer,

wherein each of the first control sections records an identification information received from a management software in the computer and transmits a completion report to the management software after recording the identification information,

wherein each of the plurality of first storage systems receives a cancellation request for cancellation of a deferment process from the management software after each of the plurality of first storage systems sends the completion report to the management software, the deferment process being a process in which write processes of a plurality of second write requests received from the computers are deferred,

wherein each of the first control sections transfers the plurality of first write data and the identification information to at least one of the plurality of second storage systems according to an asynchronous remote copy process, and

wherein each of second control sections of the plurality of second storage systems receives the plurality of first write data and the identification information from at least one of the plurality of first storage systems; and

wherein the plurality of first write data received from at least one of the plurality of first storage systems are stored in the second logical volume when the identification information is designated by another instruction sent from the management software or the other management software.