Using a standby data storage system to detect the health of a cluster of data storage servers

• US 8,615,578 B2
• Filed: 10/07/2005
• Issued: 12/24/2013
• Est. Priority Date: 10/07/2005
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Hu, Wei, Li, Yunrui, Loaiza, Juan R.
• Primary Examiner(s): Swearingen, Jeffrey R
• Assistant Examiner(s): ELFERVIG, TAYLOR A
• Application Number:
  US11/245,868
• Publication Number:
  US 20070083641A1
• Time in Patent Office: 3,000 Days
• Field of Search: 709/224, 714/4.1, 714/4.11
• US Class Current: 709/224
• CPC Class Codes: H04L 41/0681 Configuration of triggering conditions
• CPC Class Codes: H04L 41/0893 Assignment of logical groups to network elements
• CPC Class Codes: H04L 43/00 Arrangements for monitoring or testing data switching networks
• CPC Class Codes: H04L 43/0817 by checking functioning
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Detecting an operating condition in a multi-server data storage system. One or more of the data storage servers are designated with a status within the cluster. The designating may be performed by at least one of the data storage servers. The status is encoded into log data having records that report changes to the primary data storage performed by the cluster of data storage servers. The log data is transmitted from the primary data storage to the standby data storage system. The standby data storage system determines an operating condition of the cluster of data storage servers based, at least in part, on the status.

Claims

1. A method of using a standby data storage system to detect a condition of a cluster of data storage servers that control changes to a primary data storage, said method comprising computer-implemented steps of:

transmitting a stream of data comprising log data from a change log to said standby data storage system, wherein the log data specifies changes made to data stored at said primary data storage, wherein said stream of data comprising log data is used by said standby data storage system to replicate said changes made to data stored at said primary data storage to data stored at said standby data storage system;

determining an operating status for one or more of said data storage servers within said cluster;

causing a message reporting said operating status to be transmitted inband within said stream itself to said standby data storage system; and

determining, at said standby data storage system, an operating condition of said cluster of data storage servers based, at least in part, on said message reporting said operating status;

wherein the method is performed by one or more computing devices.

2. A method as recited in claim 1, wherein said operating condition comprises a particular data storage server being a disabled member of said cluster.

3. A method as recited in claim 1, wherein said operating condition comprises a particular data storage server being an enabled member of said cluster.

4. A method as recited in claim 1, wherein said operating condition comprises said cluster operating as multiple clusters.

5. A method as recited in claim 4, wherein said determining said operating condition is further based on said changes.

6. A method as recited in claim 5, wherein said determining said operating condition comprises receiving, in said log data, an indication of a change to said primary data storage made by a first data storage server at a period in which said first data storage server is designated with an operating status of a disabled member of said cluster.

7. A method as recited in claim 4, further comprising:

said standby data storage system determining a time at which said cluster was determined to be operating as multiple clusters, based on information transmitted inband within said log data; and

said standby data storage system ignoring changes to said primary data storage reported as having occurred after said time at which said cluster was determined to be operating as multiple clusters.

8. A method as recited in claim 4, further comprising:

said standby data storage system replacing said primary data storage as a new primary data storage.

9. A method as recited in claim 4, wherein:

said determining said operating status for one or more of said data storage servers within said cluster comprises each enabled data storage server designating a set of enabled members in said cluster; and

said determining said operating condition comprises detecting an inconsistency among the designated sets of enabled members in said cluster.

10. A method as recited in claim 1, wherein said determining said operating status comprises a first of said data storage servers designating a second of said data storage servers as a disabled member of said cluster.

11. A method as recited in claim 10, wherein said determining said operating status further comprises encoding a time at which said second data of said storage server is designated as a disabled member of said cluster.

12. A method as recited in claim 1, wherein said determining said operating status comprises at least two of said data storage servers designating respective sets of data storage servers as enabled members in said cluster.

13. A method as recited in claim 1, wherein said primary data storage comprises a database.

14. A method as recited in claim 1, wherein said primary data storage comprises a file system.

15. A method as recited in claim 1, wherein said primary data storage comprises a storage array.

16. A non-transitory computer readable medium carrying one of more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to perform a method of using a standby data storage system to detect an operating condition of a cluster of data storage servers that control changes to a primary data storage, said method comprising the steps of:

transmitting a stream of data comprising log data from a change log to said standby data storage system, wherein the log data specifies changes made to data stored at said primary data storage, wherein said stream of data comprising log data is used by said standby data storage system to by replicate said changes made to data stored at said primary data storage to data stored at said standby data storage system;

determining an operating status for one or more of said data storage servers within said cluster;

causing a message reporting said operating status to be transmitted inband within said stream itself to said standby data storage system; and

determining, at said standby data storage system, an operating condition of said cluster of data storage servers based, at least in part, on said message reporting said operating status.

17. A computer readable medium as recited in claim 16, wherein said operating condition comprises a particular data storage server being a disabled member of said cluster.

18. A computer readable medium as recited in claim 16, wherein said operating condition comprises a particular data storage server being a disabled member of said cluster.

19. A computer readable medium as recited in claim 16, wherein said operating condition comprises said cluster operating as multiple clusters.

20. A computer readable medium as recited in claim 19, wherein said step of determining said operating condition in said method is further based on said changes.

21. A computer readable medium as recited in claim 20, wherein said step of determining said operating condition comprises receiving, in said log data, an indication of a change to said primary data storage made by a first data storage server at a period in which said first data storage server is designated with an operating status of a disabled member of said cluster.

22. A computer readable medium as recited in claim 19, wherein:

said step of determining said operating status for one or more of said data storage servers within said cluster comprises each enabled data storage server designating a set of enabled members in said cluster; and

said step of determining said operating condition comprises detecting an inconsistency among the designated sets of enabled members in said cluster.

23. A computer readable medium as recited in claim 16, wherein said method further comprises:

said standby data storage system determining a time at which said cluster was determined to be operating as multiple clusters, based on information transmitted inband within said log data; and

said standby data storage system ignoring changes to said primary data storage reported in said changes as having occurred after said time at which said cluster was determined to be operating as multiple clusters.

24. A computer readable medium as recited in claim 16, wherein said method further comprises said standby data storage system replacing said primary data storage as a new primary data storage.

25. A computer readable medium as recited in claim 16, wherein said step of determining said operating status in said method comprises a first of said data storage servers designating a second of said data storage servers as a disabled member of said cluster.

26. A computer readable medium as recited in claim 25, wherein said step of determining said operating status in said method comprises encoding a time at which said second of said data storage server is designated as a disabled member of said cluster.

27. A computer readable medium as recited in claim 16, wherein said step of determining said operating status in said method comprises at least two of said data storage servers designating respective sets of said data storage servers as enabled members in said cluster.

28. A computer readable medium as recited in claim 16, wherein said primary data storage comprises a database.

29. A computer readable medium as recited in claim 16, wherein said primary data storage comprises a file system.

30. A computer readable medium as recited in claim 16, wherein said primary data storage comprises a storage array.