Multi-node computer system component proactive monitoring and proactive repair

• US 20070214255A1
• Filed: 03/08/2006
• Published: 09/13/2007
• Est. Priority Date: 03/08/2006
• Status: Active Grant

• Current Assignee: Harmonic Incorporated
• Original Assignee: Omneon Video Networks
• Inventors: Sfarti, Adrian, Spitz, Mark
• Granted Patent: US 7,721,157 B2
• Time in Patent Office: Days
• US Class Current: 709/224
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: H04L 41/0803 Configuration setting
• CPC Class Codes: H04L 41/085 Retrieval of network configuration; Tracking network configuration history
• CPC Class Codes: H04L 41/147 for predicting network behaviour

Abstract

A highly available multi-node computer system is operated by monitoring the aging and usage of a plurality of hardware components that are part of the system'"'"'s networked nodes. While monitoring the components, a determination is made that one of the components has aged, worn, or both, to a level that is selected as being close enough to the component'"'"'s predicted end of life in the system so as to prevent failure of the component in the system. A notification is sent to replace the component, in response to the determination. Other embodiments are also described and claimed.

Claims

1. A data storage system comprising:

a plurality of metadata server machines each to store metadata for a plurality of files that are stored in the system;

a plurality of slice server machines to store slices of said files at locations, in the slice server machines, indicated by the metadata;

a packet switching interconnect to which the metadata and slice servers are communicatively coupled;

a distributed file system to be executed in the metadata and slice server machines, the file system to hide complexity of the data storage system from clients; and

software to monitor the aging and usage of hardware components of the data storage system and implement a proactive component replacement process which sends a notification to replace a component in response to determining that the component has aged, worn, or both, to a predetermined level that is selected as being dose enough to the component'"'"'s predicted end of life in the system so as to prevent failure of the component in the system.

2. The system of claim 1 wherein the metadata server machines, slice server machines, packet switching interconnect, and distributed file system implement redundancy such that any packet switch that is part of the packet switching interconnect can be removed and replaced while the system is running without experiencing any loss of stored data.

3. The system of claim 1 wherein the software is to send the notification to an entity that will schedule a service visit to replace the aged or worn component in the system.

4. The system of claim 1 wherein the software is to be executed in the metadata server machines.

5. The system of claim 2 wherein the software is to monitor the aging and usage of hardware components including fans, power supplies, cables, mass storage units, and network switches.

6. A computer-implemented method for operating a high availability multi-node computer system, comprising:

monitoring the aging and usage of a plurality of hardware components that are part of a plurality of networked nodes of the computer system;

while monitoring the components, determining that one of the components has aged, worn, or both, to a level that is selected as being close enough to the component'"'"'s predicted end of life in the system so as to prevent failure of the component in the system; and

sending a notification to replace the component in response to the determination.

7. The method of claim 6 wherein the monitoring comprises tracking how many hours a mass storage unit that is part of the system has been operating at different levels including idle and standby.

8. The method of claim 7 wherein the monitoring further comprises tracking the total number of internal movement operations involved in read and write operations to the mass storage unit.

9. The method of claim 8 wherein the monitoring further comprises tracking the total number of power up to power down cycles of the mass storage unit.

10. The method of claim 9 wherein the monitoring further comprises obtaining build date and burn in time of the mass storage unit.

11. The method of claim 10 wherein the monitoring further comprises tracking I/O errors in the mass storage unit.

12. The method of claim 11 wherein the monitoring further comprises tracking fan operation in the system.

13. The method of claim 12 wherein the monitoring further comprises tracking operating temperature in the system.

14. The method of claim 6 further comprising:

analyzing degradation of the components in the system including calculating a reliability or degradation factor for each component, based on the monitoring; and

calculating each component'"'"'s age or wear level in the system based on the reliability factor to predict when the component should be replaced before it fails.

15. The method of claim 14 wherein the analyzing comprises calculating reliability factors for network connection cables and PHYs in the system, based on analyzed trends in dropped, corrupted, or corrected network packets.

16. A computer-implemented method for operating a multi-node computer system, comprising:

monitoring the aging and usage of a plurality of hardware components that make up the computer system;

analyzing degradation of the components including calculating a degradation factor for each component, based on the monitoring; and

calculating each component'"'"'s age or wear level in the system, based on the degradation factor, that predicts when the component should be replaced before it fails in the system.

17. The method of claim 16 further comprising sending a notification to replace one of the components in response to determining that said component'"'"'s calculated age or wear level has a reached a threshold level indicating that it should be replaced to avoid its failure in the system.

18. A method for operating a multi-node data storage system, comprising:

receiving a notification from a multi-node data storage system that identifies an operating component of the system as being ready to be replaced; and

scheduling a service visit to replace the identified component from the system.