Method and apparatus to monitor stress conditions in a system

• US 7,424,396 B2
• Filed: 09/26/2005
• Issued: 09/09/2008
• Est. Priority Date: 09/26/2005
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Mukherjee, Udayan, Wei, Wen, Chandwani, Neelam, Dodeja, Rakesh, Hiremath, Chetan
• Primary Examiner(s): Raymond; Edward
• Application Number:
  US11/235,467
• Publication Number:
  US 20070089011A1
• Time in Patent Office: 1,079 Days
• Field of Search: 702/188, 702/29, 702/33, 702/35, 702/113, 702/114, 702/115, 702/50, 702/127, 702121-123, 702/130, 702/138, 702/145, 702176-178, 702182-185, 714/25, 714/48, 701/29, 701/31, 701/32, 701/35
• US Class Current: 702/185
• CPC Class Codes: G05B 23/0235 based on a comparison with predetermined threshold or range, e.g. "classical methods", carried out during normal operation; threshold adaptation or choice; when or how to compare with the threshold
• CPC Class Codes: G06F 11/3044 where the computing system component is the mechanical casing of the computing system
• CPC Class Codes: G06F 11/3058 Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations thermal management in cooling arrangements of a computing system G06F1/206

Abstract

Faults are monitored with information from agents for a plurality of sensors located on a plurality of circuit boards. A policy containing a error event thresholds against which the stored sensor information can be compared. Actions can be initiated by a fault module when one or more of the error event thresholds is exceeded.

Claims

1. A method of monitoring faults, comprising:

storing information from a plurality of agents for a plurality of sensors located on a plurality of circuit boards;

storing a policy containing a plurality of error event thresholds against which the stored sensor information can be compared;

counting events corresponding to instances of the error event thresholds being exceeded;

initiating an action, by a fault module, based on the counted events, and

monitoring sensors for a chassis cooling fan.

2. The method according to claim 1, further comprising predicting a device failure based upon the counted events,

wherein initiating an action comprises initiating an action in response to the predicted device failure.

3. The method according to claim 1 wherein storing information comprises storing temperature and voltage information for a plurality of integrated circuits, and

wherein counting events comprises counting events in which specified temperature and/or voltage parameters exceed corresponding error event thresholds.

4. The method according to claim 3 wherein storing information comprises storing frequency information, and

wherein counting events comprises counting events in which specified temperature, voltage and/or frequency parameters exceed corresponding error event thresholds.

5. The method according to claim 1, further comprising predicting failures based upon the counted events.

6. The method according to claim 5 predicting failures using a leaky bucket counter a leaky bucket counter.

7. The method according to claim 1, further comprising predicting failures based upon a rate that the counted events accumulate.

8. The method according to claim 1 wherein initiating the action comprises generating an operator alarm.

9. The method according to claim 1, wherein the fan sensors include at least three of fan current, fan speed monitor, intake and outtake air temperature, fan motor temperature, intake and outtake differential, noise level, vibration.

10. The method according to claim 9, further including measure fan power over time.

11. The method according to claim 9, further including monitoring three or more of fan spin up time, start/stop count, power on hour count, power cycles, power consumed, temperature, spin high current, spin noise, spin speed, pressure differential.

12. A system, comprising:

a plurality of sensors for collecting information for a plurality of operating parameters;

a database to store the sensor information; and

a fault module to examine the sensor information and count events in which specified operating parameters are exceeded to predict device failures; and

a chassis to contain a plurality of blades.

13. The system according to claim 12, wherein the specified operating parameters include temperature and voltage.

14. The system according to claim 13, wherein certain ones of the temperature and voltage sensors are located proximate integrated circuits.

15. The system according to claim 12, wherein the fault module includes a leaky bucket counter to predict failures.

16. The system according to claim 12, wherein the fault module is implemented as a set of native instruction set on the processor optimized for implementing failure prediction mechanisms.

17. The system according to claim 12, wherein the fault module is implemented as a hardware functional block in an application specific integrated circuit.

18. The system according to claim 12, wherein the fault module is implemented as a hardware functional block instantiated in a processor core.

19. The system according to claim 12, wherein the fault module is implemented as a hardware functional block instantiated in a memory controller chipset.

20. The system according to claim 12, wherein the fault module is implemented as combination of hardware functional blocks instantiated in a processor core and chipset.

21. The system according to claim 12, wherein the fault module is implemented as a hardware functional block in a field programmable gate array.

22. The system according to claim 12, wherein the fault module is implemented on a dedicated core in a many-core or a multi-core processor architecture.

23. A system, comprising:

a plurality of sensors for collecting information for a plurality of operating parameters;

a database to store the sensor information; and

a fault module to examine the sensor information and count events in which specified operating parameters are exceeded to predict device failures,

wherein the plurality of sensors includes fan sensors to predict failure of a cooling fan.

24. The system according to claim 23, wherein the fan sensors include at least three of fan current, fan speed monitor, intake and outtake air temperature, fan motor temperature, intake and outtake differential, noise level, and vibration.