Predictive failure of hardware components

• US 10,346,239 B1
• Filed: 06/27/2016
• Issued: 07/09/2019
• Est. Priority Date: 06/27/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Ortega Gutierrez, Felipe Enrique, Ebisuzaki, Gavin Akira, BeSerra, Christopher James
• Primary Examiner(s): Duncan, Marc
• Assistant Examiner(s): Gibson, Jonathan D
• Application Number:
  US15/194,180
• Time in Patent Office: 1,107 Days
• CPC Class Codes: G06F 1/20 Cooling means
• CPC Class Codes: G06F 1/203 for portable computers, e.g. for laptops
• CPC Class Codes: G06F 1/206 comprising thermal management
• CPC Class Codes: G06F 1/266 Arrangements to supply power to external peripherals either directly from the computer or under computer control, e.g. supply of power through the communication port, computer controlled power-strips
• CPC Class Codes: G06F 1/28 Supervision thereof, e.g. detecting power-supply failure by out of limits supervision
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/07 Responding to the occurrence of a fault, e.g. fault tolerance
• CPC Class Codes: G06F 11/0736 in functional embedded systems, i.e. in a data processing system designed as a combination of hardware and software dedicated to performing a certain function testing or monitoring of automated control systems G05B23/02
• CPC Class Codes: G06F 11/0754 by exceeding limits
• CPC Class Codes: G06F 11/0766 Error or fault reporting or storing
• CPC Class Codes: G06F 11/0778 Dumping, i.e. gathering error/state information after a fault for later diagnosis
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3031 where the computing system component is a motherboard or an expansion card
• CPC Class Codes: G06F 11/3041 where the computing system component is an input/output interface interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units G06F13/00
• CPC Class Codes: G06F 11/3058 Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations thermal management in cooling arrangements of a computing system G06F1/206
• CPC Class Codes: G06F 11/3062 where the monitored property is the power consumption power management in a computing system G06F1/3203
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 2201/81 Threshold

Abstract

A system is described wherein power degradation can be used in conjunction with predictive failure analysis in order to accurately determine when a hardware component might fail. In one example, printed circuit boards (PCBs) can unexpectedly malfunction due to a variety of reasons including silicon power variation or air mover speed. Other hardware components can include silicon or an integrated circuit. In order to accurately monitor the hardware component, telemetry is used to automatically receive communications regarding measurements of data associated with the hardware component, such as power-related data or temperature data. The different temperature data can include junction temperature or ambient air temperature to determine an expected power usage. The actual power usage is then compared to the expected power usage to determine whether the hardware component can soon fail.

Claims

1. A computer-readable storage medium including instructions that upon execution cause a computer system to:

receive, in a management controller, from a hardware component, a measured power usage of the hardware component, wherein the hardware component is an integrated circuit or a printed circuit board including multiple integrated circuits, and wherein the hardware component is within a server computer that includes the management controller;

receive from the hardware component, a measured junction temperature of the hardware component and an air mover speed;

calculate an expected power usage based on the junction temperature, and the air mover speed;

compare the expected power usage to the measured power usage; and

transmit a predictive failure indication if the measured power usage exceeds the expected power usage by a threshold amount so as to provide notification that the hardware component is failing.

2. The computer-readable storage medium of claim 1, wherein the instructions, upon execution, further cause the computer system to:

receive an ambient air temperature upstream of the hardware component; and

calculate the expected power usage based upon the junction temperature and the ambient air temperature.

3. The computer-readable storage medium of claim 1, wherein the hardware component is silicon of an integrated circuit.

4. The computer-readable storage medium of claim 1, wherein the expected power usage is determined using the following formula:

Power expected=(T_{junction_measured}−

T_{ambient_measured})/thermal resistance;

wherein T_{junction_measured} is a measured temperature internal to the hardware component, T_{ambient_measured} is an ambient temperature outside of the hardware component, and the thermal resistance is of the hardware component and wherein the Power expected is calculated for different levels of workload of the hardware component ranging from a low utilization level of the hardware component to a high utilization level.

5. A method of predictively determining a hardware failure, comprising:

receiving a transmission in a server computer that includes data indicating a measured power usage of a hardware component and a transmission including an air mover speed in the server computer;

calculating an expected power usage as a function of junction temperature of the hardware component, an ambient temperature upstream of the hardware component, and an air mover speed associated with an air mover used to cool the hardware component;

comparing the measured power usage of the hardware component to the expected power usage; and

transmitting a predictive failure notification upon determining that the measured power usage exceeds the expected power by a threshold amount so as to indicate that the hardware component is failing.

6. The method of claim 5, wherein the expected power usage is determined using a lookup table.

7. The method of claim 5, wherein the expected power usage is determined using the following formula:

Power expected=(T_{junction_measured}−

T_{ambient_measured})/thermal resistance;

wherein T_{junction_measured} is a measured temperature internal to the hardware component, T_{ambient_measured} is an ambient temperature outside of the hardware component, and the thermal resistance is of the hardware component, and wherein the Power expected is calculated for different levels of workload ranging from a low utilization level of the hardware component to a high utilization level.

8. The method of claim 7, wherein the thermal resistance is a function of fluid flow that passes the hardware component.

9. The method of claim 8, wherein the fluid flow is associated with the air mover speed or pump speed.

10. The method of claim 5, further including a management controller coupled to the hardware component, the management controller including a processor that receives the transmission including the data indicating the power usage, and that transmits the predictive failure notification.

11. A system, comprising:

a hardware component having a power measurement sub-component built therein and a junction temperature sub-component built therein, the power measurement sub-component for determining power used by the hardware component, and the junction temperature sub-component for determining a junction temperature;

a pump that moves fluid past the hardware component, the pump being an air mover or a liquid pump to cool the hardware component; and

a management controller coupled to the hardware component, the management controller for calculating an expected power usage based on the junction temperature received from the hardware component and based on a measurement received from the pump, and for calculating a difference between the power determined by the power measurement component and the expected power usage, wherein the management controller is for transmitting a predictive failure notification when the difference exceeds a threshold so as to indicate that the hardware component is likely to fail and wherein the management controller is within a server computer and the hardware component is silicon within an integrated circuit.

12. The system of claim 11, wherein the management controller is a baseboard management controller (BMC) on a server motherboard.

13. The system of claim 11, further including an ambient temperature component to measure ambient temperature associated with the hardware component, wherein the management controller is coupled to the ambient temperature component and calculates the expected power usage based on the ambient temperature.