Implementing rate controls to limit timeout-based faults

• US 9,229,839 B2
• Filed: 01/09/2013
• Issued: 01/05/2016
• Est. Priority Date: 01/09/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Singh, Abhishek, Raghavan, Srikanth, Mani, Ajay, Syed, Saad
• Primary Examiner(s): DUNCAN, MARC M
• Application Number:
  US13/737,430
• Publication Number:
  US 20140195861A1
• Time in Patent Office: 1,091 Days
• Field of Search: 714/47.2, 714/55, 714/704, 714/708
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/076 by exceeding a count or rate limit, e.g. word- or bit count limit
• CPC Class Codes: G06F 11/1402 Saving, restoring, recovering or retrying
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring

Abstract

Embodiments are directed to implementing rate controls to limit faults detected by timeout and to learning and adjusting an optimal timeout value. In one scenario, a computer system identifies cloud components that have the potential to fail within a time frame that is specified by a timeout value. The computer system establishes a number of components that are allowed to fail during the time frame specified by the timeout value and further determines that the number of component failures within the time frame specified by the timeout value has exceeded the established number of components that are allowed to fail. In response, the computer system increases the timeout value by a specified amount of time to ensure that fewer than or equal to the established number of components fail within the time frame specified by the timeout value.

Claims

1. A computer system comprising the following:

one or more processors;

system memory;

one or more computer-readable storage media having stored thereon computer-executable instructions that are executable by the one or more processors to cause the computing system to implement rate controls to limit faults detected by timeout and to instantiate the following;

a monitor module that identifies one or more hardware or software components that have a potential to experience a timeout-based failure within a time frame, wherein the timeout-based failure is a failure in which the one or more hardware or software components is unresponsive for a specified time or takes longer to perform a task than the specified time, wherein the specified time is specified by a timeout value;

a component failure module that establishes a number of timeout-based failures the one or more hardware or software components are allowed to suffer during the time frame;

a determining module that determines that the number of timeout-based failures suffered by the one or more hardware or software components within the time frame has exceeded the established number; and

a timeout value adjusting module that increases the timeout value by a specified amount of time to ensure that fewer than or equal to the established number of timeout-based failures occur within the time frame.

2. The computer system of claim 1, wherein the established number is based on a value that indicates a number of cloud components that failed over a period of time.

3. The computer system of claim 2, wherein the established number is based on the value plus a buffer value.

4. The computer system of claim 1, wherein the computer-executable instructions are further configured, when executed, to cause the computing system to determine

that the number of timeout-based failures suffered by the one or more hardware or software components within the time frame is below the established number;

and

to decrease the timeout value by a specified amount of time.

5. The computer system of claim 4, wherein a proportional-integral-derivative (PID) controller is used to decrease the timeout value.

6. The computer system of claim 4, wherein the timeout value is dynamically adjusted over time.

7. The computer system of claim 6, wherein the timeout value is dynamically adjusted to minimize the timeout value while ensuring that fewer than or equal to the established number of timeout-based failures suffered by the one or more hardware or software components occur within the time frame.

8. The computer system of claim 1, wherein increasing the timeout value by a specified amount of time to ensure that fewer than or equal to the established number of timeout-based failures suffered by the one or more hardware or software components occur within the time frame prevents one or more cascading component failures.

9. The computer system of claim 1, wherein the determining module determines that the number of timeout-based failures has exceeded the established number in response to the monitoring module monitoring one or more hardware or software cloud components using cloud component monitoring requests.

10. The computer system of claim 9, wherein the monitoring requests are sent to the one or more hardware or software cloud components on a scheduled basis according to a specified schedule.

11. The computer system of claim 10, wherein the specified schedule is based on a user-established policy.

12. The computer system of claim 10, wherein the timeout value is adjusted at specified time intervals.

13. The computer system of claim 12, wherein the specified time intervals for adjusting the timeout value are listed in a cloud management policy.

14. The computing system of claim 1, wherein the computer system adjusts a number of retries that are permitted to occur for the one or more hardware or software components, each retry comprising an attempted restart for a corresponding hardware or software component.

15. A computer system comprising the following:

one or more processors;

system memory;

one or more computer-readable storage media having stored thereon computer-executable instructions that are executable, by the one or more processors to cause, the computing system to perform learning and adjusting of a timeout value and to instantiate, the following;

a monitor module that monitors a number of timeout-based failures of one or more hardware or software components that occur due to timeout during a specified timeframe, the timeouts being defined by the timeout value, and wherein the timeout value is based on monitored time distributions for at least one of the following;

application deployments, application updates, virtual machine migrations or node power-downs;

a determining module that determines that the timeout value is too high or too low based on the determined number of timeout-based failures that occurred due to timeout during the specified timeframe; and

a timeout value adjusting module that adjusts the timeout value to ensure that fewer than or equal to a specified number of timeout-based failures occur during the specified timeframe.

16. The computer system of claim 15, wherein different component-specific timeout values are specified for a plurality of different components.

17. The computer system of claim 16, wherein the component-specific timeout values are adjusted according to a specified time interval.

18. The computer system of claim 17, wherein the component-specific timeout values are adjusted manually based on user input.

19. The computing system of claim 15, wherein the computer system adjusts a number of retries that are permitted to occur during the specified timeframe.

20. A computer system comprising the following:

one or more processors;

system memory;

one or more computer-readable storage media having stored thereon computer-executable instructions that are executable, by the one or more processors to cause the computing system to perform a method for learning and adjusting a timeout value and to instantiate the following;

a monitor module that monitors one or more hardware or software components for a number of hardware or software timeout-based failures of the one or more hardware or software components that occur due to timeout during a specified timeframe, each of the timeout-based failures comprising a failure in which the one or more hardware or software components is unresponsive for a specified time or takes longer to perform a task than the specified time, wherein the specified time is by the timeout value;

a determining module that determines that the timeout value is too high or too low based on the determined number of failures that occurred due to timeout during the specified timeframe; and

a timeout value adjusting module that adjusts the timeout value to ensure that fewer than or equal to a specified number of failures occur during the specified timeframe; and

wherein the computer system adjusts a number of retries that are permitted to occur for the one or more hardware or software components, each retry comprising an attempted restart for a corresponding hardware or software component.