Preventing unnecessary data recovery

• US 9,898,360 B1
• Filed: 12/28/2015
• Issued: 02/20/2018
• Est. Priority Date: 02/25/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Schrock, Christian Eric, Cypher, Robert, Schirripa, Steven Robert
• Primary Examiner(s): Kudirka, Joseph
• Application Number:
  US14/980,633
• Time in Patent Office: 785 Days
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0757 by exceeding a time limit, i.e. time-out, e.g. watchdogs
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1088 Reconstruction on already foreseen single or plurality of spare disks

Abstract

A method that prevents unnecessary data recovery includes receiving, at a data processing device, a status of a resource of a distributed system. When the status of the resource indicates a resource failure, the method includes executing instructions on the data processing device to determine whether the resource failure is correlated to any other resource failures within the distributed system. When the resource failure is correlated to other resource failures within the distributed system, the method includes delaying execution on the data processing device of a remedial action associated with the resource. However, when the resource failure is uncorrelated to other resource failures within the distributed system, the method includes initiating execution on the data processing device of the remedial action associated with the resource.

Claims

1. A method comprising:

receiving, at a data processing device, a status of a resource of a distributed system;

when the status of the resource indicates a resource failure, executing instructions on the data processing device to determine whether the resource failure is correlated to any other resource failures within the distributed system based on a system hierarchy of the distributed system, the system hierarchy comprising system domains, each system domain having an active state or an inactive state, the resource belonging to at least one system domain, wherein the resource failure is correlated to other resource failures when a statistically significant number of resources having failures reside in a same system domain;

when the resource failure is correlated to other resource failures within the distributed system, delaying execution on the data processing device of a remedial action associated with the resource; and

when the resource failure is uncorrelated to other resource failures within the distributed system, initiating execution on the data processing device of the remedial action associated with the resource.

2. The method of claim 1, further comprising, when the resource comprises non-transitory memory, initiating data reconstruction as the remedial action for any data stored on the non-transitory memory.

3. The method of claim 2, wherein the data comprises chunks of a file, the file divided into stripes comprising data chunks and non-data chunks.

4. The method of claim 1, further comprising, when the resource comprises a computer processor, migrating or restarting a job previously executing on a failed computer processor to an operational computer processor.

5. The method of claim 1, further comprising determining the resource failure as correlated to other resource failures when the resource resides in an inactive system domain.

6. The method of claim 1, wherein the system hierarchy comprises system levels comprising:

a first system level corresponding to host machines of data processing devices, non-transitory memory devices, or network interface controllers, each host machine having a system domain;

a second system level corresponding to power deliverers, communication deliverers, or cooling deliverers of racks housing the host machines, each power deliverer, communication deliverer, or cooling deliverer of the rack having a system domain;

a third system level corresponding to power deliverers, communication deliverers, or cooling deliverers of cells having associated racks, each power deliverer, communication deliverer, or cooling deliverer of the cell having a system domain; and

a fourth system level corresponding to a distribution center module of the cells, each distribution center module having a system domain.

7. A method comprising:

receiving, at a data processing device, a status of a resource of a distributed system;

when the status of the resource indicates a resource failure, executing instructions on the data processing device to determine whether the resource failure is correlated to any other resource failures within the distributed system based on a system hierarchy of the distributed system, the system hierarchy comprising system domains, each system domain having an active state or an inactive state, the resource belonging to at least one system domain;

when the resource failure is correlated to other resource failures within the distributed system, delaying execution on the data processing device of a remedial action associated with the resource until after a first threshold period of time; and

when the resource failure is uncorrelated to the other resource failures within the distributed system, initiating execution on the data processing device of the remedial action associated with the resource after a second threshold period of time,

wherein the first threshold period of time is greater than the second threshold period of time.

8. The method of claim 7, wherein the second threshold period of time is between about 15 minutes and about 30 minutes.

9. A recovery system for a distributed system, the recovery system comprising:

a data processing device in communication with resources of the distributed system, the data processing device receiving a status of a resource of the distributed system;

when the status of the resource indicates a resource failure, the data processing device executing instructions to determine whether the resource failure is correlated to any other resource failures within the distributed system based on a system hierarchy of the distributed system, the system hierarchy comprising system domains, each system domain having an active state or an inactive state, the resource belonging to at least one system domain, wherein the resource failure is correlated to other resource failures when a statistically significant number of resources having failures reside in a same system domain;

when the resource failure is correlated to other resource failures within the distributed system, the data processing device delaying execution of a remedial action associated with the resource; and

when the resource failure is uncorrelated to other resource failures within the distributed system, the data processing device initiating execution of the remedial action associated with the resource.

10. The recovery system of claim 9, wherein, when the resource comprises non-transitory memory, the data processing device initiates data reconstruction as the remedial action for any data stored on the non-transitory memory.

11. The recovery system of claim 10, wherein the data comprises chunks of a file, the file divided into stripes comprising data chunks and non-data chunks.

12. The recovery system of claim 9, wherein, when the resource comprises a computer processor, the data processing device migrates or restarts a job previously executing on a failed computer processor to an operational computer processor.

13. The recovery system of claim 9, wherein the data processing device determines the resource failure as correlated to other resource failures, when the resource resides in an inactive system domain.

14. The recovery system of claim 9, wherein the system hierarchy comprises system levels comprising:

a first system level corresponding to host machines of data processing devices, non-transitory memory devices, or network interface controllers, each host machine having a system domain;

a second system level corresponding to power deliverers, communication deliverers, or cooling deliverers of racks housing the host machines, each power deliverer, communication deliverer, or cooling deliverer of the rack having a system domain;

a third system level corresponding to power deliverers, communication deliverers, or cooling deliverers of cells having associated racks, each power deliverer, communication deliverer, or cooling deliverer of the cell having a system domain; and

a fourth system level corresponding to a distribution center module of the cells, each distribution center module having a system domain.

15. A recovery system for a distributed system, the recovery system comprising:

a data processing device in communication with resources of the distributed system, the data processing device receiving a status of a resource of the distributed system;

when the status of the resource indicates a resource failure, the data processing device executing instructions to determine whether the resource failure is correlated to any other resource failures within the distributed system based on a system hierarchy of the distributed system, the system hierarchy comprising system domains, each system domain having an active state or an inactive state, the resource belonging to at least one system domain;

when the resource failure is correlated to other resource failures within the distributed system, the data processing device delays execution of a remedial action associated with the resource for a first threshold period of time; and

when the resource failure is uncorrelated to the other resource failures within the distributed system, the data processing device initiates execution of the remedial action associated with the resource after a second threshold period of time,

wherein the first threshold period of time is greater than the second threshold period of time.

16. The recovery system of claim 15, wherein the second threshold period of time is between about 15 minutes and about 30 minutes.

17. A method comprising:

receiving, at a data processing device, a status of a resource of a distributed system;

when the status of the resource indicates a resource failure, executing instructions on the data processing device to determine;

a correlation between the resource failure and any other resource failures within the distributed system based on a system hierarchy of the distributed system, the system hierarchy comprising system domains, each system domain having an active state or an inactive state, the resource belonging to at least one system domain; and

a time duration of the resource failure;

when the resource failure is correlated to other resource failures within the distributed system and the time duration is greater than a first threshold period of time, executing on the data processing device a remedial action associated with the resource; and

when the resource failure is uncorrelated to other resource failures within the distributed system, and the time duration is greater than a second threshold period of time, executing on the data processing device the remedial action associated with the resource, wherein the first threshold period of time is greater than the second threshold period of time.

18. The method of claim 17, further comprising determining the resource failure as correlated to other resource failures, when the resource resides in an inactive system domain.

19. The method of claim 17, wherein the system hierarchy comprises system levels comprising:

a first system level corresponding to host machines of data processing devices, non-transitory memory devices, or network interface controllers, each host machine having a system domain;

a second system level corresponding to power deliverers, communication deliverers, or cooling deliverers of racks housing the host machines, each power deliverer, communication deliverer, or cooling deliverer of the rack having a system domain;

a third system level corresponding to power deliverers, communication deliverers, or cooling deliverers of cells having associated racks, each power deliverer, communication deliverer, or cooling deliverer of the cell having a system domain; and

a fourth system level corresponding to a distribution center module of the cells, each distribution center module having a system domain.

20. A method comprising:

receiving, at a data processing device, a status of a resource of a distributed system;

when the status of the resource indicates a resource failure, executing instructions on the data processing device to determine;

a correlation between the resource failure and any other resource failures within the distributed system based on a system hierarchy of the distributed system, the system hierarchy comprising system domains, each system domain having an active state or an inactive state, the resource belonging to at least one system domain; and

a time duration of the resource failure;

when the resource failure is correlated to other resource failures within the distributed system and the time duration is greater than a first threshold period of time, executing on the data processing device a remedial action associated with the resource;

when the resource failure is uncorrelated to other resource failures within the distributed system, and the time duration is greater than a second threshold period of time, executing on the data processing device the remedial action associated with the resource, wherein the first threshold period of time is greater than the second threshold period of time;

when the resource comprises non-transitory memory, initiating data reconstruction as the remedial action for any data stored on the non-transitory memory; and

when the resource comprises a computer processor, migrating or restarting a job previously executing on a failed computer processor to an operational computer processor.

21. The method of claim 20, further comprising determining the resource failure as correlated to other resource failures, when a statistically significant number of resources having failures reside in a same system domain.