Fault isolation using code paths

• US 8,458,511 B2
• Filed: 09/17/2010
• Issued: 06/04/2013
• Est. Priority Date: 09/17/2010
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Ng, Tsz Him (Adrian), Tsukerman, Alex, Umamageswaran, Kothanda, Hansen, Roger, Panchaksharaiah, Umesh
• Primary Examiner(s): DUNCAN, MARC M
• Application Number:
  US12/885,323
• Publication Number:
  US 20120072776A1
• Time in Patent Office: 991 Days
• Field of Search: 714/4.11, 714/4.12, 714/4.2, 714/11, 714/38.1
• US Class Current: 714/4.2
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites

Abstract

Techniques are provided for isolating faults in a software program by providing at least two code paths that are capable of performing the same operation. When a fault occurs while the one of the code paths is being used to perform an operation, data that indicates the circumstances under which the fault occurred is stored. For example, a fault-recording mechanism may store data that indicates the entities that were involved in the failed operation. Because they were involved in an operation that experienced a fault, one or more of those entities may be “quarantined”. When subsequent requests arrive to perform the operation, a check may be performed to determine whether the requested operation involves any of the quarantined entities. If the requested operation involves a quarantined entity, a different code path is used to perform the operation, rather than the code path from which the entity is quarantined.

Claims

1. A method comprising:

receiving a first request to perform a first operation that may be performed by a first code path within a server;

detecting occurrence of a fault during performance of the first operation by the first code path;

identifying a first entity involved in the first operation that experienced the fault;

wherein the first entity is a first type of entity, wherein the first entity is one of;

a SQL plan step, a SQL plan, a SQL statement, or a storage location targeted by an I/O request;

in response to the first entity being involved in the first operation that experienced the fault, determining that quarantine criteria for the first code path has been satisfied relative to the first entity;

in response to determining that quarantine criteria for the first code path has been satisfied relative to the first entity, storing data that indicates that the first entity is quarantined relative to the first code path while continuing to allow the first code path to be executed for operations involving other entities of the first type of entity;

after storing data that indicates that the first entity is quarantined relative to the first code path, receiving a second request to perform a second operation;

in response to the second request, determining whether the first entity is involved in the second operation;

in response to determining that the first entity is involved in the second operation and that the first entity is quarantined relative to the first code path, responding to the second request without using the first code path;

wherein the method is performed by one or more computing devices.

2. The method of claim 1 wherein the step of responding to the second request without using the first code path includes responding to the second request without performing the second operation.

3. The method of claim 1 wherein:

the first operation and the second operation are operations that may be performed by any one of a plurality of code paths within the server;

the plurality of code paths include the first code path and a second code path;

the second code path is different from the first code path; and

the step of responding to the second request without using the first code path includes selecting the second code path to perform the second operation.

4. The method of claim 3 wherein:

the quarantine criteria for the first code path applies to the first type of entity;

the first operation involves a second entity that is a second type of entity to which second quarantine criteria applies; and

the method further comprises, after the first operation experienced the fault, determining that the second quarantine criteria has not been satisfied relative to the second entity.

5. The method of claim 3 wherein the quarantine criteria for the first code path is satisfied by the first entity being in a particular number of operations, performed by the first code path, that experience faults.

6. The method of claim 5 wherein the particular number is greater than one.

7. The method of claim 3 further comprising:

receiving a third request to perform a third operation that may be performed by any one of the plurality of code paths within the server;

selecting the second code path to perform the third operation;

detecting occurrence of a fault during performance of the third operation by the second code path;

identifying a second entity involved in the third operation that experienced the fault;

in response to the second entity being involved in the third operation that experienced the fault, determining that quarantine criteria for the second code path has been satisfied relative to the second entity;

in response to determining that quarantine criteria for the second code path has been satisfied relative to the second entity, storing data that indicates that the second entity is quarantined relative to the second code path.

8. The method of claim 7 further comprising:

after storing data that indicates that the second entity is quarantined relative to the second code path, receiving a fourth request to perform a fourth operation that can be performed by any one of the plurality of code paths;

in response to the fourth request, determining whether the second entity is involved in the fourth operation;

in response to determining that the second entity is involved in the fourth operation and that the second entity is quarantined relative to the second code path, selecting a code path, of the plurality of code paths, other than the second code path, to perform the fourth operation.

9. The method of claim 8 wherein selecting a code path, of the plurality of code paths, other than the second code path, to perform the fourth operation includes selecting the first code path to perform the fourth operation.

10. The method of claim 3 wherein the first and second code paths are code paths within a storage server that services I/O requests from database servers.

11. The method of claim 10 wherein the first entity is a disk location.

12. The method of claim 1 wherein the first entity is a SQL plan step.

13. The method of claim 1 wherein the first entity is a SQL plan.

14. The method of claim 1 wherein the first entity is a SQL statement.

15. The method of claim 1 wherein the first entity is a storage location targeted by the particular I/O request.

16. A method comprising:

receiving a first request to perform a first operation that may be performed by a first code path within a server;

detecting occurrence of a fault during performance of the first operation by the first code path;

identifying a first entity involved in the first operation that experienced the fault;

in response to the first entity being involved in the first operation that experienced the fault, determining that quarantine criteria for the first code path has been satisfied relative to the first entity;

in response to determining that quarantine criteria for the first code path has been satisfied relative to the first entity, storing data that indicates that the first entity is quarantined relative to the first code path;

after storing data that indicates that the first entity is quarantined relative to the first code path, receiving a second request to perform a second operation;

in response to the second request, determining whether the first entity is involved in the second operation;

in response to determining that the first entity is involved in the second operation and that the first entity is quarantined relative to the first code path, responding to the second request without using the first code path;

wherein the first entity has a hierarchical relationship relative to a plurality of other entities;

wherein the quarantine criteria is satisfied relative to the first entity in response to a particular number of said other entities being quarantined relative to the first code path;

wherein the method is performed by one or more computing devices.

17. The method of claim 16 wherein the first entity is one of:

a SQL plan step, a SQL plan, or a database server.

18. A non-transitory computer-readable storage that stores instructions which, when executed by one or more processors, causes the one or more processors to perform steps comprising:

receiving a first request to perform a first operation that may be performed by any one of a plurality of code paths within a server;

selecting a first code path, of the plurality of code paths, to perform the first operation;

detecting occurrence of a fault during performance of the first operation by the first code path;

identifying a first entity involved in the first operation that experienced the fault;

wherein the first entity is a first type of entity, wherein the first entity is one of;

a SQL plan step, a SQL plan, a SQL statement, or a storage location targeted by an I/O request;

in response to the first entity being involved in the first operation that experienced the fault, determining that quarantine criteria for the first code path has been satisfied relative to the first entity;

in response to determining that quarantine criteria for the first code path has been satisfied relative to the first entity, storing data that indicates that the first entity is quarantined relative to the first code path while continuing to allow the first code path to be executed for operation involving other entity of the first type of entity;

after storing data that indicates that the first entity is quarantined relative to the first code path, receiving a second request to perform a second operation that can be performed by any one of the plurality of code paths;

in response to the second request, determining whether the first entity is involved in the second operation;

in response to determining that the first entity is involved in the second operation and that the first entity is quarantined relative to the first code path, selecting a second code path of the plurality of code paths to perform the second operation;

wherein the second code path is different from the first code path.

19. The non-transitory computer-readable storage of claim 18 wherein:

the quarantine criteria for the first path applies to the first type of entity;

the first operation involves a second entity that is a second type of entity to which second quarantine criteria applies; and

the steps further comprise, after the first operation experienced the fault, determining that the second quarantine criteria has not been satisfied relative to the second entity.

20. The non-transitory computer-readable storage of claim 18 wherein the quarantine criteria for the first code path is satisfied by the first entity being in a particular number of operations, performed by the first code path, that experience faults.

21. The non-transitory computer-readable storage of claim 20 wherein the particular number is greater than one.

22. The non-transitory computer-readable storage of claim 18 wherein the steps further comprise:

receiving a third request to perform a third operation that may be performed by any one of the plurality of code paths within the server;

selecting the second code path to perform the third operation;

detecting occurrence of a fault during performance of the third operation by the second code path;

identifying a second entity involved in the third operation that experienced the fault;

in response to the second entity being involved in the third operation that experienced the fault, determining that quarantine criteria for the second code path has been satisfied relative to the second entity;

in response to determining that quarantine criteria for the second code path has been satisfied relative to the second entity, storing data that indicates that the second entity is quarantined relative to the second code path.

23. The non-transitory computer-readable storage of claim 22 wherein the steps further comprise:

after storing data that indicates that the second entity is quarantined relative to the second code path, receiving a fourth request to perform a fourth operation that can be performed by any one of the plurality of code paths;

in response to the fourth request, determining whether the second entity is involved in the fourth operation;

in response to determining that the second entity is involved in the fourth operation and that the second entity is quarantined relative to the second code path, selecting a code path, of the plurality of code paths, other than the second code path, to perform the fourth operation.

24. The non-transitory computer-readable storage of claim 23 wherein selecting a code path, of the plurality of code paths, other than the second code path, to perform the fourth operation includes selecting the first code path to perform the fourth operation.

25. The non-transitory computer-readable storage of claim 18 wherein the first and second code paths are code paths within a storage server that services I/O requests from database servers.

26. The non-transitory computer-readable storage of claim 25 wherein the first entity is a disk location.

27. A non-transitory computer-readable storage that stores instructions which, when executed by one or more processors, causes the one or more processors to perform steps comprising:

receiving a first request to perform a first operation that may be performed by any one of a plurality of code paths within a server;

selecting a first code path, of the plurality of code paths, to perform the first operation;

detecting occurrence of a fault during performance of the first operation by the first code path;

identifying a first entity involved in the first operation that experienced the fault;

in response to the first entity being involved in the first operation that experienced the fault, determining that quarantine criteria for the first code path has been satisfied relative to the first entity;

in response to determining that quarantine criteria for the first code path has been satisfied relative to the first entity, storing data that indicates that the first entity is quarantined relative to the first code path;

after storing data that indicates that the first entity is quarantined relative to the first code path, receiving a second request to perform a second operation that can be performed by any one of the plurality of code paths;

in response to the second request, determining whether the first entity is involved in the second operation;

in response to determining that the first entity is involved in the second operation and that the first entity is quarantined relative to the first code path, selecting a second code path of the plurality of code paths to perform the second operation;

wherein the second code path is different from the first code path;

wherein the first entity has a hierarchical relationship relative to a plurality of other entities; and

wherein the quarantine criteria is satisfied relative to the first entity in response to a particular number of said other entities being quarantined relative to the first code path.

28. The non-transitory computer-readable storage of claim 27 wherein the first entity is one of:

a SQL plan step, a SQL plan, or a database server.

29. A storage server comprising:

storage for storing data that belongs to databases managed by one or more database servers;

a first code path for handling I/O requests from the one or more database servers;

a second code path for handling I/O requests from the one or more database servers;

a code path selector configured to determine whether entities involved in I/O requests are quarantined from the first code path and, when an entity involved in a particular I/O request is quarantined from the first code path, causing the particular I/O request to be handled by the second code path;

wherein the entity is a first type of entity, wherein the entity is one of a SQL plan step, a SQL plan, a SQL statement, or a storage location targeted by the particular I/O request;

a fault handler configured to store information about which entities are involved in faults when faults occur while the first code path is being used to service I/O requests; and

a quarantine handler configured to determine, based on which entities are involved in faults, whether quarantine criteria is satisfied for those entities, and to store data that indicates that the entities are quarantined in response to determining that the quarantine criteria is satisfied while continuing to allow the first code path to be executed for operations involving other non-quarantined entities of the first type of entity.