Automatic targeted system suspension based upon downstream system failure detection

• US 10,157,092 B2
• Filed: 04/14/2016
• Issued: 12/18/2018
• Est. Priority Date: 04/27/2015
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Sengupta, Aninda, Wolfangel, Steven Gene, Le Mouel, Philippe, Fuglsang, Dennis, Tang, Lixin, Weng, Sherry, Stern, Bo
• Primary Examiner(s): Mehrmanesh, Elmira
• Application Number:
  US15/099,323
• Publication Number:
  US 20160314032A1
• Time in Patent Office: 978 Days
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/076 by exceeding a count or rate limit, e.g. word- or bit count limit
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 2201/88 Monitoring involving counting

Abstract

Techniques described herein relate to automatic system suspension based upon downstream system failure in service-oriented architecture (SOA) applications. A system management module of a SOA system may be configured to detect a problem with a downstream service provider involved with a SOA application, and in response, automatically suspend particular processing within the application to prevent requests that may end up at the downstream service provider from entering the SOA application. The system management module may implement a circuit breaker module that maintains a circuit breaker structure specific to a downstream endpoint leading to a downstream service provider. Upon a triggering of a circuit breaker for a downstream endpoint, one or more upstream service providers are identified as associated with faulted traffic causing the triggering, and the system management module can automatically cause further traffic from these upstream service providers to be suspended.

Claims

1. A system, comprising:

one or more application servers, each application server configured to;

execute a plurality of composites of an application;

receive and process service requests from one or more upstream service providers; and

determine and initiate interactions with one or more downstream service providers, in response to the service requests received from the upstream service providers; and

a computing device configured to execute a system management module, the computing device comprising;

a processing unit comprising one or more processors;

one or more network interfaces communicatively coupled with the one or more processors; and

memory coupled with and readable by the processing unit and storing therein a set of instructions which, when executed by the processing unit, causes the computing device to;

receive data identifying a plurality of service requests received by the application from one or more upstream service providers;

receive data identifying a plurality of interactions initiated by the application and directed to one or more downstream service providers, wherein the plurality of interactions are initiated by the application during processing of the service requests received from the upstream service providers;

detect one or more faults within the plurality of interactions initiated by the application and directed to a first downstream service provider;

identify first circuit breaker conditions associated with the first downstream service provider;

determine that the one or more faults detected within the plurality of interactions initiated by the application and directed to the first downstream service provider, satisfy the first circuit breaker conditions;

identify at least one of the upstream service providers associated with the one or more faults; and

cause the application to suspend processing of requests from the identified upstream service providers, wherein causing the application to suspend processing of requests comprises;

(i) identifying one or more components of the application; and

(ii) causing the one or more components to stop accepting requests from the identified upstream service providers, said causing comprising at least one of;

(a) causing the one or more components to stop retrieving requests from an incoming queue associated with the identified upstream service providers, or (b) causing the one or more components to, in response to a receipt of a request message from an identified upstream service provider, transmit a response message indicating a denial of the request.

2. The system of claim 1, wherein the plurality of service requests received by the application are received from a plurality of upstream service providers, and wherein causing the application to suspend processing of requests comprises:

causing the application to suspend processing of requests from a first subset of the plurality of upstream service providers; and

allowing the application to continue processing requests from a second subset of the plurality of upstream service providers.

3. The system of claim 1, wherein causing the application to suspend processing of requests comprises:

identifying one or more components of the application; and

causing the one or more components to stop accepting requests from the identified upstream service providers; and

wherein the identifying the one or more components of the application comprises;

identifying, for at least one of the identified upstream service providers, a component serving as an initial processing component within the application for the upstream service provider.

4. The system of claim 3, wherein the identifying the one or more components of the application comprises:

identifying, for at least one of the identified upstream service providers, a first component after a last queue within the application preceding a point of exit to the downstream service provider.

5. The system of claim 1, wherein determining that the first circuit breaker conditions are satisfied comprises determining whether a predetermined number of faults detected within the plurality of interactions initiated by the application and directed to the first downstream service provider have occurred within a predetermined amount of time.

6. The system of claim 1, wherein the memory of the computing device comprises further instructions which, when executed by the processing unit, causes the computing device to:

detecting one or more additional faults within the plurality of interactions initiated by the application and directed to a second downstream service provider; and

identifying, by the system management module, second circuit breaker conditions associated with the second downstream service provider, wherein the first circuit breaker conditions are different from the second circuit breaker conditions.

7. The system of claim 1, wherein the application comprises a service-oriented architecture (SOA) application including one or more components, and wherein the memory of the computing device comprises further instructions which, when executed by the processing unit, causes the computing device to:

receive data from the one or more components of the SOA application, the received data including the data identifying the plurality of service requests received by the application, and the data identifying the plurality of interactions initiated by the application.

8. The system of claim 1, wherein the application comprises a service-oriented architecture (SOA) application including one or more components, and wherein the memory of the computing device comprises further instructions which, when executed by the processing unit, causes the computing device to:

receive data from an infrastructure system service engine controlling the execution of the SOA application, the received data including the data identifying the plurality of service requests received by the application, and the data identifying the plurality of interactions initiated by the application.

9. A method, comprising:

receiving, by a system management module executing on a computing device, data identifying a plurality of service requests received by an application from one or more upstream service providers;

receiving, by the system management module, data identifying a plurality of interactions initiated by the application and directed to one or more downstream service providers, wherein the plurality of interactions are initiated by the application during processing of the service requests received from the upstream service providers;

detecting, by the system management module, one or more faults within the plurality of interactions initiated by the application and directed to a first downstream service provider;

identifying, by the system management module, first circuit breaker conditions associated with the first downstream service provider;

determining, by the system management module, that the one or more faults detected within the plurality of interactions initiated by the application and directed to the first downstream service provider, satisfy the first circuit breaker conditions;

identifying, by the system management module, at least one of the upstream service providers associated with the one or more faults; and

causing, by the system management module, the application to suspend processing of requests from the identified upstream service providers, wherein causing the application to suspend processing of requests comprises;

(i) identifying one or more components of the application; and

(ii) causing the one or more components to stop accepting requests from the identified upstream service providers, said causing comprising at least one of;

(a) causing the one or more components to stop retrieving requests from an incoming queue associated with the identified upstream service providers, or (b) causing the one or more components to, in response to a receipt of a request message from an identified upstream service provider, transmit a response message indicating a denial of the request.

10. The method of claim 9, wherein the plurality of service requests received by the application are received from a plurality of upstream service providers, and wherein causing the application to suspend processing of requests comprises:

causing the application to suspend processing of requests from a first subset of the plurality of upstream service providers; and

allowing the application to continue processing requests from a second subset of the plurality of upstream service providers.

11. The method of claim 9, wherein causing the application to suspend processing of requests comprises:

identifying one or more components of the application; and

causing the one or more components to stop accepting requests from the identified upstream service providers; and

wherein the identifying the one or more components of the application comprises;

identifying, for at least one of the identified upstream service providers, a component serving as an initial processing component within the application for the upstream service provider.

12. The method of claim 11, wherein the identifying the one or more components of the application comprises:

identifying, for at least one of the identified upstream service providers, a first component after a last queue within the application preceding a point of exit to the downstream service provider.

13. The method of claim 9, wherein determining that the first circuit breaker conditions are satisfied comprises determining whether a predetermined number of faults detected within the plurality of interactions initiated by the application and directed to the first downstream service provider have occurred within a predetermined amount of time.

14. The method of claim 9, further comprising:

detecting one or more additional faults within the plurality of interactions initiated by the application and directed to a second downstream service provider; and

identifying, by the system management module, second circuit breaker conditions associated with the second downstream service provider, wherein the first circuit breaker conditions are different from the second circuit breaker conditions.

15. The method of claim 9, wherein the application comprises a service-oriented architecture (SOA) application including one or more components, the method further comprising:

receiving data from the one or more components of the SOA application, the received data including the data identifying the plurality of service requests received by the application, and the data identifying the plurality of interactions initiated by the application.

16. The method of claim 9, wherein the application comprises a service-oriented architecture (SOA) application including one or more components, the method further comprising:

receiving data from an infrastructure system service engine controlling the execution of the SOA application, the received data including the data identifying the plurality of service requests received by the application, and the data identifying the plurality of interactions initiated by the application.

17. A method, comprising:

receiving, by a system management module executing on a computing device, data identifying a plurality of service requests received by an application from one or more upstream service providers;

receiving, by the system management module, data identifying a plurality of interactions initiated by the application and directed to one or more downstream service providers, wherein the plurality of interactions are initiated by the application during processing of the service requests received from the upstream service providers;

detecting, by the system management module, one or more faults within the plurality of interactions initiated by the application and directed to a first downstream service provider;

identifying, by the system management module, first circuit breaker conditions associated with the first downstream service provider;

determining, by the system management module, that the one or more faults detected within the plurality of interactions initiated by the application and directed to the first downstream service provider, satisfy the first circuit breaker conditions;

identifying, by the system management module, at least one of the upstream service providers associated with the one or more faults; and

causing, by the system management module, the application to suspend processing of requests from the identified upstream service providers, wherein causing the application to suspend processing of requests comprises;

identifying one or more components of the application; and

causing the one or more components to stop accepting requests from the identified upstream service providers; and

wherein the identifying the one or more components of the application comprises identifying, for at least one of the identified upstream service providers, (i) a first component serving as an initial processing component within the application for the upstream service provider, and (ii) a second component after a last queue within the application preceding a point of exit to the downstream service provider.

18. The method of claim 17, wherein the plurality of service requests received by the application are received from a plurality of upstream service providers, and wherein causing the application to suspend processing of requests comprises:

causing the application to suspend processing of requests from a first subset of the plurality of upstream service providers; and

allowing the application to continue processing requests from a second subset of the plurality of upstream service providers.

19. The method of claim 17, wherein determining that the first circuit breaker conditions are satisfied comprises determining whether a predetermined number of faults detected within the plurality of interactions initiated by the application and directed to the first downstream service provider have occurred within a predetermined amount of time.

20. The method of claim 17, further comprising:

detecting one or more additional faults within the plurality of interactions initiated by the application and directed to a second downstream service provider; and

identifying, by the system management module, second circuit breaker conditions associated with the second downstream service provider, wherein the first circuit breaker conditions are different from the second circuit breaker conditions.