Monitoring node usage in a distributed system

• US 10,671,474 B2
• Filed: 05/02/2018
• Issued: 06/02/2020
• Est. Priority Date: 11/03/2015
• Status: Active Grant

• Current Assignee: Advanced New Technologies Company Limited (Ant Group Co., Ltd.)
• Original Assignee: Alibaba Group Holding Ltd.
• Inventors: Xia, Yucai, Chang, Erpeng, Wang, Jie
• Primary Examiner(s): Patel, Kamini B
• Application Number:
  US15/969,364
• Publication Number:
  US 20180253350A1
• Time in Patent Office: 762 Days
• Field of Search: 714 471
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: H04L 41/0654 using network fault recovery ring fault isolation or reconfiguration in loop networks without recovery actions by a network management system H04L12/437
• CPC Class Codes: H04L 41/5016 based on statistics of service availability, e.g. in percentage or over a given time
• CPC Class Codes: H04L 41/5054 Automatic deployment of services triggered by the service manager, e.g. service implementation by automatic configuration of network components

Abstract

One or more service call chains in a distributed service system are identified, each service call chain including one or more service nodes that are sequentially called by clients of the distributed service system. A service call model corresponding to a particular service call chain is generated from the one or more service call chains. Calls to the particular service call chain by the clients of the distributed service system are identified. Performance statistics in the service call model are updated for the particular service call chain, wherein the performance statistics is updated based on the identified calls to the particular service call chain. A performance problem associated with the particular service call chain is identified based on the updated performance statistics. In response to identifying the performance problem associated with the particular service call chain, corrective action for the identified performance problem is initiated.

Claims

1. A computer-implemented method, comprising:

identifying one or more service call chains in a distributed service system, each service call chain including one or more service nodes that are sequentially called by clients of the distributed service system;

performing aggregation processing on the one or more service call chains with reference to call feedback information of the one or more service nodes in the one or more service call chains;

generating a service call model corresponding to a particular service call chain from the one or more service call chains, wherein the service call model comprises one or more service call topologies, and wherein each service call topology is acquired by performing aggregation processing on the one or more service call chains;

identifying calls to the particular service call chain by the clients of the distributed service system;

updating performance statistics in the service call model for the particular service call chain, wherein the performance statistics are updated based on the identified calls to the particular service call chain;

comparing, based on a predetermined duration comparison rule, each statistical result with corresponding call feedback information of each service node in the particular service call chain corresponding to the service call topology;

when a difference exceeds a predetermined fluctuation threshold, generating alert information about the particular service call chain corresponding to the difference, wherein the alert information identifies a service node corresponding to the difference;

identifying, based on the alert information, a performance problem associated with the particular service call chain; and

in response to identifying the performance problem associated with the particular service call chain, initiating corrective action for the identified performance problem.

2. The computer-implemented method of claim 1, wherein identifying the one or more service call chains comprises:

acquiring service call log information from the distributed service system; and

extracting the one or more service call chains from the service call log information, wherein each service call chain comprises the one or more service nodes that are sequentially called.

3. The computer-implemented method of claim 1, wherein generating the service call model comprises generating the service call model by performing aggregation processing on the service call chain, wherein the service call model comprises one or more service call topologies, wherein each service call topology is acquired by performing aggregation processing on one or more additional service call chains different than the particular service call chain;

and

wherein processing the service call chain comprises processing the service call chain according to a service call topology.

4. The computer-implemented method of claim 3, further comprising:

performing a data cleaning operation on the service call topology, wherein updating performance statistics in the service call model for the particular service call chain comprises updating performance statistics in the service call model for the particular service call chain according to a cleaned service call topology.

5. The computer-implemented method of claim 4, wherein the data cleaning operation comprises at least one of:

deleting a predetermined service node in the service call topology;

deleting a service node that is in the service call topology and that does not return call result information;

deleting an occasionally-used service node in the service call topology, wherein a number of accumulated appearance times of the occasionally-used service node in the service call chain corresponding to the service call topology is less than or equal to a predetermined occasionally-used threshold;

or

deleting the occasionally-used service node in the service call topology, wherein the number of accumulated appearance times of the occasionally-used service node in the service call chain corresponding to the service call topology is less than or equal to the predetermined occasionally-used threshold, and wherein a number of accumulated appearance times of another service node different from the occasionally-used service node is greater than or equal to a predetermined frequently used threshold.

6. A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform operations comprising:

identifying one or more service call chains in a distributed service system, each service call chain including one or more service nodes that are sequentially called by clients of the distributed service system;

performing aggregation processing on the one or more service call chains with reference to call feedback information of the one or more service nodes in the one or more service call chains;

generating a service call model corresponding to a particular service call chain from the one or more service call chains, wherein the service call model comprises one or more service call topologies, and wherein each service call topology is acquired by performing aggregation processing on the one or more service call chains;

identifying calls to the particular service call chain by the clients of the distributed service system;

updating performance statistics in the service call model for the particular service call chain, wherein the performance statistics are updated based on the identified calls to the particular service call chain;

comparing, based on a predetermined duration comparison rule, each statistical result with corresponding call feedback information of each service node in the particular service call chain corresponding to the service call topology;

when a difference exceeds a predetermined fluctuation threshold, generating alert information about the particular service call chain corresponding to the difference, wherein the alert information identifies a service node corresponding to the difference;

identifying, based on the alert information, a performance problem associated with the particular service call chain; and

in response to identifying the performance problem associated with the particular service call chain, initiating corrective action for the identified performance problem.

7. The non-transitory, computer-readable medium of claim 6, wherein identifying the one or more service call chains comprises:

acquiring service call log information from the distributed service system; and

extracting the one or more service call chains from the service call log information, wherein each service call chain comprises the one or more service nodes that are sequentially called.

8. The non-transitory, computer-readable medium of claim 6, wherein generating the service call model comprises generating the service call model by performing aggregation processing on the service call chain, wherein the service call model comprises one or more service call topologies, wherein each service call topology is acquired by performing aggregation processing on one or more additional service call chains different than the particular service call chain;

and

wherein processing the service call chain comprises processing the service call chain according to a service call topology.

9. The non-transitory, computer-readable medium of claim 8, the operations further comprising:

performing a data cleaning operation on the service call topology, wherein updating performance statistics in the service call model for the particular service call chain comprises updating performance statistics in the service call model for the particular service call chain according to a cleaned service call topology.

10. The non-transitory, computer-readable medium of claim 9, wherein the data cleaning operation comprises at least one of:

deleting a predetermined service node in the service call topology;

deleting a service node that is in the service call topology and that does not return call result information;

deleting an occasionally-used service node in the service call topology, wherein a number of accumulated appearance times of the occasionally-used service node in the service call chain corresponding to the service call topology is less than or equal to a predetermined occasionally-used threshold;

or

deleting the occasionally-used service node in the service call topology, wherein the number of accumulated appearance times of the occasionally-used service node in the service call chain corresponding to the service call topology is less than or equal to the predetermined occasionally-used threshold, and wherein a number of accumulated appearance times of another service node different from the occasionally-used service node is greater than or equal to a predetermined frequently used threshold.

11. A computer-implemented system, comprising:

one or more computers; and

one or more computer memory devices interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers, perform one or more operations comprising;

identifying one or more service call chains in a distributed service system, each service call chain including one or more service nodes that are sequentially called by clients of the distributed service system;

performing aggregation processing on the one or more service call chains with reference to call feedback information of the one or more service nodes in the one or more service call chains;

generating a service call model corresponding to a particular service call chain from the one or more service call chains, wherein the service call model comprises one or more service call topologies, and wherein each service call topology is acquired by performing aggregation processing on the one or more service call chains;

identifying calls to the particular service call chain by the clients of the distributed service system;

updating performance statistics in the service call model for the particular service call chain, wherein the performance statistics are updated based on the identified calls to the particular service call chain;

comparing, based on a predetermined duration comparison rule, each statistical result with corresponding call feedback information of each service node in the particular service call chain corresponding to the service call topology;

when a difference exceeds a predetermined fluctuation threshold, generating alert information about the particular service call chain corresponding to the difference, wherein the alert information identifies a service node corresponding to the difference;

identifying, based on the alert information, a performance problem associated with the particular service call chain; and

in response to identifying the performance problem associated with the particular service call chain, initiating corrective action for the identified performance problem.

12. The computer-implemented system of claim 11, wherein identifying the one or more service call chains comprises:

acquiring service call log information from the distributed service system; and

extracting the one or more service call chains from the service call log information, wherein each service call chain comprises the one or more service nodes that are sequentially called.

13. The computer-implemented system of claim 11, wherein generating the service call model comprises generating the service call model by performing aggregation processing on the service call chain, wherein the service call model comprises one or more service call topologies, wherein each service call topology is acquired by performing aggregation processing on one or more additional service call chains different than the particular service call chain;

and

wherein processing the service call chain comprises processing the service call chain according to a service call topology.

14. The computer-implemented system of claim 13, the operations further comprising:

performing a data cleaning operation on the service call topology, wherein updating performance statistics in the service call model for the particular service call chain comprises updating performance statistics in the service call model for the particular service call chain according to a cleaned service call topology.

15. The computer-implemented system of claim 14, wherein the data cleaning operation comprises at least one of:

deleting a predetermined service node in the service call topology;

deleting a service node that is in the service call topology and that does not return call result information;

deleting an occasionally-used service node in the service call topology, wherein a number of accumulated appearance times of the occasionally-used service node in the service call chain corresponding to the service call topology is less than or equal to a predetermined occasionally-used threshold;

or

deleting the occasionally-used service node in the service call topology, wherein the number of accumulated appearance times of the occasionally-used service node in the service call chain corresponding to the service call topology is less than or equal to the predetermined occasionally-used threshold, and wherein a number of accumulated appearance times of another service node different from the occasionally-used service node is greater than or equal to a predetermined frequently used threshold.