QUORUM BASED DISTRIBUTED ANOMALY DETECTION AND REPAIR

• US 20170031743A1
• Filed: 07/31/2015
• Published: 02/02/2017
• Est. Priority Date: 07/31/2015
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: AppDynamics, Inc. (Cisco Systems, Inc.)
• Inventors: Borah, Gautam, Gupta, Pankaj
• Granted Patent: US 9,886,337 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0721 within a central processing unit [CPU]
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 2209/5013 Request control
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals

Abstract

Quorum based anomaly detection utilizes multiple entities to detect and attempt to configure a repair task for an anomaly. Once the repair task is generated, a system is used to assign the task to a worker entity while recording the responsibility of that task with the worker in a persistent storage. If the worker entity crashes, the degraded worker status will eventually be detected, and all tasks associated with that worker will be re-assigned. Once a worker finishes a task, the assignment information for the task is transitioned to a completed state.

Claims

1. A method for processing metrics, comprising:

providing a group of collectors and a group of aggregators on one or more servers, the group of collectors receiving metric data from a plurality of agents one or more remote servers and each of the aggregators receiving and aggregating selected metrics received from one or more collectors;

detecting an anomaly among the collectors and aggregators;

creating a task based on the anomaly by a producer of a plurality of producers that attempt to create the task; and

assigning the task to a worker.

2. The method of claim 1, wherein the status of the task is maintained in a persistent scheduler.

3. The method of claim 2, further comprising:

registering the worker by adding a worker identifier to a worker context within the persistent scheduler; and

adding the worker identifier to an assignment context, the task assigned to the worker in the assignment context.

4. The method of claim 3, further comprising:

detecting that the worker is unavailable;

removing the worker identifier from the worker context; and

maintaining the worker identifier and any task identifier associated with the worker identifier in the assignment context.

5. The method of claim 1, wherein one of the collectors implement each of the plurality of producers.

6. The method of claim 1, wherein an aggregator implements the worker.

7. The method of claim 1, wherein each of the plurality of producers that attempt to create the task are prevented from successfully creating the task after the task is created by the producer that creates the task.

8. A non-transitory computer readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for processing metrics, the method comprising:

providing a group of collectors and a group of aggregators on one or more servers, the group of collectors receiving metric data from a plurality of agents one or more remote servers and each of the aggregators receiving and aggregating selected metrics received from one or more collectors;

detecting an anomaly among the collectors and aggregators;

creating a task based on the anomaly by a producer of a plurality of producers that attempt to create the task; and

assigning the task to a worker.

9. The non-transitory computer readable storage medium of claim 8, wherein the status of the task is maintained in a persistent scheduler.

10. The non-transitory computer readable storage medium of claim 9, further comprising:

registering the worker by adding a worker identifier to a worker context within the persistent scheduler; and

adding the worker identifier to an assignment context, the task assigned to the worker in the assignment context.

11. The non-transitory computer readable storage medium of claim 10, further comprising:

detecting that the worker is unavailable;

removing the worker identifier from the worker context; and

maintaining the worker identifier and any task identifier associated with the worker identifier in the assignment context.

12. The non-transitory computer readable storage medium of claim 8, wherein one of the collectors implement each of the plurality of producers.

13. The non-transitory computer readable storage medium of claim 8, wherein an aggregator implements the worker.

14. The non-transitory computer readable storage medium of claim 8, wherein each of the plurality of producers that attempt to create the task are prevented from successfully creating the task after the task is created by the producer that creates the task.

15. A system for processing metrics, comprising:

a processor;

a memory; and

one or more modules stored in memory and executable by a processor to provide a group of collectors and a group of aggregators on one or more servers, the group of collectors receiving metric data from a plurality of agents one or more remote servers and each of the aggregators receiving and aggregating selected metrics received from one or more collectors, detect an anomaly among the collectors and aggregators, create a task based on the anomaly by a producer of a plurality of producers that attempt to create the task, and assign the task to a worker.

16. The system of claim 15, wherein the status of the task is maintained in a persistent scheduler.

17. The system of claim 16, the one or more modules further executable to register the worker by adding a worker identifier to a worker context within the persistent scheduler, and add the worker identifier to an assignment context, the task assigned to the worker in the assignment context.

18. The system of claim 17, the one or more modules further executable to detect that the worker is unavailable, remove the worker identifier from the worker context, and maintain the worker identifier and any task identifier associated with the worker identifier in the assignment context.

19. The system of claim 15, wherein one of the collectors implement each of the plurality of producers.

20. The system of claim 15, wherein an aggregator implements the worker.

21. The system of claim 15, wherein each of the plurality of producers that attempt to create the task are prevented from successfully creating the task after the task is created by the producer that creates the task.