ADAPTIVE CONTROL OF DATA COLLECTION REQUESTS SENT TO EXTERNAL DATA SOURCES

• US 20170085447A1
• Filed: 01/30/2016
• Published: 03/23/2017
• Est. Priority Date: 09/21/2015
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Chen, Ken, Tao, Gang, DING, LAI QIANG, HAO, JAMES, Wang, Ting, Haddad, Elias, BITINCKA, DRITAN
• Granted Patent: US 10,007,710 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: H04L 43/024 by adaptive sampling
• CPC Class Codes: H04L 43/0817 by checking functioning

Abstract

Techniques and mechanisms are disclosed that enable a data collection system to adaptively control collection of data from one or more external data sources. At a high level, adaptively controlling collection of data from external data sources may include collecting performance information related to one or more data collection nodes and, in response to analyzing the collected performance information, adapting rates at which the data collection nodes send data collection requests to external data sources. Data collection performance information generally may include, but is not limited to, network traffic data, error messages generated by external data sources and/or data collection nodes, computing device performance information, and any other types of information related to a data collection node'"'"'s ability to collect data from external data sources.

Claims

1. A method, comprising:

receiving, by a controller node, data collection performance information from a data collection node of a plurality of data collection nodes, the data collection performance information related to data collection requests sent by the data collection node to a cloud computing service provider;

generating, based on the data collection performance information, instructions for the data collection node to modify a rate at which the data collection node sends data collection requests to the cloud computing service provider;

sending the instructions to the data collection node.

2. The method of claim 1, wherein the data collection requests are sent to an application programming interface (API) associated with the cloud computing service provider.

3. The method of claim 1, wherein the instructions instruct the data collection node to increase the rate at which the data collection node sends the data collection requests to the cloud computing service provider.

4. The method of claim 1, wherein the instructions instruct the data collection node to decrease the rate at which the data collection node sends the data collection requests to the cloud computing service provider.

5. The method of claim 1, wherein the data collection performance information includes information indicating one or more error messages received by the data collection node from the cloud computing service provider.

6. The method of claim 1, wherein the data collection performance information includes network traffic data.

7. The method of claim 1, wherein the data collection performance information includes CPU utilization information for the data collection node.

8. The method of claim 1, wherein the data collection performance information includes information indicating one or more attributes of the cloud computing service provider.

9. The method of claim 1, wherein the data collection performance information includes information indicating one or more data collection policies of the cloud computing service provider.

10. The method of claim 1, wherein the controller node receives data collection performance information from each data collection node of the plurality of data collection nodes.

11. The method of claim 1, wherein the controller node receives data collection performance information from each data collection node of the plurality of data collection nodes, and wherein the plurality of data collection nodes sends data collection requests to a plurality of cloud computing service providers.

12. The method of claim 1, wherein the data collection node sends data collection requests to a plurality of cloud computing service providers.

13. The method of claim 1, further comprising:

subsequent to generating the instructions, receiving additional data collection performance information from the data collection node;

generating, based at least in part on the additional data collection performance information, updated instructions for the data collection node to modify the rate at which the data collection node sends data collection requests to the cloud computing service provider;

sending the updated instructions to the data collection node.

14. The method of claim 1, wherein the data collection node sends data collection requests to a plurality of cloud computing service providers, and wherein the controller node generates and sends to the data collection node separate instructions for each different cloud computing service provider of the plurality of cloud computing service providers.

15. The method of claim 1, further comprising:

generating, based on the data collection performance information, second instructions for a different data collection node of the plurality of data collection nodes to send data collection requests to the cloud computing service provider;

sending the second instructions to the different data collection node.

16. The method of claim 1, wherein the data collection requests sent to the cloud computing service provider request data related to performance and/or security information for one or more computing resources managed by the cloud computing service provider.

17. The method of claim 1, wherein data collected from the cloud computing service providers is sent to one or more indexers to be parsed and stored as event data reflecting events of one or more resources of an information technology environment.

18. The method of claim 1, wherein data collected from the cloud computing service providers comprises raw data sent to one or more indexers, and wherein the one or more indexers parse a plurality of timestamped events from the raw data, and wherein each timestamped event of the plurality of timestamped events includes a portion of the raw data.

19. The method of claim 1, further comprising:

receiving, by the data collection node, the instructions;

sending, by the data collection node, data collection requests to the cloud computing service provider at a rate based on the instructions.

20. The method of claim 1, further comprising:

receiving, by the controller node, an indication that new data collection node has joined the plurality of data collection nodes;

modifying instructions for one or more data collection nodes of the plurality based on the indication that the new data collection node has joined the plurality of data collection nodes.

21. The method of claim 1, further comprising:

receiving, by the controller node, an indication that data collection node has left the plurality of data collection nodes;

modifying instructions for one or more data collection nodes of the plurality based on the indication that a data collection node has left the plurality of data collection nodes.

22. The method of claim 1, further comprising:

determining that the data collection performance information indicates that the cloud computing service provider has upgraded its data collection capabilities;

in response to determining that the cloud computing service provider has upgraded its data collection capabilities, instructing the data collection node to increase the rate at which the data collection node sends the data collection requests to the cloud computing service provider.

23. The method of claim 1, further comprising:

wherein the data collection performance information indicates that the cloud computing service provider has upgraded its data collection capabilities based on one or more of;

one or more status messages, one or more version identifiers, an operational pattern of cloud computing service provider resources, and data retrieved from a website separate from the cloud computing service provider;

in response to determining that the data collection performance information includes information indicating an upgrade to the cloud computing service provider, instructing the data collection node to increase the rate at which the data collection node sends the data collection requests to the cloud computing service provider.

24. One or more non-transitory storage media storing instructions which, when executed by one or more computing devices, cause:

receiving, by a controller node, data collection performance information from a data collection node of a plurality of data collection nodes, the data collection performance information related to data collection requests sent by the data collection node to a cloud computing service provider;

generating, based on the data collection performance information, instructions for the data collection node to modify a rate at which the data collection node sends data collection requests to the cloud computing service provider;

sending the instructions to the data collection node.

25. An apparatus, comprising:

a subsystem, implemented at least partially in hardware, that receives, by a controller node, data collection performance information from a data collection node of a plurality of data collection nodes, the data collection performance information related to data collection requests sent by the data collection node to a cloud computing service provider;

a subsystem, implemented at least partially in hardware, that generates, based on the data collection performance information, instructions for the data collection node to modify a rate at which the data collection node sends data collection requests to the cloud computing service provider;

a subsystem, implemented at least partially in hardware, that sends the instructions to the data collection node.