CONFIGURING A SYSTEM TO COLLECT AND AGGREGATE DATASETS

• US 20110246816A1
• Filed: 09/08/2010
• Published: 10/06/2011
• Est. Priority Date: 03/31/2010
• Status: Active Grant

• Current Assignee: Cloudera Incorporated
• Original Assignee: Cloudera Incorporated
• Inventors: Hsieh, Jonathan Ming-Cyn, Robinson, Henry Noel
• Granted Patent: US 9,317,572 B2
• Time in Patent Office: Days
• US Class Current: 714/4.12
• CPC Class Codes: G06F 11/2023 Failover techniques
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: H04L 41/12 Discovery or management of network topologies
• CPC Class Codes: H04L 67/1042 using topology management mechanisms
• CPC Class Codes: H04L 67/1087 using cross-functional networking aspects

Abstract

Methods for configuring a system to collect and aggregate datasets are disclosed. One embodiment includes, identifying a data source in the system from where dataset is to be collected, configuring a machine in the system that generates the dataset to be collected, to send the dataset to the data source, identifying an arrival location where the dataset that is collected is to be aggregated or written, and/or configuring an agent node by specifying a source for the agent node as the data source in the system and specifying a sink for the agent node as the arrival location.

Claims

1. A method for configuring a system to collect and aggregate datasets, the method, comprising:

identifying a data source in the system from where dataset is to be collected;

configuring a machine in the system that generates the dataset to be collected, to send the dataset to the data source;

identifying an arrival location where the dataset that is collected is to be aggregated or written;

configuring an agent node by specifying a source for the agent node as the data source in the system and specifying a sink for the agent node as the arrival location.

2. The method of claim 1, wherein, the system includes multiple machines which generate datasets that are to be collected.

3. The method of claim 1, further comprising:

specifying the arrival location as a collector source of a collector node;

specifying a distributed file system as a collector sink of the collector node;

wherein, the distributed file system is the Hadoop distributed file system (HDFS)

4. The method of claim 1, wherein, the agent node is configurable to perform analytics on the dataset.

5. The method of claim 1, further comprising, launching a web page on a machine to access a master for the system using the master'"'"'s universal resource locator (URL).

6. The method of claim 5, further comprising, scaling the system by implementing multiple masters on multiple machines.

7. The method of claim 6, further comprising, maintaining consistency among the multiple masters using a consensus protocol for order critical state changes.

8. The method of claim 7, wherein, the order critical state changes include data flow configurations and node-to-machine mappings.

9. The method of claim 6, further comprising, maintaining consistency among the multiple masters using a gossip protocol.

10. The method of claim 9, wherein, the gossip protocol is used to maintain weak consistency among data including but not limited to node status updates, reporting information, and metrics information.

11. The method of claim 5, wherein, configuration of the agent node is performed by accessing a link on the web page for the master.

12. The method of claim 1, wherein, the data source includes, one or more of, a syslog port, an incoming network connection, and an IRC channel.

13. The method of claim 1, wherein, the data source includes, a static file source.

14. The method of claim 1, wherein, the data source includes, one or more of, a status update and a feed.

15. The method of claim 1, wherein, the data source includes, output from execution of a program.

16. The method of claim 1, wherein, the data sink includes a network connection.

17. The method of claim 1, wherein, the data sink includes a physical disk.

18. The method of claim 1, wherein, the data sink includes a console.

19. A method for configuring a system having multiple machines to collect datasets from the multiple machines and to perform analytics on the datasets, the method, comprising:

identifying data sources on the multiple machines wherein datasets are to be collected from;

configuring the multiple machines in the system that generate the datasets are to be collected, to send the datasets to the data source;

identifying an arrival location where dataset that is collected is to be logged;

specifying configurations for the multiple machines simultaneously by accessing a master through a web page and specifying the data sources for agent nodes;

wherein, each agent node is associated with each of the multiple machines;

specifying a sink for each of the agent node as the arrival location;

specifying the arrival location as a collector source of a collector node;

specifying a distributed file system as a collector sink of the collector node;

wherein, the agent node or the collector node performs analytics on the datasets.

20. The method of claim 19, wherein, the master is used to configure and dynamically reconfigure the agent nodes.

21. The method of claim 19, wherein, statuses of the agent nodes are visible via the master.

22. The method of claim 19, wherein, the distributed file system is the Hadoop distributed file system (HDFS).

23. The method of claim 19,

further comprising, launching a web page on a machine to access the master using the master'"'"'s universal resource locator (URL);

wherein, configuration of the agent nodes for the multiple machines is performed by accessing a link on the web page for the master.

24. The method of claim 19, further comprising, configuring the sink to be a fan-out sink that sends incoming datasets to child sinks coupled to the sink for data replication or for splitting off a CPU intensive tap.

25. The method of claim 19, further comprising, configuring the sink as a failover sink that specifies alternate collectors in case the collector fails.

26. The method of claim 25,

wherein, the sink is a failover sink that specifies a local disk to store data until the sink that has failed recovers.