COLLECTING AND AGGREGATING LOG DATA WITH FAULT TOLERANCE

• US 20110246826A1
• Filed: 09/08/2010
• Published: 10/06/2011
• Est. Priority Date: 03/31/2010
• Status: Active Grant

• Current Assignee: Cloudera Incorporated
• Original Assignee: Cloudera Incorporated
• Inventors: Hsieh, Jonathan Ming-Cyn, Robinson, Henry Noel
• Granted Patent: US 9,081,888 B2
• Time in Patent Office: Days
• US Class Current: 714/20
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 2201/875 Monitoring of systems including the internet
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 67/125 involving control of end-device applications over a network

Abstract

Systems and methods of collecting and aggregating log data with fault tolerance are disclosed. One embodiment includes, one or more devices that generate log data, the one or more machines each associated with an agent node to collect the log data, wherein, the agent node generates a batch comprising multiple messages from the log data and assigns a tag to the batch. In one embodiment, the agent node further computes a checksum for the batch of multiple messages. The system may further include a collector device, the collector device being associated with a collector tier having a collector node to which the agent sends the log data; wherein, the collector determines the checksum for the batch of multiple messages received from the agent node.

Claims

1. A system for collecting and aggregating log data with fault tolerance, the method, comprising:

one or more devices that generate log data, the one or more machines each associated with an agent node to collect the log data;

wherein, the agent node generates a batch comprising multiple messages from the log data and assigns a tag to the batch;

the agent node further computes a checksum for the batch of multiple messages;

a collector device, the collector device being associated with a collector tier having a collector node to which the agent sends the log data;

wherein, the collector determines the checksum for the batch of multiple messages received from the agent node.

2. The system of claim 1, wherein, the agent node, in assigning the tag, associates a begin event tag and end event tag to the batch.

3. The system of claim 1, wherein, the agent node writes the tag and the checksum to a write-ahead-log (WAL) in persistent storage of the one or more machines.

4. The system of claim 1, wherein, the collector device is distinct from the one or more devices.

5. The system of claim 1, wherein, the collector device is one of the one or more devices.

6. The system of claim 1,

wherein, in response to verifying the checksum of the batch, the collector adds the tag for the batch to a map;

wherein, the map is associated with multiple tags for multiple batches.

7. The system of claim 1, further comprising,

a storage system coupled to the collector machine to which the collector stores the log data, the storage system having a file system;

wherein, the collector, when a file has been written on file system, stores, in the map, tags associated with batches in the file that has been written.

8. The system of claim 1, further comprising, a master in communication with the one or more machines and the collector machine.

9. The system of claim 1, wherein, the master, using the tags that are associated with batches in the map, acknowledges the agent node and the collector node of the batches that have been safely stored in the file system.

10. The system of claim 8, wherein, the master is executed on another machine that is distinct from the one or more machines and the collector machine.

11. The system of claim 8, wherein, the master is executed on the one or more machines.

12. A method for collecting and aggregating datasets for storage in a file system with fault tolerance, the method, comprising:

collecting datasets from a data source on a machine where the datasets are generated;

generating, a batch comprising multiple messages from the datasets;

assigning a tag to the batch and computing a checksum for the batch;

writing the tag, the batch of multiple messages, and the checksum to an entry in a write-ahead-log (WAL) in storage;

sending the datasets to a receiving location;

in response to verifying the checksum of the batch of multiple messages at the receiving location, adding the tag to a map;

writing a file to destination location;

identifying, in the file, tags associated with the batches in the file that have been written to the destination location.

13. The method of claim 12, wherein, the storage for the WAL is on the machine where the datasets are generated.

14. The method of claim 12, wherein, the map is associated with multiple tags assigned to multiple batches of messages from the datasets.

15. The method of claim 12, wherein, the destination location is a storage tier comprising a distributed file system.

16. The method of claim 12, further comprising, querying for an acknowledgement message that the batch has been written to the destination location.

17. The method of claim 12, further comprising, in response to determining that the batch has been written in the destination location, deleting the entry in the storage of the machine.

18. The method of claim 12, further comprising, in response to determining that the batch has not been written to a file in the destination location, resending the batch.

19. A method for collecting and aggregating datasets with fault tolerance using a store on failure mechanism, the method, comprising:

collecting a dataset from a data source on a machine where the dataset is generated;

wherein, the dataset is collected by an agent node executed on the machine;

sending the dataset to receiving location which aggregates the dataset;

in response to determining that receiving location which is mapped to receive the dataset has failed, storing, by the agent node, the dataset in persistent storage of the machine until the receiving location has been repaired or until another destination is identified.

20. The method of claim 19, wherein, the receiving location is a collector node in a collector tier on the machine where the dataset is generated or another machine.

21. The method of claim 19, wherein, the receiving location is a file system in a storage tier comprised in one or more storage devices.