Collecting and aggregating log data with fault tolerance

• US 9,081,888 B2
• Filed: 09/08/2010
• Issued: 07/14/2015
• Est. Priority Date: 03/31/2010
• Status: Active Grant

• Current Assignee: Cloudera Incorporated
• Original Assignee: Cloudera Incorporated
• Inventors: Hsieh, Jonathan Ming-Cyn, Robinson, Henry Noel
• Primary Examiner(s): BUTLER, SARAI E
• Application Number:
  US12/877,913
• Publication Number:
  US 20110246826A1
• Time in Patent Office: 1,770 Days
• Field of Search: 714/49, 714/48
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 2201/875 Monitoring of systems including the internet
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 67/125 involving control of end-device applications over a network

Abstract

Systems and methods of collecting and aggregating log data with fault tolerance are disclosed. One embodiment includes, one or more devices that generate log data, the one or more machines each associated with an agent node to collect the log data, wherein, the agent node generates a batch comprising multiple messages from the log data and assigns a tag to the batch. In one embodiment, the agent node further computes a checksum for the batch of multiple messages. The system may further include a collector device, the collector device being associated with a collector tier having a collector node to which the agent sends the log data; wherein, the collector determines the checksum for the batch of multiple messages received from the agent node.

Claims

1. A system for collecting and aggregating log data with fault tolerance, the system comprising:

one or more devices that generate log data;

one or more machines each having an agent node to collect the log data,

wherein, at least one agent node;

generates a batch comprising multiple messages from the log data,

assigns a tag to the batch, and

computes a checksum for the batch of multiple messages;

a collector device associated with a collector tier and having a collector node,

wherein the at least one agent node sends the batch of multiple messages as a single aggregate event, the checksum and the tag to the collector node, and

wherein, the collector node verifies the checksum for the batch of multiple messages received from the at least one agent node,

at least one master in communication with the one or more machines and the collector device,

wherein, the at least one master, using tags that are assigned to batches, acknowledges that the collector node has written the batches into a file system and informs the at least one agent node that the batches have been safely stored in the file system.

2. The system of claim 1, wherein, the at least one agent node, in assigning the tag, associates a begin event tag and end event tag to the batch.

3. The system of claim 1, wherein, the at least one agent node writes the tag and the checksum to a write-ahead-log (WAL) in persistent storage of the one or more machines.

4. The system of claim 1, wherein, the collector device is distinct from the one or more devices.

5. The system of claim 1, wherein, the collector device is one of the one or more devices.

6. The system of claim 1,

wherein, in response to verifying the checksum of the batch, the collector node adds the tag for the batch to a map,

wherein, the map is associated with multiple tags for multiple batches.

7. The system of claim 1, further comprising,

a storage system coupled to the collector device to which the collector node stores the log data, the storage system having a file system,

wherein, the collector node, when a file has been written on the file system, stores, in a map, tags associated with batches in the file that has been written.

8. The system of claim 1, wherein, the one master is executed on another machine that is distinct from the one or more machines and the collector device.

9. The system of claim 1, wherein, the at least one master is executed on the one or more machines.

10. The method of claim 1, wherein, in response to verifying the checksum of the batch, the collector node adds the tag for the batch to a map.

11. The method of claim 10, wherein, the map is associated with multiple tags for multiple batches.

12. A method of collecting and aggregating log data with fault tolerance, the method comprising:

accessing, by one or more devices, log data;

collecting, by one or more machines each having an agent node, the log data,

wherein, at least one agent node;

generates a batch comprising multiple messages from the log data,

assigns a tag to the batch, and

computes a checksum for the batch of multiple messages;

sending, by the at least one agent node, the batch, the checksum and the tag to a collector device having a collector node,

wherein the batch is sent as a single aggregate event,

wherein the collector device is associated with a collector tier,

wherein the collector node verifies the checksum for the batch of multiple messages received from the at least one agent node,

acknowledging, by at least one master in communication with the one or more machines and the collector device, that the collector node has written the batches into a file system and informing the at least one agent node that the batches have been safely stored in the file system.

13. The method of claim 12, wherein, the at least one agent node, in assigning the tag, further associates a begin event tag and end event tag to the batch.

14. The method of claim 12, wherein, the at least one agent node writes the tag and the checksum to a write-ahead-log (WAL) in persistent storage of the one or more machines.

15. The method of claim 12, wherein, the collector device is distinct from the one or more devices.

16. The method of claim 12, wherein, the collector device is one of the one or more devices.

17. The method of claim 12, wherein the collector node stores the log data to a storage system coupled to the collector device, the storage system having a file system, and wherein the collector node, when a file has been written on the file system, stores, in a map, tags associated with batches in the file that has been written.

18. The method of claim 12, wherein, the one master is executed on another machine that is distinct from the one or more machines and the collector device.

19. The method of claim 12, wherein, the at least one master is executed on the one or more machines.

20. A system for collection and aggregation of log data with fault tolerance, the system comprising:

means for accessing, by one or more devices, log data;

means for collecting, by one or more machines each having an agent node, the log data,

wherein, at least one agent node;

generates a batch comprising multiple messages from the log data,

assigns a tag to the batch, and

computes a checksum for the batch of multiple messages;

means for sending, by the at least one agent node, the batch, the checksum and the tag to a collector device having a collector node,

wherein the batch is sent as a single aggregate event,

wherein the collector device is associated with a collector tier,

wherein the collector node verifies the checksum for the batch of multiple messages received from the at least one agent node,

means for acknowledging, by at least one master in communication with the one or more machines and the collector device, that the collector node has written the batches into a file system and informing the at least one agent node that the batches have been safely stored in the file system.