Collecting and aggregating log data with fault tolerance

• US 9,817,859 B2
• Filed: 06/01/2016
• Issued: 11/14/2017
• Est. Priority Date: 03/31/2010
• Status: Active Grant

• Current Assignee: Cloudera Incorporated
• Original Assignee: Cloudera Incorporated
• Inventors: Hsieh, Jonathan Ming-Cyn, Robinson, Henry Noel
• Primary Examiner(s): Butler, Sarai
• Application Number:
  US15/170,824
• Publication Number:
  US 20160275136A1
• Time in Patent Office: 531 Days
• Field of Search: 714 49
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 2201/875 Monitoring of systems including the internet
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 67/125 involving control of end-device applications over a network

Abstract

Systems and methods of collecting and aggregating log data with fault tolerance are disclosed. One embodiment includes, one or more devices that generate log data, the one or more machines each associated with an agent node to collect the log data, wherein, the agent node generates a batch comprising multiple messages from the log data and assigns a tag to the hatch. In one embodiment, the agent node further computes a checksum for the batch of multiple messages. The system may further include a collector device, the collector device being associated with a collector tier having a collector node to which the agent sends the log data; wherein, the collector determines the checksum for the hatch of multiple messages received from the agent node.

Claims

1. A method for collecting and aggregating datasets with fault tolerance, the method comprising:

collecting a dataset from a data source on a machine where the dataset is generated, wherein the dataset is collected by an agent node executed on the machine;

generating a batch comprising messages from the dataset;

assigning a tag to the batch and computing a checksum for the batch;

writing the tag, the batch comprising the messages, and the checksum to an entry in a write-ahead-log (WAL) in a storage;

sending the batch comprising the messages to a receiving location;

verifying the checksum of the batch comprising the messages at the receiving location; and

in response to determining that receiving location has failed, storing, by the agent node, the dataset in a persistent storage of the machine until the receiving location is repaired or until another destination is identified.

2. The method of claim 1, further comprising:

writing a file to a destination location; and

identifying, in the file, tags associated with batches in the file that have been written to the destination location.

3. The method of claim 2, wherein the destination location is a storage tier comprising a distributed file system.

4. The method of claim 1, further comprising:

in response to verifying the checksum of the batch comprising messages at the receiving location, adding the tag to a map.

5. The method of claim 4, wherein the map is associated with multiple tags assigned to multiple batches of messages from the dataset.

6. The method of claim 1, further comprising querying for an acknowledgement message that the batch has been written to the destination location.

7. The method of claim 1, further comprising, in response to determining that the batch has been written in the destination location, deleting the entry in the storage of the machine.

8. The method of claim 1, further comprising, in response to determining that the batch has not been written to a file in the destination location, resending the batch.

9. The method of claim 1, wherein the receiving location is a collector node in a collector tier other than the machine where the dataset is generated.

10. The method of claim 1, wherein the receiving location is a file system in a storage tier comprising one or more storage devices.

11. The method of claim 1, wherein the storage for the WAL is on the machine where the dataset is generated.

12. An apparatus for collecting and aggregating datasets for storage in a file system with fault tolerance, the apparatus including a memory storing instructions that, when executed by a processor of the apparatus, cause the apparatus to perform a method comprising:

collecting a dataset from a data source on a machine where the dataset is generated, wherein the dataset is collected by an agent node executed on the machine;

generating a batch comprising messages from the dataset;

assigning a tag to the batch and computing a checksum for the batch;

writing the tag, the batch comprising the messages, and the checksum to an entry in a write-ahead-log (WAL) in a storage;

sending the batch comprising the messages to a receiving location;

verifying the checksum of the batch comprising the messages at the receiving location; and

in response to determining that receiving location has failed, storing, by the agent node, the dataset in a persistent storage of the machine until the receiving location is repaired or until another destination is identified.

13. The apparatus of claim 12, the method further comprising:

writing a file to a destination location; and

identifying, in the file, tags associated with batches in the file that have been written to the destination location.

14. The apparatus of claim 13, wherein the destination location is a storage tier comprising a distributed file system.

15. The apparatus of claim 12, the method further comprising:

in response to verifying the checksum of the batch comprising messages at the receiving location, adding the tag to a map.

16. The apparatus of claim 15, wherein the map is associated with multiple tags assigned to multiple batches of messages from the dataset.

17. The apparatus of claim 12, the method further comprising querying for an acknowledgement message that the batch has been written to the destination location.

18. The apparatus of claim 12, the method further comprising, in response to determining that the batch has been written in the destination location, deleting the entry in the storage of the machine.

19. The apparatus of claim 12, the method further comprising, in response to determining that the batch has not been written to a file in the destination location, resending the batch.

20. The apparatus of claim 12, wherein the receiving location is a collector node in a collector tier other than the machine where the dataset is generated.

21. The apparatus of claim 12, wherein the receiving location is a file system in a storage tier comprising one or more storage devices.

22. The apparatus of claim 12, wherein the storage for the WAL is on the machine where the dataset is generated.