Collecting and aggregating log data with fault tolerance

• US 9,361,203 B2
• Filed: 07/10/2015
• Issued: 06/07/2016
• Est. Priority Date: 03/31/2010
• Status: Active Grant

• Current Assignee: Cloudera Incorporated
• Original Assignee: Cloudera Incorporated
• Inventors: Hsieh, Jonathan Ming-Cyn, Robinson, Henry Noel
• Primary Examiner(s): BUTLER, SARAI E
• Application Number:
  US14/796,812
• Publication Number:
  US 20150317231A1
• Time in Patent Office: 333 Days
• Field of Search: 714/49
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 2201/875 Monitoring of systems including the internet
• CPC Class Codes: H04L 41/046 comprising network management agents or mobile agents therefor
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 67/125 involving control of end-device applications over a network

Abstract

Systems and methods of collecting and aggregating log data with fault tolerance are disclosed. One embodiment includes, one or more devices that generate log data, the one or more machines each associated with an agent node to collect the log data, wherein, the agent node generates a batch comprising multiple messages from the log data and assigns a tag to the batch. In one embodiment, the agent node further computes a checksum for the batch of multiple messages. The system may further include a collector device, the collector device being associated with a collector tier having a collector node to which the agent sends the log data; wherein, the collector determines the checksum for the batch of multiple messages received from the agent node.

Claims

1. A method for collecting and aggregating datasets for storage in a file system with fault tolerance, the method comprising:

collecting datasets from a data source on a machine where the datasets are generated;

generating a batch comprising multiple messages from the datasets;

assigning a tag to the batch and computing a checksum for the batch;

writing the tag, the batch of multiple messages, and the checksum to an entry in a write-ahead-log (WAL) in storage;

sending the datasets to a receiving location;

in response to verifying the checksum of the batch of multiple messages at the receiving location, adding the tag to a map;

writing a file to destination location; and

identifying, in the file, tags associated with the batches in the file that have been written to the destination location.

2. The method of claim 1, wherein the storage for the WAL is on the machine where the datasets are generated.

3. The method of claim 1, wherein the map is associated with multiple tags assigned to multiple batches of messages from the datasets.

4. The method of claim 1, wherein the destination location is a storage tier comprising a distributed file system.

5. The method of claim 1, further comprising querying for an acknowledgement message that the batch has been written to the destination location.

6. The method of claim 1, further comprising, in response to determining that the batch has been written in the destination location, deleting the entry in the storage of the machine.

7. The method of claim 1, further comprising, in response to determining that the batch has not been written to a file in the destination location, resending the batch.

8. An apparatus for collecting and aggregating datasets for storage in a file system with fault tolerance, the apparatus including a memory storing instructions that, when executed by a processor of the apparatus, cause the apparatus to perform a method comprising:

collecting datasets from a data source on a machine where the datasets are generated;

generating a batch comprising multiple messages from the datasets;

assigning a tag to the batch and computing a checksum for the batch;

writing the tag, the batch of multiple messages, and the checksum to an entry in a write-ahead-log (WAL) in storage;

sending the datasets to a receiving location;

in response to verifying the checksum of the batch of multiple messages at he receiving location, adding the tag to a map;

writing a file to destination location; and

identifying, in the file, tags associated with the batches in the file that have been written to the destination location.

9. The apparatus of claim 8, wherein the storage for the WAL is on the machine where the datasets are generated.

10. The apparatus of claim 8, wherein the map is associated with multiple tags assigned to multiple batches of messages from the datasets.

11. The apparatus of claim 8, wherein the destination location is a storage tier comprising a distributed file system.

12. The apparatus of claim 8, further comprising querying for an acknowledgement message that the batch has been written to the destination location.

13. The apparatus of claim 8, further comprising, in response to determining that the batch has been written in the destination location, deleting the entry in the storage of the machine.

14. The apparatus of claim 8, further comprising, in response to determining that the batch has not been written to a file in the destination location, resending the batch.

15. A non-transitory computer-readable medium for collecting and aggregating datasets for storage in a file system with fault tolerance, the medium storing instructions that, when executed by a processor of an apparatus, cause the apparatus to perform a method comprising:

collecting datasets from a data source on a machine where the datasets are generated;

generating a batch comprising multiple messages from the datasets;

assigning a tag to the batch and computing a checksum for the batch;

writing the tag, the batch of multiple messages, and the checksum to an entry in a write-ahead-log (WAL) in storage;

sending the datasets to a receiving location;

in response to verifying the checksum of the batch of multiple messages at the receiving location, adding the tag to a map;

writing a file to destination location; and

identifying, in the file, tags associated with the batches in the file that have been written to the destination location.

16. The medium of claim 15, wherein the storage for the WAL is on the machine where the datasets are generated.

17. The medium of claim 15, wherein the map is associated with multiple tags assigned to multiple batches of messages from the datasets.

18. The medium of claim 15, wherein the destination location is a storage tier comprising a distributed file system.

19. The medium of claim 15, the method further comprising querying for an acknowledgement message that the batch has been written to the destination location.

20. The medium of claim 15, the method further comprising, selectively performing deleting the entry in the storage of the machine or resending the batch based on whether the batch has been successfully written in the destination location.