Distributed sorting of event log files

• US 10,223,367 B2
• Filed: 03/04/2015
• Issued: 03/05/2019
• Est. Priority Date: 08/27/2014
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Pradeep, Aakash
• Primary Examiner(s): Lu, Kuen S
• Application Number:
  US14/638,799
• Publication Number:
  US 20160063044A1
• Time in Patent Office: 1,462 Days
• Field of Search: 707739
• CPC Class Codes: G06F 16/134 Distributed indices
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/182 Distributed file systems

Abstract

In an example, composite keys for an event log may be provided. A partitioner may be configured to extract a natural key from the composite keys and distribute log lines of event log files to a plurality of reducer nodes based on a value of the natural key. A comparator may use a log time of the composite key to sort a received portion of the distributed log lines.

Claims

1. A method, comprising:

providing a plurality of composite keys for an event log, each of the composite keys composed of more than one column of a relational database and being a combination of a first time value and at least a portion of a plurality of attributes of a natural key;

wherein one of the attributes of each plurality of attributes includes a second time value, and wherein the second time values have a different accuracy than the first time values;

extracting, by a processing device, natural key values from the composite keys;

distributing, by the processing device, log lines of event log files to a plurality of reducer nodes, the distributing being based on the extracted natural key values,

wherein a first log line having a first natural key value is distributed to a first one of the reducer nodes and a second log line having a second different natural key value is distributed to a second one of the reducer nodes, and

wherein the first one of the reducer nodes includes a first processor and the second one of the reducer nodes includes a second processor that is different than the first processor, and

wherein the processing device comprises a third processor that is different than the first and second processors; and

processing, by one of the first and second processors, a portion of the distributed log lines,

the processing comprising i) sorting log lines of the portion based on corresponding ones of the first time values, and ii) grouping the sorted log lines of the portion based on the natural keys; and

producing, by the one of the first and second processors, an executable and linkable format (ELF) file responsive to the grouping.

2. The method of claim 1, wherein the first time values have a greater accuracy than the second time values.

3. The method of claim 2, wherein the first time values have accuracy to a millisecond and the second time values have accuracy to only a unit greater than a millisecond.

4. The method of claim 3, wherein the second time values have accuracy to only a day.

5. The method of claim 1, wherein the first time values comprise log times.

6. The method of claim 1, further comprising:

hashing the extracted natural key values to determine hash values; and

wherein distributing, by the processing device, log lines of event log files to a plurality of reducer nodes further comprises distributing the log lines of the event log files to the plurality of reducer nodes based on the determined hash values.

7. The method of claim 1, wherein each plurality of attributes includes an identifier attribute corresponding to organization identity.

8. The method of claim 1, wherein each plurality of attributes includes an attribute corresponding to event type.

9. The method of claim 1, wherein:

sorting log lines of the portion based on corresponding ones of the first time values further comprises chronologically sorting the log lines of the portion based on the corresponding ones of the first time values.

10. The method of claim 1, wherein the event log files comprise information downloaded using an asynchronous file application programming interface.

11. A memory device having instructions stored thereon that, in response to execution by a processing device, cause the processing device to perform operations comprising:

identifying a plurality of composite keys for an event log, each of the composite keys composed of more than one column of a relational database and being a combination of a first time value and at least a portion of a plurality of attributes of a natural key;

wherein one of the attributes of each plurality of attributes includes a second time value, and wherein the second time values have a different accuracy than the first time values; and

extracting natural key values from the composite keys; and

distributing log lines of event log files to a plurality of reducer nodes, the distributing being based on the extracted natural key values,

wherein a first log line having a first natural key value is distributed to a first one of the reducer nodes and a second log line having a second different natural key value is distributed to a second one of the reducer nodes,

wherein the first one of the reducer nodes includes a first processor and the second one of the reducer nodes includes a second processor that is different than the first processor, and

wherein the processing device comprises a third processor that is different than the first and second processors, said distributing to control processing, by the first and second processors, of the distributed log lines,

the processing comprising i) sorting log lines based on the first time values, and ii) grouping the sorted log lines based on the natural keys,

wherein said processing produces an executable and linkable format (ELF) file.

12. The memory device of claim 11, wherein the first time values have a greater accuracy than the second time values.

13. The memory device of claim 12, wherein the first time values have accuracy to a millisecond and the second time values have accuracy to only a unit greater than a millisecond.

14. The memory device of claim 13, wherein the second time values have accuracy to only a day.

15. The memory device of claim 11, wherein the first time values comprise log times.

16. The memory device of claim 11, wherein the operations further comprise:

hashing the extracted natural key values to determine hash values; and

wherein distributing log lines of event log files to a plurality of reducer nodes further comprises distributing the log lines of the event log files to the plurality of reducer nodes based on the determined hash values.

17. The memory device of claim 11, wherein each plurality of attributes includes an identifier attribute corresponding to organization identity.

18. The memory device of claim 11, wherein each plurality of attributes includes an attribute corresponding to event type.

19. A memory device having instructions stored thereon that, in response to execution by a processing device, cause the processing device to perform operations comprising:

processing a received portion of distributed log lines distributed to a plurality of reducer nodes based on natural key that are associated with the log lines,

wherein the processing device comprises a first processor of a first one of the reducer nodes and wherein a second different one of the reducer nodes includes a second processor that is different than the first processor,

wherein the log lines are distributed by a third processor that is different than the first and second processors,

each log line of the received portion corresponding to a common time value of a corresponding one of the natural keys, the processing comprising;

sorting the log lines of the received portion based on different time values of composite keys associated with the log lines of the received portion, each of the composite keys composed of more than one column of a relational database;

wherein each of the different time values of the composite keys has a different accuracy than the common time value of the natural keys;

grouping the sorted log lines of the received portion based on the natural keys; and

producing an executable and linkable format (ELF) file responsive to the grouping.

20. The memory device of claim 19, wherein:

sorting the log lines of the received portion based on different time values of composite keys associated with the log lines of the received portion further comprises chronologically sorting the log lines of the first received portion based on the different time values of the composite keys.