Scalable analytical processing of structured data

• US 8,543,694 B2
• Filed: 11/23/2011
• Issued: 09/24/2013
• Est. Priority Date: 11/24/2010
• Status: Active Grant

• Current Assignee: Logrhythm Incorporated
• Original Assignee: Logrhythm Incorporated
• Inventors: Petersen, Chris, Villella, Phillip, Aisa, Brad
• Primary Examiner(s): Park, Jeong S
• Application Number:
  US13/303,526
• Publication Number:
  US 20120131185A1
• Time in Patent Office: 671 Days
• Field of Search: 709/224, 726 22- 33
• US Class Current: 709/224
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: H04L 41/069 using logs of notifications; Post-processing of notifications
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation
• CPC Class Codes: H04L 43/16 Threshold monitoring
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

An advanced intelligence engine (AIE) for use in identifying what may be complex events or developments on one or more data platforms or networks from various types of structured or normalized data generated by one or more disparate data sources. The AIE may conduct one or more types of quantitative, correlative, behavioral and corroborative analyses to detect events from what may otherwise be considered unimportant or non-relevant information spanning one or more time periods. Events generated by the AIE may be passed to an event manager to determine whether further action is required such as reporting, remediation, and the like.

Claims

1. A method for use in monitoring one or more platforms of one or more data systems, comprising:

receiving, at a processing engine, structured data generated by one or more platforms over at least one communications network;

first evaluating, at the processing engine using a first rule block, at least some of the data, wherein the first evaluating comprises making a determination about a content of one or more fields of the at least some of the data;

first determining, from the first evaluating, that a result is one of at least first and second outcomes;

depending upon the first determining, second evaluating, at the processing engine using a second rule block, at least some of the data, wherein the second evaluating comprises making a determination about a content of one or more fields of the at least some of the data, and wherein a content of one of the fields in the first evaluating matches a content of one of the fields in the second evaluating; and

second determining, from the second evaluating, whether a result is one of at least first and second outcomes, wherein the results are analyzed to determine an event of interest.

2. The method of claim 1, wherein the first determining comprises:

ascertaining that the result of the first evaluating is the first outcome.

3. The method of claim 2, further comprising:

ascertaining that the result of the second evaluating is the first outcome; and

generating an event after the result of the second evaluating is determined to be the first outcome.

4. The method of claim 2, wherein the first outcome of the first determining occurs at a first time that is identified by at least one time stamp of the at least some of the data.

5. The method of claim 4, wherein the second evaluating comprises:

considering data of the at least some of the data that is associated with at least one time stamp including a second time that is the same as or after the first time.

6. The method of claim 4, wherein the second evaluating comprises:

considering data of the at least some of the data that is associated with at least one time stamp including a second time that is before the first time.

7. The method of claim 1, wherein the second evaluating comprises:

utilizing the matching content as a key into an index structure of the at least some of the data to determine if the result is one of the first and second outcomes.

8. The method of claim 1, wherein the first outcome of at least one of the first and second determining comprises data of the at least some of the data matching first criteria.

9. The method of claim 8, wherein the first criteria comprises a particular content of one or more fields of the data of the at least some of the data.

10. The method of claim 8, wherein the first outcome further comprises a particular quantitative value of the data matching the first criteria being equal to or exceeding a threshold for the particular quantitative value.

11. The method of claim 8, wherein the first outcome further comprises a particular field of the data comprising at least a threshold number of unique contents of the particular field.

12. The method of claim 11, wherein the first outcome further comprises the particular field of the data comprising at least the threshold number of unique contents of the particular field within a particular time period.

13. The method of claim 12, wherein the particular time period is measured in relation to a time at which the first outcome of one of the first and second evaluating occurred.

14. The method of claim 1, wherein the result of each of the first and second evaluating comprises the first outcomes, and wherein the method further comprises:

third evaluating, at the processing engine using a third rule block, at least some of the data, wherein the results are analyzed to determine an event of interest.

15. The method of claim 1, wherein the content of the one of the fields in the first evaluating matching the content of the one of the fields in the second evaluating comprises the content about which a determination was made in the first evaluating.

16. A system for use in monitoring one or more platforms of one or more data systems, comprising:

A processor; and

a memory connected to the processor and comprising a set of computer readable instructions that are executable by the processor to;

receive structured data generated by one or more platforms over at least one communications network;

first evaluate, using a first rule block, at least some of the data by making a determination about a content of one or more fields of the at least some of the data;

first determine, from the first evaluating, that a result is one of at least first and second outcomes; and

depending upon the first determining, second evaluate, using a second rule block, at least some of the data by making a determination about a content of one or more fields of the at least some of the data, wherein a content of one of the fields that are first evaluated matches a content of one of the fields that are second evaluated; and

second determine, from the second evaluating, whether a result is one of at least first and second outcomes, wherein the results are analyzed to determine an event of interest.

17. A method for use in monitoring one or more platforms of one or more data systems, comprising:

receiving, at a processing engine, structured data generated by one or more platforms over at least one communications network;

first evaluating, at the processing engine using one of first and second rule blocks, at least some of the data, wherein the first evaluating comprises making a determination about a content of one or more fields of the at least some of the data;

first determining that a result of the first evaluating is a first of at least first and second outcomes, wherein the at least some of the data leading to the first outcome is identified by a time stamp that corresponds to a first time;

second evaluating, at the processing engine using the other of the first and second rule blocks, at least some of the data associated with one or more time stamps that correspond to one or more second times having a specified relationship to the first time, wherein the second evaluating comprises making a determination about a content of one or more fields of the at least some of the data;

second determining, from the second evaluating, whether a result is one of at least first and second outcomes, wherein the results are analyzed to determine an event of interest, and wherein a content of one of the fields in the

first evaluating matches a content of one of the fields in the second evaluating.

18. The method of claim 17, wherein the specified relationship comprises the one or more second times occurring before the first time.

19. The method of claim 17, wherein the specified relationship comprises the one or more second times occurring at the same time as or after the first time.

20. The method of claim 17, wherein the first evaluating uses the second rule block and the second evaluating uses the first rule block.

21. The method of claim 17, wherein the first evaluating uses the first rule block and the second evaluating uses the second rule block.

22. The method of claim 17, wherein the second determining comprises second determining that the result is the first outcome, and wherein the method further comprises:

generating an event in response to the second determining comprising the first outcome.

23. The method of claim 17, wherein the second determining comprises second determining that the result is the second outcome, and wherein the method further comprises after the second determining:

third evaluating, at the processing engine using the other of the first and second rule blocks, at least some of the data;

third determining that a result of the third evaluating is a first of at least first and second outcomes, wherein the at least some of the data leading to the first outcome is identified by a time stamp that corresponds to a third time;

fourth evaluating, at the processing engine using the one of the first and second rule blocks, at least some of the data associated with the time stamp corresponding to the first time having a specified relationship to the third time;

fourth determining, from the fourth evaluating, whether a result is one of at least first and second outcomes, wherein the results are analyzed to determine an event of interest.

24. The method of claim 23, wherein the fourth determining comprises fourth determining that the result is the first outcome, and wherein the method further comprises:

generating an event in response to the fourth determining comprising the first outcome.

25. The method of claim 17, wherein the second evaluating comprises:

utilizing the matching content as a key into an index structure of the at least some of the data to determine if the result is one of the first and second outcomes.