Multi-tier aggregation for complex event correlation in streams

• US 10,135,853 B2
• Filed: 09/20/2016
• Issued: 11/20/2018
• Est. Priority Date: 09/20/2016
• Status: Active Grant

• Current Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Original Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Inventors: Bose, Brock D., Avasarala, Bhargav R., Steiner, Donald D.
• Primary Examiner(s): Chen, Shin-Hon (Eric)
• Application Number:
  US15/270,219
• Publication Number:
  US 20180083992A1
• Time in Patent Office: 791 Days
• Field of Search: None
• CPC Class Codes: H04L 63/1425 Traffic logging, e.g. anomaly detection

Abstract

A system and method for detecting anomalous activity, the method includes collecting data from a plurality of data sources, wherein each data source generates a data stream; harmonizing each data stream using a computer processor so that the harmonized data is in a common format; generating behavior models based on the harmonized data using the computer processor; analyzing the harmonized data at a first level using the behavior models and the computer processor to generate meta-events, wherein the meta-events represent anomalous behavior; analyzing the meta-events at a second level using the computer processor to determine if an alert should be issued; and when an alert should be issued, displaying the alert is disclosed.

Claims

1. A method for detecting anomalous activity, the method comprising:

collecting data from a plurality of data sources, wherein each data source generates a data stream;

harmonizing each data stream using a computer processor so that the harmonized data is in a common format;

generating behavior models based on the harmonized data using the computer processor;

analyzing the harmonized data at a first level using the behavior models and the computer processor to identify meta-events, wherein the meta-events represent anomalous behavior and analyzing the harmonized data at the first level identifies a meta-event based on a pre-defined set of anomalous activities;

analyzing the meta-events at a second level using the computer processor to determine if an alert should be issued, wherein;

the second level is a higher level of operation than the first level and encompasses the meta-events identified by analyzing the harmonized data at the first level, and

analyzing the meta-events at the second level includes determining whether an alert should be issued based on multiple meta-events; and

when an alert should be issued, displaying the alert.

2. The method of claim 1, wherein the harmonizing utilizes enrichment services.

3. The method of claim 2, wherein the enrichment services comprise:

providing identity resolution services for events in each data stream; and

enriching events in each data stream.

4. The method of claim 3, wherein enriching further comprises providing metadata associated with an event in each data stream.

5. The method of claim 1, wherein the common format is a homomorphism.

6. The method of claim 1, further comprising:

time ordering the harmonized data to generate a cumulative harmonized data stream.

7. The method of claim 1, wherein the behavior models are generated based on associations between events in the harmonized data.

8. The method of claim 1, wherein the behavioral models comprise a set of rules, models, and profiles which establish patterns of normalcy of an organization.

9. The method of claim 1, wherein the behavior models are modified based on newly collected data.

10. The method of claim 1, further comprising:

defining a set of anomalous activities using a complex event engine.

11. The method of claim 1, further comprising:

analyzing the meta-events at a third or higher level using the computer processor to determine if an alert should be issued.

12. A system for detecting anomalous activity, the system comprising:

a computer-readable storage device configured to store computer-executable instructions;

a hardware computer processor configured to execute the computer-executable instructions, the computer-executable instructions comprising;

collecting data from a plurality of data sources, wherein each data source generates a data stream;

harmonizing each data stream using the computer processor so that the harmonized data is in a common format;

generating behavior models based on the harmonized data using the computer processor;

analyzing the harmonized data at a first level using the behavior models and the computer processor to identify meta-events, wherein the meta-events represent anomalous behavior and analyzing the harmonized data at the first level identifies a meta-event based on a pre-defined set of anomalous activities;

analyzing the meta-events at a second level using the computer processor to determine if an alert should be issued, wherein;

the second level is a higher level of operation than the first level and encompasses the meta-events identified by analyzing the harmonized data at the first level, and

analyzing the meta-events at the second level includes determining whether an alert should be issued based on multiple meta-events; and

a display configured to display the alert.

13. The system of claim 12, wherein the harmonizing utilizes enrichment services.

14. The system of claim 13, wherein the enrichment services comprise:

providing identity resolution services for events in each data stream; and

enriching events in each data stream.

15. The system of claim 14, wherein enriching further comprises providing metadata associated with an event in each data stream.

16. The system of claim 12, wherein the common format is a homomorphism.

17. The system of claim 12, wherein the computer-executable instructions further comprise:

time ordering the harmonized data to generate a cumulative harmonized data stream.

18. The system of claim 12, wherein the behavior models are generated based on associations between events in the harmonized data.

19. The system of claim 12, wherein the behavioral models comprise a set of rules, models, and profiles which establish patterns of normalcy of an organization.

20. The system of claim 12, wherein the behavior models are modified based on newly collected data.

21. The system of claim 12, wherein the computer-executable instructions further comprise:

defining a set of anomalous activities using a complex event engine.

22. The system of claim 12, wherein the computer-executable instructions further comprise:

analyzing the meta-events at a third or higher level using the computer processor to determine if an alert should be issued.