SYSTEM AND METHOD FOR REAL-TIME DETECTION OF ANOMALIES IN DATABASE USAGE

• US 20150355957A1
• Filed: 06/05/2015
• Published: 12/10/2015
• Est. Priority Date: 06/09/2014
• Status: Active Grant

• Current Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Original Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Inventors: Steiner, Donald, Day, John
• Granted Patent: US 10,409,665 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0772 Means for error signaling, e.g. using interrupts, exception flags, dedicated error registers
• CPC Class Codes: G06F 11/0787 Storage of error reports, e.g. persistent data storage, storage using memory protection
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 21/6227 where protection concerns the structure of data, e.g. records, types, queries
• CPC Class Codes: G06F 2201/86 Event-based monitoring

Abstract

A system and method for real-time detection of anomalies in database or application usage is disclosed. Embodiments provide a mechanism to detect anomalies in database or application usage, such as data exfiltration attempts, first by identifying correlations (e.g., patterns of normalcy) in events across different heterogeneous data streams (such as those associated with ordinary, authorized and benign database usage, workstation usage, user behavior or application usage) and second by identifying deviations/anomalies from these patterns of normalcy across data streams in real-time as data is being accessed. An alert is issued upon detection of an anomaly, wherein a type of alert is determined based on a characteristic of the detected anomaly.

Claims

1. A method for real-time detection of anomalies comprising:

receiving a plurality of heterogeneous data streams, wherein the heterogeneous data streams are received from at least two of a group consisting of agents located at databases, agents located at applications, audit programs located at user workstations, and sensors located in, or at access points to, a network;

correlating the heterogeneous data streams, wherein the correlation identifies corresponding events in different ones of the heterogeneous data streams;

identifying patterns of events across the correlated heterogeneous data streams;

building a model of normalcy from the identified pattern of events, wherein the model of normalcy is stored in an analysis database;

creating rules that determine how and whether anomalies are detected, how a detected anomaly is treated and characterized, and what reaction to employ upon detection of the anomaly;

receiving a plurality of additional heterogeneous data streams from the at least two of a group consisting of the agents, audit programs, and sensors;

applying, using an analysis engine, the model of normalcy and rules to the additional heterogeneous data streams and analyzing data from the additional heterogeneous data streams against the model of normalcy and rules;

detecting an anomaly in real-time by determining whether an anomalous event is present, by the application of the rules and whether events, in relation to other events within the additional heterogeneous data streams, fit or do not fit the model of normalcy;

determining at least one characteristic of the detected anomaly; and

issuing an alert upon detection of the anomaly, wherein a type of alert is determined based on the at least one determined characteristic of the detected anomaly.

2. The method of claim 1 wherein the correlating is performed using a complex event processor for correlating the heterogeneous data streams and integrating the heterogeneous data streams into a single integrated data stream.

3. The method of claim 2 wherein the correlating includes synchronizing the time of each of the heterogeneous data streams in order to correlate events across time.

4. The method of claim 2 wherein the correlating is performed by application of a small-space algorithm (SSA).

5. The method of claim 2 wherein, in the applying step, the analysis engine uses the complex event processor for applying the model of normalcy and rules to the additional heterogeneous data streams and for analyzing data from the additional heterogeneous data streams against the model of normalcy and rules.

6. The method of claim 5 wherein events with the additional heterogeneous data streams are processed using an automatic event tabulator and correlator (AETAC) algorithm to reduce event data complexity and facilitate search, retrieval, and correlation of the event data to thereby produce uniform event data whereby complex event processing of the additional heterogeneous data streams by the complex event processor is simplified.

7. The method of claim 2 wherein events with the heterogeneous data streams are processed using an automatic event tabulator and correlator (AETAC) algorithm to reduce event data complexity and facilitate search, retrieval, and correlation of the event data to thereby produce uniform event data whereby complex event processing of the heterogeneous data streams by the complex event processor is simplified.

8. The method of claim 1 wherein the identified pattern of events are associated with ordinary, authorized, and benign database usage, workstation usage, user behavior or application usage.

9. The method of claim 1 wherein the detected anomaly is indicative of unauthorized manipulation or falsification of data, sabotage of a database, or exfiltration of data.

10. The method of claim 1 wherein the heterogeneous data streams are multi-modal asynchronous signals.

11. The method of claim 1 wherein one of the heterogeneous data streams corresponds to an ordinary, authorized, and benign database query and another of the heterogeneous data streams corresponds to an ordinary, authorized, and benign user interaction at the user workstation, and wherein the identified pattern of events is the ordinary, authorized, and benign database query and the ordinary, authorized, and benign user interaction at the user workstation.

12. The method of claim 11 wherein the detecting step detects the anomaly when an event within the additional heterogeneous data streams corresponding to a database query is not preceded by a user interaction at the user workstation within a predetermined period of time resulting in the event not fitting the model of normalcy.

13. The method of claim 1 wherein the alert comprises at least one of a group consisting of alarm message, a communication triggering further analysis and/or action, a command instructing the restriction or shutting down of an affected workstation, database, network or network access, initiation of additional targeted monitoring, analysis, and/or applications to capture additional detailed information regarding an attack, continued monitoring of a user, placement of a flag in a file for further follow-up, restricting access to a network, alerting security, and restricting or locking down a building or a portion of a building.

14. The method of claim 1 wherein, in the receiving a plurality of heterogeneous data streams step, the heterogeneous data streams are received by the analysis engine.

15. The method of claim 1 wherein the databases, applications, workstations, or networks are in an enterprise environment.

16. A system for real-time detection of anomalies comprising:

one or more computers, each computer including a processor and memory, wherein the memory includes instructions that are executed by the processor for;

receiving a plurality of heterogeneous data streams, wherein the heterogeneous data streams are received from at least two of a group consisting of agents located at databases, agents located at applications, audit programs located at user workstations, and sensors located in, or at access points to, a network;

correlating the heterogeneous data streams, wherein the correlation identifies corresponding events in different ones of the heterogeneous data streams;

identifying patterns of events across the correlated heterogeneous data streams;

building a model of normalcy from the identified pattern of events, wherein the model of normalcy is stored in an analysis database;

creating rules that determine how and whether anomalies are detected, how a detected anomaly is treated and characterized, and what reaction to employ upon detection of the anomaly;

receiving a plurality of additional heterogeneous data streams from the at least two of a group consisting of the agents, audit programs, and sensors;

applying, using an analysis engine, the model of normalcy and rules to the additional heterogeneous data streams and analyzing data from the additional heterogeneous data streams against the model of normalcy and rules;

detecting an anomaly in real-time by determining whether an anomalous event is present, by the application of the rules and whether events, in relation to other events within the additional heterogeneous data streams, fit or do not fit the model of normalcy;

determining at least one characteristic of the detected anomaly; and

issuing an alert upon detection of the anomaly, wherein a type of alert is determined based on the at least one determined characteristic of the detected anomaly.

17. The system of claim 16 wherein the correlating is performed using a complex event processor for correlating the heterogeneous data streams and integrating the heterogeneous data streams into a single integrated data stream.

18. The system of claim 17 wherein the correlating includes synchronizing the time of each of the heterogeneous data streams in order to correlate events across time.

19. The system of claim 17 wherein the correlating is performed by application of a small-space algorithm (SSA).

20. The system of claim 17 wherein, in the applying step, the analysis engine uses the complex event processor for applying the model of normalcy and rules to the additional heterogeneous data streams and for analyzing data from the additional heterogeneous data streams against the model of normalcy and rules.

21. The system of claim 20 wherein events with the additional heterogeneous data streams are processed using an automatic event tabulator and correlator (AETAC) algorithm to reduce event data complexity and facilitate search, retrieval, and correlation of the event data to thereby produce uniform event data whereby complex event processing of the additional heterogeneous data streams by the complex event processor is simplified.

22. The system of claim 17 wherein events with the heterogeneous data streams are processed using an automatic event tabulator and correlator (AETAC) algorithm to reduce event data complexity and facilitate search, retrieval, and correlation of the event data to thereby produce uniform event data whereby complex event processing of the heterogeneous data streams by the complex event processor is simplified.

23. The system of claim 16 wherein the identified pattern of events are associated with ordinary, authorized, and benign database usage, workstation usage, user behavior or application usage.

24. The system of claim 16 wherein the detected anomaly is indicative of unauthorized manipulation or falsification of data, sabotage of a database, or exfiltration of data.

25. The system of claim 16 wherein the heterogeneous data streams are multi-modal asynchronous signals.

26. The system of claim 16 wherein one of the heterogeneous data streams corresponds to an ordinary, authorized, and benign database query and another of the heterogeneous data streams corresponds to an ordinary, authorized, and benign user interaction at the user workstation, and wherein the identified pattern of events is the ordinary, authorized, and benign database query and the ordinary, authorized, and benign user interaction at the user workstation.

27. The system of claim 26 wherein the detecting step detects the anomaly when an event within the additional heterogeneous data streams corresponding to a database query is not preceded by a user interaction at the user workstation within a predetermined period of time resulting in the event not fitting the model of normalcy.

28. The system of claim 16 wherein the alert comprises at least one of a group consisting of alarm message, a communication triggering further analysis and/or action, a command instructing the restriction or shutting down of an affected workstation, database, network or network access, initiation of additional targeted monitoring, analysis, and/or applications to capture additional detailed information regarding an attack, continued monitoring of a user, placement of a flag in a file for further follow-up, restricting access to a network, alerting security, and restricting or locking down a building or a portion of a building.

29. The system of claim 16 wherein, in the receiving a plurality of heterogeneous data streams step, the heterogeneous data streams are received by the analysis engine.

30. The system of claim 16 wherein the databases, applications, workstations, or networks are in an enterprise environment.