System and method for real-time detection of anomalies in database usage

• US 10,409,665 B2
• Filed: 06/05/2015
• Issued: 09/10/2019
• Est. Priority Date: 06/09/2014
• Status: Active Grant

• Current Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Original Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Inventors: Steiner, Donald, Day, John
• Primary Examiner(s): Wilson, Yolanda L
• Application Number:
  US14/732,162
• Publication Number:
  US 20150355957A1
• Time in Patent Office: 1,558 Days
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0772 Means for error signaling, e.g. using interrupts, exception flags, dedicated error registers
• CPC Class Codes: G06F 11/0787 Storage of error reports, e.g. persistent data storage, storage using memory protection
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 21/552 involving long-term monitoring or reporting
• CPC Class Codes: G06F 21/6227 where protection concerns the structure of data, e.g. records, types, queries
• CPC Class Codes: G06F 2201/86 Event-based monitoring

Abstract

A system and method for real-time detection of anomalies in database or application usage is disclosed. Embodiments provide a mechanism to detect anomalies in database or application usage, such as data exfiltration attempts, first by identifying correlations (e.g., patterns of normalcy) in events across different heterogeneous data streams (such as those associated with ordinary, authorized and benign database usage, workstation usage, user behavior or application usage) and second by identifying deviations/anomalies from these patterns of normalcy across data streams in real-time as data is being accessed. An alert is issued upon detection of an anomaly, wherein a type of alert is determined based on a characteristic of the detected anomaly.

Claims

1. A computerized system for real-time detection of anomalies occurring in an enterprise computer network, comprising:

at least one processor; and

at least one memory including program code that, when executed by the at least one processor, causes the system to;

receive a plurality of heterogeneous data streams from sources in the network, the sources including two levels, first level sources and second level sources,

wherein the first level sources include one or more selected from a group consisting of agents located at databases;

agents located at applications;

audit programs located at user workstations;

sensors located in the network; and

sensors located at access points to the network;

wherein the second level sources include one or more selected from a group consisting of data access, user behavior, computer activity and network activity; and

wherein the first level sources monitor event streams of the second level sources and generate data streams indicative of corresponding second level source activity in a uniform format;

process the heterogeneous data streams obtained by combining at least two of the first level sources to identify events therein, each event being identified by at least a unique ID, a timestamp, and an event type, wherein the processing of the heterogeneous data streams includes combining at least two of the first level sources into a single data stream;

correlate the processed heterogeneous data streams to form an integrated data stream comprising a plurality of identified events;

detect the existence and at least one characteristic of an anomaly in the computer network by application of a predetermined model of normalcy and one or more anomaly rules to the integrated data stream comprising the plurality of identified events; and

issue an alert based on the at least one characteristic of the anomaly.

2. The computerized system of claim 1, wherein the executed program code causes the system to create the predetermined model of normalcy and the one or more anomaly rules by:

receiving additional data comprising the plurality of heterogeneous data streams, wherein the additional data corresponds to authorized and benign usage of network resources;

processing the heterogeneous data streams to identify events therein, each even being identified by at least a unique ID, a timestamp, and an event type;

correlating the processed data streams to form an integrated data stream comprising a plurality of identified events;

identifying one or more patterns from relations between identified events comprising the integrated data stream; and

creating the model of normalcy and the one or more anomaly rules based on the identified one or more patterns.

3. The computerized system of claim 2, wherein the additional data corresponds to an ordinary, authorized, and benign database query and an ordinary, authorized, and benign user interaction at a user workstation.

4. The computerized system of claim 3, wherein the predetermined model of normalcy comprises the user interaction at the user workstation within a predetermined period of time before the database query.

5. The computerized system of claim 3, wherein the detected anomaly comprises a lack of user interaction at the user workstation within a predetermined period of time before the database query.

6. The computerized system of claim 1, wherein the one or more anomaly rules relate to at least one of:

how and whether anomalies are detected, how a detected anomaly is treated and characterized, and what reaction to employ in response to the detected anomaly.

7. The computerized system of claim 1, wherein the detected anomaly is indicative of unauthorized manipulation or falsification of data, sabotage of a database, or exfiltration of data.

8. The computerized system of claim 1, wherein the heterogeneous data streams comprise multi-modal asynchronous signals.

9. The computerized system of claim 1, wherein the alert comprises at least one of an alarm message, a communication triggering further analysis and/or action, a command instructing the restriction or shutting down of an affected workstation, database, network or network access, initiation of additional targeted monitoring, analysis, and/or applications to capture additional detailed information regarding an attack, continued monitoring of a user, placement of a flag in a file for further follow-up, restricting access to a network, alerting security, and restricting or locking down a building or a portion of a building.

10. The computerized system of claim 1, wherein the program code includes an algorithm that detects and extracts persistent events among the plurality of identified events in at least one of the plurality of heterogeneous data streams, and wherein the persistent events are time-stamped data that appear regularly overtime.

11. The computerized system of claim 10, wherein the persistent events appear in different distributed streams among the plurality of heterogeneous data streams.

12. The computerized system of claim 10, wherein the at least one of the plurality of heterogeneous data streams is statistically sampled to reduce stream size of the at least one of the plurality of heterogeneous data streams, without overlooking the persistent events.

13. The computerized system of claim 1, wherein the processing of the heterogeneous data streams includes transformation of heterogeneous data stream inputs into a standardized data structure to produce homomorphism.

14. The computerized system of claim 1, wherein the processing of the heterogeneous data streams includes using an automatic event tabulator and correlator.

15. The computerized system of claim 1, wherein the processing of the heterogeneous data streams includes operating on the single data stream using an algorithm that identifies temporal relationships.

16. The computerized system of claim 1, wherein the processing of the heterogeneous data streams includes operating on the single data stream using an algorithm that identifies spatial relationships.

17. A computerized system for real-time detection of anomalies occurring in an enterprise computer network, comprising:

at least one processor; and

at least one memory including program code that, when executed by the at least one processor, causes the system to;

receive a plurality of heterogeneous data streams from sources in the network, the sources including two levels, first level sources and second level sources,

wherein the first level sources include the following;

agents located at databases;

agents located at applications;

audit programs located at user workstations;

sensors located in the network; and

sensors located at access points to the network; and

wherein the second level sources include the following;

email exchanges;

instant messages;

voice mails;

documents;

database content; and

user interactions and database queries;

process the heterogeneous data streams obtained by combining at least one of the first level sources with at least one of the second level sources to identify events therein, each event being identified by at least a unique ID, a timestamp, and an event type;

correlate the processed heterogeneous data streams to form an integrated data stream comprising a plurality of identified events;

detect the existence and at least one characteristic of an anomaly in the computer network by application of a predetermined model of normalcy and one or more anomaly rules to the integrated data stream comprising the plurality of identified events; and

issue an alert based on the at least one characteristic of the anomaly, wherein the executed program code causes the system to correlate the processed data streams by;

estimating the number or frequency of one or more event types in the processed data stream without searching the entire processed data stream; and

determining one or more temporal, spatial, or generalized associations between a plurality of events in the processed data stream.

18. A non-transitory, computer-readable medium comprising a set of instructions for real-time detection of anomalies occurring in an enterprise computer network, wherein execution of said instructions by a system comprising at least one processor cause the system to:

receive a plurality of heterogeneous data streams from sources in the network, the sources including two levels, first level sources and second level sources,

wherein the first level sources include one or more selected from a group consisting of agents located at databases;

agents located at applications;

audit programs located at user workstations;

sensors located in the network; and

sensors located at access points to the network;

wherein the second level sources include one or more selected from a group consisting of data access, user behavior, computer activity and network activity; and

wherein the first level sources monitor event streams of the second level sources and generate data streams indicative of corresponding second level source activity in a uniform format;

process the heterogeneous data streams obtained by combining at least two of the first level sources to identify events therein, each event being identified by at least a unique ID, a timestamp, and an event type, wherein the processing of the heterogeneous data streams includes combining at least two of the first level sources into a single data stream;

correlate the processed heterogeneous data streams to form an integrated data stream comprising a plurality of identified events;

detect the existence and at least one characteristic of an anomaly in the computer network by application of a predetermined model of normalcy and one or more anomaly rules to the integrated data stream comprising the plurality of identified events; and

issue an alert based on the at least one characteristic of the anomaly.

19. The non-transitory, computer-readable medium of claim 18, wherein the executed instructions cause the system to create the predetermined model of normalcy and the one or more anomaly rules by:

receiving additional data comprising the plurality of heterogeneous data streams, wherein the additional data corresponds to authorized and benign usage of network resources;

processing the heterogeneous data streams to identify events therein, each even being identified by at least a unique ID, a timestamp, and an event type;

correlating the processed data streams to form an integrated data stream comprising a plurality of identified events;

identifying one or more patterns from relations between identified events comprising the integrated data stream; and

creating the model of normalcy and the one or more anomaly rules based on the identified one or more patterns.

20. The non-transitory, computer-readable medium of claim 19, wherein the additional data corresponds to an ordinary, authorized, and benign database query and an ordinary, authorized, and benign user interaction at a user workstation.

21. The non-transitory, computer-readable medium of claim 20, wherein the predetermined model of normalcy comprises the user interaction at the user workstation within a predetermined period of time before the database query.

22. The non-transitory, computer-readable medium of claim 20, wherein the detected anomaly comprises a lack of user interaction at the user workstation within a predetermined period of time before the database query.

23. The non-transitory, computer-readable medium of claim 18, wherein the one or more anomaly rules relate to at least one of:

how and whether anomalies are detected, how a detected anomaly is treated and characterized, and what reaction to employ in response to the detected anomaly.

24. The non-transitory, computer-readable medium of claim 18, wherein the detected anomaly is indicative of unauthorized manipulation or falsification of data, sabotage of a database, or exfiltration of data.

25. The non-transitory, computer-readable medium of claim 18, wherein the heterogeneous data streams comprise multi-modal asynchronous signals.

26. The non-transitory, computer-readable medium of claim 18, wherein the alert comprises at least one of an alarm message, a communication triggering further analysis and/or action, a command instructing the restriction or shutting down of an affected workstation, database, network or network access, initiation of additional targeted monitoring, analysis, and/or applications to capture additional detailed information regarding an attack, continued monitoring of a user, placement of a flag in a file for further follow-up, restricting access to a network, alerting security, and restricting or locking down a building or a portion of a building.

27. The non-transitory, computer-readable medium of claim 18, wherein the program code includes an algorithm that detects and extracts persistent events among the plurality of identified events in at least one of the plurality of heterogeneous data streams, and wherein the persistent events are time-stamped data that appear regularly over time.

28. The non-transitory, computer-readable medium of claim 27, wherein the persistent events appear in different distributed streams among the plurality of heterogeneous data streams.

29. The non-transitory, computer-readable medium of claim 27, wherein the at least one of the plurality of heterogeneous data streams is statistically sampled to reduce stream size of the at least one of the plurality of heterogeneous data streams, without overlooking the persistent events.

30. The non-transitory, computer-readable medium of claim 18, wherein the processing of the heterogeneous data streams includes transformation of heterogeneous data stream inputs into a standardized data structure to produce homomorphism.

31. The non-transitory, computer-readable medium of claim 18, wherein the processing of the heterogeneous data streams includes using an automatic event tabulator and correlator.

32. The non-transitory, computer-readable medium of claim 18, wherein the processing of the heterogeneous data streams includes operating on the single data stream using an algorithm that identifies temporal relationships.

33. The non-transitory, computer-readable medium of claim 18, wherein the processing of the heterogeneous data streams includes operating on the single data stream using an algorithm that identifies spatial relationships.

34. A non-transitory, computer-readable medium comprising a set of instructions for real-time detection of anomalies occurring in an enterprise computer network, wherein execution of said instructions by a system comprising at least one processor cause the system to:

receive a plurality of heterogeneous data streams from sources in the network, the sources including two levels, first level sources and second level sources,

wherein the first level sources include the following;

agents located at databases;

agents located at applications;

audit programs located at user workstations;

sensors located in the network; and

sensors located at access points to the network; and

wherein the second level sources include the following;

email exchanges;

instant messages;

voice mails;

documents;

database content; and

user interactions and database queries;

process the heterogeneous data streams obtained by combining at least one of the first level sources with at least one of the second level sources to identify events therein, each event being identified by at least a unique ID, a timestamp, and an event type;

correlate the processed heterogeneous data streams to form an integrated data stream comprising a plurality of identified events;

detect the existence and at least one characteristic of an anomaly in the computer network by application of a predetermined model of normalcy and one or more anomaly rules to the integrated data stream comprising the plurality of identified events; and

issue an alert based on the at least one characteristic of the anomaly, wherein the executed instructions cause the system to correlate the processed data streams by;

estimating the number or frequency of one or more event types in the processed data stream without searching the entire processed data stream; and

determining one or more temporal, spatial, or generalized associations between a plurality of events in the processed data stream.

35. A computerized system for real-time detection of anomalies occurring in an enterprise computer network, comprising:

at least one processor; and

at least one memory including program code that, when executed by the at least one processor, causes the system to;

receive a plurality of heterogeneous data streams from sources in the network, the sources including two levels, first level sources and second level sources,

wherein the first level sources include one or more selected from a group consisting of agents located at databases;

agents located at applications;

audit programs located at user workstations;

sensors located in the network; and

sensors located at access points to the network;

wherein the second level sources include one or more selected from a group consisting of data access, user behavior, computer activity and network activity; and

wherein the first level sources monitor event streams of the second level sources and generate data streams indicative of corresponding second level source activity in a uniform format;

process the heterogeneous data streams obtained by combining at least two of the first level sources to identify events therein, each event being identified by at least a unique ID, a timestamp, and an event type, wherein the processing of the heterogeneous data streams includes;

combining at least two of the first level sources into a single data stream; and

operating on the single data stream using an algorithm that identifies spatiotemporal relationships;

correlate the processed heterogeneous data streams to form an integrated data stream comprising a plurality of identified events;

detect the existence and at least one characteristic of an anomaly in the computer network by application of a predetermined model of normalcy and one or more anomaly rules to the integrated data stream comprising the plurality of identified events; and

issue an alert based on the at least one characteristic of the anomaly.

36. A non-transitory, computer-readable medium comprising a set of instructions for real-time detection of anomalies occurring in an enterprise computer network, wherein execution of said instructions by a system comprising at least one processor cause the system to:

receive a plurality of heterogeneous data streams from sources in the network, the sources including two levels, first level sources and second level sources,

wherein the first level sources include one or more selected from a group consisting of agents located at databases;

agents located at applications;

audit programs located at user workstations;

sensors located in the network; and

sensors located at access points to the network;

wherein the second level sources include one or more selected from a group consisting of data access, user behavior, computer activity and network activity; and

wherein the first level sources monitor event streams of the second level sources and generate data streams indicative of corresponding second level source activity in a uniform format;

process the heterogeneous data streams obtained by combining at least two of the first level sources to identify events therein, each event being identified by at least a unique ID, a timestamp, and an event type, wherein the processing of the heterogeneous data streams includes;

combining at least two of the first level sources into a single data stream; and

operating on the single data stream using an algorithm that identifies spatiotemporal relationships;

correlate the processed heterogeneous data streams to form an integrated data stream comprising a plurality of identified events;

detect the existence and at least one characteristic of an anomaly in the computer network by application of a predetermined model of normalcy and one or more anomaly rules to the integrated data stream comprising the plurality of identified events; and

issue an alert based on the at least one characteristic of the anomaly.

37. A system for real-time detection of anomalies occurring in a computer network, comprising:

at least one processor; and

at least one memory including program code that, when executed by the at least one processor, causes the system to;

receive a plurality of heterogeneous data streams from sources in the network, the sources including first level sources and second level sources,

wherein the first level sources include one or more selected from a group consisting of agents located at databases, agents located at applications, audit programs located at user workstations, sensors located in the network, and sensors located at access points to the network;

wherein the second level sources include event streams to be analyzed;

wherein the first level sources monitor the event streams of the second level sources and generate data streams indicative of corresponding second level source activity in a uniform format; and

wherein each of the heterogeneous data streams is obtained by combining at least two of the first level sources into a data stream;

process the heterogeneous data streams to identify events therein, each event being identified by at least a unique ID, a timestamp, and an event type;

correlate the processed heterogeneous data streams to form an integrated data stream comprising a plurality of identified events;

detect the existence and at least one characteristic of an anomaly in the computer network by application of a predetermined model of normalcy and one or more anomaly rules to the integrated data stream comprising the plurality of identified events; and

issue an alert based on the at least one characteristic of the anomaly.

38. The system of claim 37, wherein the second level sources include one or more selected from the group consisting of data access, user behavior, computer activity and network activity.