Automated Anomaly Detection Service on Heterogeneous Log Streams

• US 20170139806A1
• Filed: 11/15/2016
• Published: 05/18/2017
• Est. Priority Date: 11/18/2015
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Laboratories America Inc (NEC Corporation)
• Inventors: Xu, Jianwu, Zhang, Hui, Jiang, Guofei, Debnath, Biplob, Arora, Nipun
• Granted Patent: US 9,928,155 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/0706 the processing taking place on a specific hardware platform or in a specific software environment
• CPC Class Codes: G06F 11/0766 Error or fault reporting or storing
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 11/3636 by tracing the execution of the program

Abstract

Systems and methods are disclosed for handling log data from one or more applications, sensors or instruments by receiving heterogeneous logs from arbitrary/unknown systems or applications; generating regular expression patterns from the heterogeneous log sources using machine learning and extracting a log pattern therefrom; generating models and profiles from training logs based on different conditions and updating a global model database storing all models generated over time; tokenizing raw log messages from one or more applications, sensors or instruments running a production system; transforming incoming tokenized streams are into data-objects for anomaly detection and forwarding of log messages to various anomaly detectors; and generating an anomaly alert from the one or more applications, sensors or instruments running a production system.

Claims

1. A method for handling log data from one or more applications, sensors or instruments, comprising:

receiving heterogeneous logs from arbitrary/unknown systems or applications;

generating regular expression patterns from the heterogeneous log sources using machine learning and extracting a log pattern therefrom;

generating models and profiles from training logs based on different conditions and updating a global model database storing all models generated over time;

tokenizing raw log messages from one or more applications, sensors or instruments running a production system;

transforming incoming tokenized streams are into data-objects for anomaly detection and forwarding of log messages to various anomaly detectors; and

generating an anomaly alert from the one or more applications, sensors or instruments running a production system.

2. The method of claim 1, wherein each log message comprises a time stamp and text content.

3. The method of claim 1, comprising dividing data into training logs and testing logs.

4. The method of claim 1, wherein each anomaly violation comprises a semantic, syntactic or statistical violation.

5. The method of claim 1, comprising generating a visual alert to a user.

6. The method of claim 1, wherein the log pattern extraction generates a regular expression for the incoming logs using unsupervised learning.

7. The method of claim 1, wherein the log patterns have variable fields with a wildcard pattern and each field has a key name attributed to known patterns and to unknown fields with generic names.

8. The method of claim 1, wherein generating the model comprises creating training profiles based on syntactic and semantic analysis.

9. The method of claim 1, comprising generating a Content Profile Model to create a frequency profile of various values for each key in a pattern or regular expression of a category of logs.

10. The method of claim 1, comprising generating a Sequence Order Model that extracts sequential ordering relationships between patterns.

11. The method of claim 1, comprising generating a Volume Model that maintains a frequency distribution of logs of each pattern and detecting unusual spikes of certain patterns and report the spikes as alerts.

12. The method of claim 1, comprising performing distributed learning and incremental learning dependent on the model.

13. The method of claim 12, wherein the global model database provides an interface to assist, and to allow for a learning service which can maintain trained profiles in a database.

14. The method of claim 1, comprising selecting a model based on timestamp, sources, model category or based on complex queries including join, group, aggregate for grouping model categories, and aggregating models across different time ranges.

15. The method of claim 1, comprising performing model-updates to support incremental or distributed learning processes, wherein an update of the model depends on a learning algorithm and a model profile.

16. The method of claim 1, comprising forming a model database with a hierarchical schema and each model includes <

TimeStamp, Time Range, Category, Source, Model>

.

17. The method of claim 1, comprising transforming incoming data streams into data-objects for common anomaly detection tasks.

18. The method of claim 17, comprising pushing transformed data objects to a publish/subscribe module, and subscribing to the module for anomaly detection.

19. The method of claim 18, comprising applying violation or anomaly checking to check if an incoming transformed log-stream indicates an anomaly in the system.

20. A system, comprising

a processor;

an Internet-of-Thing (IoT) sensor providing data to the processor; and

computer readable code executed by the processor to;

receiving heterogeneous logs from arbitrary/unknown systems or applications;

generating regular expression patterns from the heterogeneous log sources using machine learning and extracting a log pattern therefrom;

generating models and profiles from training logs based on different conditions and updating a global model database storing all models generated over time;

tokenizing raw log messages from one or more applications, sensors or instruments running a production system;

transforming incoming tokenized streams are into data-objects for anomaly detection and forwarding of log messages to various anomaly detectors; and

generating an anomaly alert from one or more applications, sensors or instruments running a production system.