Systems and Methods with a Realtime Log Analysis Framework

• US 20180129579A1
• Filed: 10/16/2017
• Published: 05/10/2018
• Est. Priority Date: 11/10/2016
• Status: Abandoned Application

• Current Assignee: NEC Laboratories America Inc (NEC Corporation)
• Original Assignee: NEC Laboratories America Inc (NEC Corporation)
• Inventors: Debnath, Biplob, Arora, Nipun, Zhang, Hui, Jiang, Guofei, Solaimani, Mohiuddin, Gulzar, Muhammad Ali
• Application Number:
  US15/784,393
• Publication Number:
  US 20180129579A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/0706 the processing taking place on a specific hardware platform or in a specific software environment
• CPC Class Codes: G06F 11/0775 Content or structure details of the error report, e.g. specific table structure, specific error fields
• CPC Class Codes: G06F 11/0787 Storage of error reports, e.g. persistent data storage, storage using memory protection
• CPC Class Codes: G06F 11/3065 Monitoring arrangements determined by the means or processing involved in reporting the monitored data error or fault reporting or logging G06F11/0766
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

Systems and methods are disclosed for processing a stream of logged data by: creating one or more models from a set of training logs during a training phase; receiving testing data in real-time and generating anomalies using the models created during the training phase; updating the one or more models during real-time processing of a live stream of logs; and detecting a log anomaly from the live stream of logs.

Claims

1. A method for processing a stream of logged data, comprising:

creating one or more models from a set of training logs during a training phase;

receiving testing data in real-time and generating anomalies using the models created during the training phase;

updating the one or more models during real-time processing of a live stream of logs; and

detecting a log anomaly from the live stream of logs.

2. The method of claim 1, comprising performing real-time heterogeneous log anomaly detection.

3. The method of claim 1, comprising using unsupervised machine learning for log parsing and tokenization for anomaly detection

4. The method of claim 1, comprising dynamically updating distributed immutable in-memory models in a streaming application.

5. The method of claim 1, wherein the streaming application is spark streaming.

6. The method of claim 1, wherein the receiving of testing data comprises spark streaming the logs.

7. The method of claim 1, comprising extensible plug and play framework for common anomaly detection patterns such as stateless, stateful, and time-series anomaly detection.

8. The method of claim 1, comprising highlighting potential anomalies in real-time.

9. The method of claim 1, comprising detecting anomaly detection patterns including stateless, stateful, and time-series anomaly detection.

10. The method of claim 1, comprising automating log mining and management processes for administrators.

20. The method of claim 1, wherein the logs are received an Internet of Things (IOT) device, a software system, point of sales system.

11. A system for processing a stream of logged data, comprising:

a database to store one or more models created from a set of training logs during a training phase;

a processor with code for;

receiving testing data in real-time and generating anomalies using the models created during the training phase;

updating the one or more models during real-time processing of a live stream of logs; and

detecting a log anomaly from the live stream of logs.

12. The system of claim 11, comprising code for performing real-time heterogeneous log anomaly detection.

13. The system of claim 11, comprising code for using unsupervised machine learning for log parsing and tokenization for anomaly detection

14. The system of claim 11, comprising code for dynamically updating distributed immutable in-memory models in a streaming application.

15. The system of claim 11, wherein the streaming application is spark streaming.

16. The system of claim 11, wherein the receiving of testing data comprises spark streaming the logs.

17. The system of claim 11, comprising an extensible plug and play framework for common anomaly detection patterns such as stateless, stateful, and time-series anomaly detection.

18. The system of claim 11, comprising code for highlighting potential anomalies in real-time.

19. The system of claim 11, comprising code for detecting anomaly detection patterns including stateless, stateful, and time-series anomaly detection.