Method and system for creating enriched log data

• US 9,501,345 B1
• Filed: 12/23/2013
• Issued: 11/22/2016
• Est. Priority Date: 12/23/2013
• Status: Active Grant

• Current Assignee: Intuit, Inc.
• Original Assignee: Intuit, Inc.
• Inventors: Lietz, M. Shannon, Cabrera, Luis Felipe
• Primary Examiner(s): Chu, Gabriel
• Application Number:
  US14/139,449
• Time in Patent Office: 1,065 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0784 Routing of error reports, e.g. with a specific transmission path or data flow
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 17/40 Data acquisition and logging for input to computer G06F3/00

Abstract

Access to first log data from a first log data source and second log data from a second log data source is obtained. Trigger event log data is defined and the second log data from the second log data source is monitored to detect the defined trigger event log data in the second log data. If the defined trigger event log data is detected in the second log data from the second log data source, the detected trigger event log data in the second log data from the second log data source is correlated with the first log data from the first log data source, and/or at least part of the second log data from the second log data source is inserted into the first log data from the first log data source.

Claims

1. A system for creating enriched log data comprising:

at least one processor; and

at least one memory coupled to the at least one processor, the at least one memory having stored therein instructions which when executed by any set of the at least one processors, perform a process for creating enriched log data, the process for creating enriched log data including;

obtaining access to first log data of a first log data source, the first log data source being a first virtual machine instance;

obtaining access to distinct log data of a plurality of distinct log data sources, each of the plurality of log data sources being of different virtual machine instances distinct from virtual machine instances of each other log data source of the plurality of log data sources;

defining at least two virtual machine instance trigger events, the defined at least two virtual machine instance trigger events including a first trigger event being a security requirement setting event and a second trigger event being the creation or changing of access control lists;

defining, responsive to the at least two virtual machine instance trigger events being defined, trigger event log entry data associated with the defined at least two virtual machine instance trigger events;

monitoring, responsive to the at least two virtual machine instance trigger event being defined, log data of each log data source of the plurality of log data sources;

detecting, as a result of the monitoring, trigger event log entry data in log data of a second log data source of the plurality of log data sources;

inserting, into the first log data of the first log data source, the trigger event log data of the second log data representing the trigger event of a second virtual machine instance occurring;

detecting, as a result of the monitoring, trigger event log entry data in log data of a third log data source of the plurality of log data sources; and

inserting, into the first log data of the first log data source, the trigger event log data of the third log data representing the trigger event of the third virtual machine instance occurring.

2. The system for creating enriched log data of claim 1 wherein at least one of the first log data source and the second log data source is selected from the group of log data sources consisting of:

a virtual machine instance;

a virtual server instance;

a data store instance;

any instance in a cloud computing environment;

a non-virtual computing system;

a non-virtual server system;

a non-virtual data store;

a cloud computing environment access system;

part of a mobile device;

part of a remote sensor;

part of a laptop computing system;

part of a desktop computing system;

part of a point-of-sale computing system;

part of an ATM;

a workstation;

a storage cluster;

a switching system;

a router;

any communications system;

any form of proxy system;

a gateway system;

a firewall system;

a load balancing system;

an application;

a service; and

any virtual or non-virtual asset associated with one or more cloud computing environments.

3. The system for creating enriched log data of claim 1 wherein the at least two defined trigger events are selected from the group of trigger events consisting of:

account creation events;

the creation and or removal of communications channels;

performance metrics analysis or results;

peer communications and peer communication attempts;

user interaction and log data source availability;

failed access attempts;

successful access attempts

the abnormal termination of an application;

the increase of error rates in response to customer requests;

the detection that an Internet networking event that may impact performance is happening;

the alerting by a third party of a denial of service attack;

a change in the topology or layout used in the deployment of an application; and

information that may alter the security characteristics of an application.

4. The system for creating enriched log data of claim 1 wherein correlating the detected trigger event log entry data in the second log data from the second log data source with the first log data from the first log data source includes generating linking data, the linking data indicating the first log data should be reviewed with reference to the detected trigger event log entry data in the second log data.

5. The system for creating enriched log data of claim 4 wherein the linking data is inserted into the first log data from the first log data source manually.

6. The system for creating enriched log data of claim 4 wherein the linking data is inserted into the first log data from the first log data source semi-automatically subject to review and/or approval.

7. The system for creating enriched log data of claim 4 wherein the linking data is inserted into the first log data from the first log data source automatically.

8. A system for creating enriched log data comprising:

a first log data source, the first log data source generating first log data;

a plurality of log data sources, the plurality of log data sources generating log data independent of and external to log data of any other log data source; and

a computing system, the computing system including at least one processor and at least one memory coupled to the at least one processor, the at least one memory having stored therein instructions which when executed by any set of the one or more processors, perform a process for creating enriched log data, the process for creating enriched log data including;

obtaining access to the first log data from the first log data source;

obtaining access to the log data of the plurality of log data sources;

defining at least two virtual machine instance trigger events, the defined at least two virtual machine instance trigger events including a first trigger event being a security requirement setting event and a second trigger event being the creation or changing of access control lists;

defining, responsive to the at least two virtual machine instance trigger events being defined, trigger event log entry data associated with the defined at least two virtual machine instance trigger events;

monitoring, responsive to the at least two virtual machine instance trigger event being defined, log data of each log data source of the plurality of log data sources;

detecting, as a result of the monitoring, trigger event log entry data in log data of a second log data source of the plurality of log data sources;

inserting, into the first log data of the first log data source, the trigger event log data of the second log data representing the trigger event of a second virtual machine instance occurring;

detecting, as a result of the monitoring, trigger event log entry data in log data of a third log data source of the plurality of log data sources; and

inserting, into the first log data of the first log data source, the trigger event log data of the third log data representing the trigger event of the third virtual machine instance occurring.

9. The system for creating enriched log data of claim 8 wherein at least one of the first log data source and the second log data source is selected from the group of log data sources consisting of:

a virtual machine instance;

a virtual server instance;

a data store instance;

any instance in a cloud computing environment;

a non-virtual computing system;

a non-virtual server system;

a non-virtual data store;

a cloud computing environment access system;

part of a mobile device;

part of a remote sensor;

part of a laptop computing system;

part of a desktop computing system;

part of a point-of-sale computing system;

part of an ATM;

a workstation;

a storage cluster;

a switching system;

a router;

any communications system;

any form of proxy system;

a gateway system;

a firewall system;

a load balancing system;

an application;

a service; and

any virtual or non-virtual asset associated with one or more cloud computing environments.

10. The system for creating enriched log data of claim 8 wherein the at least two defined trigger events include one or more trigger events selected from the group of trigger events consisting of:

account creation events;

the creation and or removal of communications channels;

performance metrics analysis or results;

peer communications and peer communication attempts;

user interaction and log data source availability;

failed access attempts;

successful access attempts

the abnormal termination of an application;

the increase of error rates in response to customer requests;

the detection that an Internet networking event that may impact performance is happening;

the alerting by a third party of a denial of service attack;

a change in the topology or layout used in the deployment of an application; and

information that may alter the security characteristics of an application.

11. The system for creating enriched log data of claim 8 wherein correlating the detected trigger event log entry data in the second log data from the second log data source with the first log data from the first log data source includes generating linking data, the linking data indicating the first log data should be reviewed with reference to the detected trigger event log entry data in the second log data from the second log data source.

12. The system for creating enriched log data of claim 11 wherein the linking data is inserted into the first log data from the first log data source manually.

13. The system for creating enriched log data of claim 11 wherein the linking data is inserted into the first log data from the first log data source semi-automatically subject to review and/or approval.

14. The system for creating enriched log data of claim 11 wherein the linking data is inserted into the first log data from the first log data source automatically.

15. A system for creating enriched log data comprising:

at least one processor; and

at least one memory coupled to the at least one processor, the at least one memory having stored therein instructions which when executed by any set of the one or more processors, perform a process for creating enriched log data, the process for creating enriched log data including;

obtaining access to first log data of a first log data source, the first log data source being a first virtual machine instance;

obtaining access to distinct log data of a plurality of distinct log data sources, each of the plurality of log data sources being of different virtual machine instances distinct from virtual machine instances of each other log data source of the plurality of log data sources;

defining at least two virtual machine instance trigger events, the defined at least two virtual machine instance trigger events including a first trigger event being a removal or addition of security requirements and a second trigger event being the creation or changing of access control lists;

defining, responsive to the at least two virtual machine instance trigger events being defined, trigger event log entry data associated with the defined at least two virtual machine instance trigger events;

monitoring, responsive to the at least two virtual machine instance trigger event being defined, log data of each log data source of the plurality of log data sources;

detecting, as a result of the monitoring, trigger event log entry data in log data of a second log data source of the plurality of log data sources;

inserting, into the first log data of the first log data source, the trigger event log data of the second log data representing the trigger event of a second virtual machine instance occurring;

detecting, as a result of the monitoring, trigger event log entry data in log data of a third log data source of the plurality of log data sources; and

inserting, into the first log data of the first log data source, the trigger event log data of the third log data representing the trigger event of the third virtual machine instance occurring.

16. A system for creating enriched log data comprising:

a first log data source, the first log data source generating first log data;

a second log data source, the second log data source generating second log data, the second log data source being distinct from the first log data source such that the second log data is external log data with respect to the first log data; and

a computing system, the computing system including at least one processor and at least one memory coupled to the at least one processor, the at least one memory having stored therein instructions which when executed by any set of the one or more processors, perform a process for creating enriched log data, the process for creating enriched log data including;

obtaining access to first log data of a first log data source, the first log data source being a first virtual machine instance;

obtaining access to distinct log data of a plurality of distinct log data sources, each of the plurality of log data sources being of different virtual machine instances distinct from virtual machine instances of each other log data source of the plurality of log data sources;

defining at least two virtual machine instance trigger events, the defined at least two virtual machine instance trigger events including a first trigger event being a removal or addition of security requirements and a second trigger event being the creation or changing of access control lists;

defining, responsive to the at least two virtual machine instance trigger events being defined, trigger event log entry data associated with the defined at least two virtual machine instance trigger events;

monitoring, responsive to the at least two virtual machine instance trigger event being defined, log data of each log data source of the plurality of log data sources;

detecting, as a result of the monitoring, trigger event log entry data in log data of a second log data source of the plurality of log data sources;

inserting, into the first log data of the first log data source, the trigger event log data of the second log data representing the trigger event of a second virtual machine instance occurring;

detecting, as a result of the monitoring, trigger event log entry data in log data of a third log data source of the plurality of log data sources; and

inserting, into the first log data of the first log data source, the trigger event log data of the third log data representing the trigger event of the third virtual machine instance occurring.