Virtual machine instance data aggregation

• US 10,733,002 B1
• Filed: 06/28/2016
• Issued: 08/04/2020
• Est. Priority Date: 06/28/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Mackay, Eric, Zahabi, Sammy, Neogy, Amit
• Primary Examiner(s): Channavajjala, Srirama
• Application Number:
  US15/195,773
• Time in Patent Office: 1,498 Days
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/188 Virtual file systems
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 16/24556 Aggregation; Duplicate elimination
• CPC Class Codes: G06F 16/316 Indexing structures
• CPC Class Codes: G06F 2009/45583 Memory management, e.g. access or allocation
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects

Abstract

A service provider launches an index analysis computing system instance to evaluate indexes generated by a virtual computing system server to identify events encountered by the server. In response to a notification from an index subject indicating presence of a new index for the server, the index analysis computing system instance obtains the index from a datastore and evaluates the index to identify a log for the server that specifies data that can be used to identify the events. The index analysis computing system instance obtains, from a second datastore, the identified log and used the log to identify the events. The index analysis computing system instance provides event data corresponding to the events to a data processing datastore where the data can be aggregated and processed.

Claims

1. A system, comprising:

one or more processors; and

memory that stores computer-executable instructions that, as a result of being executed, cause the one or more processors to;

obtain, in response to a notification indicating a presence of a data log index for a virtual computer system server;

evaluate, in response to the notification, the data log index to identify a first data log generated by a virtual computing system server and associated with a virtual computing instance;

obtain, from a datastore that stores data logs, the first data log;

identify, by a computing system instance performing index analysis, one or more events indicated in the first data log and a second data log that store processor state information associated with the virtual computing system server and includes information associated with the one or more events, where the second data log is identified based at least in part on the first data log;

in response to identifying the second data log, obtain the second data log associated with the virtual computer system server;

generate event data based at least in part on processor state information obtained from the second data log associated with the virtual computer system server corresponding to the one or more events;

cause the event data to be processed;

based at least in part on a result of processing the event data, cause a report to be generated, the report containing information indicating at least one event of the one or more events and an associated processor state of the processor state information; and

a datastore that processes data by at least;

obtaining the event data from the index analysis computing system instance;

obtaining the processor state information from a data log analysis computing system instance;

aggregating the event data and the processor state information to generate aggregated data for the virtual computing system server; and

generating, using the aggregated data for the virtual computing system server, a set of reports specifying the aggregated data for the virtual computing system server, wherein the report is a member of the set of reports.

2. The system of claim 1, wherein the index analysis computing system instance and the data log analysis computing system instance are software container instances that execute code and terminate upon execution of the code.

3. The system of claim 1, wherein the data log analysis computing system instance is initiated in response to detecting the presence of the data log index for the virtual computer system server.

4. The system of claim 1, wherein the event includes a domain crash of the virtual computing system server.

5. A computer-implemented method, comprising:

obtaining an index for a virtual computing system server from a first datastore;

evaluating, by a computing system instance responsible for index analysis, the index to identify a first log for the virtual computing system server, the first log specifying an event associated with the virtual computing system server executing a virtual machine instance and a second log specifying processor state information of the virtual computing system server corresponding to the event;

obtaining, from a second datastore, the second log;

identifying, using the first log, the event;

generating event data based at least in part on the second log and the event identified using the first log, where the event data indicates a processor state of the virtual computing system server corresponding to the event obtained from the second log;

transmitting the event data to a third datastore to be processed; and

causing the third datastore to generate a report based at least in part on the event data and the processor state of the virtual computing system server indicated in the event data, the report associating the event with the processor state of the virtual computing system server by at least aggregating the event data included in the first log with the processor state information included in the second log.

6. The computer-implemented method of claim 5, wherein:

the first datastore transmits a notification in response to receiving the index from the virtual computing system server, the notification indicating presence of the index; and

the index is obtained in response to the notification.

7. The computer-implemented method of claim 5, wherein the method further comprises transmitting a second notification to an administrator of the virtual computing system server to indicate availability of the event data.

8. The computer-implemented method of claim 5, wherein the event data includes at least a timestamp for the event, an identifier for the virtual machine instance, and hardware information for the virtual computing system server.

9. The computer-implemented method of claim 5, wherein the report further comprises visual information associated with the event data.

10. The computer-implemented method of claim 5, wherein the method further comprises converting the event data to a comma-separated value format.

11. The computer-implemented method of claim 5, wherein the method further comprises:

identifying, based at least in part on the event data, one or more tasks to be performed by a second computing system instance to obtain additional information corresponding to the event; and

providing the one or more tasks to the second computing system instance.

12. The computer-implemented method of claim 5, wherein the event includes a domain crash event for the virtual computing system server.

13. One or more non-transitory computer-readable storage media storing thereon executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to:

obtain a first log from a datastore, the first log generated by a virtual computing system server executing a virtual machine instance;

parse through the first log to obtain data corresponding to one or more events;

identify, based at least in part on the one or more events, one or more tasks performed by a first computing system instance to obtain a second log storing processor state information associated with the virtual computing system server including processor state data corresponding to the one or more events;

provide the one or more tasks to the first computing system instance thereby causing the first computing system instance to obtain the processor state data corresponding to the one or more events;

obtain, from the first computing system instance, the second log;

generate event data based at least in part on the second log and the first log; and

cause one or more reports to be generated, using the event data, the one or more reports indicating the processor state data corresponding to the one or more events, where the one or more reports are generated by a second computer system instance by at least aggregating the data associated with the one or more events in the first log with data associated with the processor state data included in the second log.

14. The one or more non-transitory computer-readable storage media of claim 13, wherein the one or more events encountered by the virtual computing system server include state transition events corresponding to the virtual computing system server.

15. The one or more non-transitory computer-readable storage media of claim 13, wherein the instructions further include instructions that, as a result of being executed by the one or more processors, cause the computer system to receive a notification indicating presence of an index for the virtual computing system server within the datastore.

16. The one or more non-transitory computer-readable storage media of claim 15, wherein the datastore transmits the notification in response to receiving the index from the virtual computing system server.

17. The one or more non-transitory computer-readable storage media of claim 15, wherein the instructions further include instructions that, as a result of being executed by the one or more processors, cause the computer system to receive the notification indicating presence of the index for the virtual computing system server within the datastore.

18. The one or more non-transitory computer-readable storage media of claim 13, wherein the instructions further include instructions that, as a result of being executed by the one or more processors, cause the computer system to store the data corresponding to the one or more events within a datastore responsible for maintaining diagnostic information.

19. The one or more non-transitory computer-readable storage media of claim 13, wherein the one or more reports further include a visual representation of the event data.

20. The one or more non-transitory computer-readable storage media of claim 13, wherein the instructions that cause the computer system to generate event data based at least in part on the second log and the first log further include instructions that, as a result of being executed by the one or more processors, cause the computer system to generate the event data to indicate a correspondence between a first event of the one or more events and a particular processor state of the processor state data.