Virtual machine instance data aggregation based on work definition metadata

• US 10,489,179 B1
• Filed: 06/28/2016
• Issued: 11/26/2019
• Est. Priority Date: 06/28/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Mackay, Eric, Zahabi, Sammy, Neogy, Amit
• Primary Examiner(s): Chau, Dung K
• Application Number:
  US15/195,857
• Time in Patent Office: 1,246 Days
• Field of Search: 707741
• CPC Class Codes: G06F 11/301 where the computing system is a virtual computing platform, e.g. logically partitioned systems virtual machines G06F9/45533; logical partitioning of resources G06F9/5077
• CPC Class Codes: G06F 11/3055 Monitoring arrangements for monitoring the status of the computing system or of the computing system component, e.g. monitoring if the computing system is on, off, available, not available error or fault processing without redundancy G06F11/0703; error detection or correction by redundancy in data representation G06F11/08; error detection or correction by redundancy in operation G06F11/14; error detection or correction by redundancy in hardware G06F11/16
• CPC Class Codes: G06F 11/3082 the data filtering being achieved by aggregating or compressing the monitored data
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/164 File meta data generation
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 2009/4557 Distribution of virtual machine instances; Migration and load balancing
• CPC Class Codes: G06F 2009/45583 Memory management, e.g. access or allocation
• CPC Class Codes: G06F 2009/45591 Monitoring or debugging support
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects

Abstract

A service provider launches an index analysis computing system instance to evaluate logs generated by a virtual computing system server to identify events encountered by the server. The index analysis computing system instance obtains, from a datastore, a data log for the virtual computing system server and uses the log to identify the events. The index analysis computing system instance identifies, based on data from the log associated with the events, tasks to be performed to obtain processor state data for the server. The index analysis computing system instance generates, based on the identified tasks, a metadata for a file, where the metadata represents the tasks to be performed. This file is transmitted to a scheduler computing system instance which can identify the tasks based on the metadata for the file.

Claims

1. A system, comprising:

one or more processors; and

memory that stores computer-executable instructions that, if executed, cause the system to;

implement an index analysis computing system instance that;

obtains, from a virtual machine instance data log datastore, a data log for a virtual computing system server, the data log specifying data associated with one or more events encountered by the virtual computing system server;

identifies, based at least in part on the data associated with the one or more events, one or more tasks to be performed to obtain processor state data for the virtual computing system server associated with the one or more events;

generates, based at least in part on the one or more tasks to be performed, a metadata for a file, the metadata representing the one or more tasks;

stores the file within a second datastore; and

updates an index of the datastore to specify the metadata for the file; and

implement a scheduler computing system instance that;

obtains the index of the second datastore;

identifies the metadata for the file;

decodes the metadata for the file to identify the one or more tasks to be performed;

generates, based at least in part on the one or more tasks to be performed, a task definitions file specifying the one or more tasks to be performed; and

provides the task definitions file so that one or more data logs can be obtained to perform the one or more tasks.

2. The system of claim 1, wherein the task definitions file indicates a time at which the one or more tasks are to be performed and information required to obtain the one or more data logs.

3. The system of claim 1, wherein the scheduler computing system instance further transmits the task definitions file to a diagnostics notification subject to which the data log analysis computing system instance is subscribed to provide the task definitions file to the data log analysis computing system instance.

4. The system of claim 1, wherein the computer-executable instructions further cause the system to implement an analysis computing system instance that:

obtains the task definitions file;

determines the one or more tasks to be performed;

accesses the virtual machine instance data log datastore to obtain the one or more data logs;

performs the one or more tasks to generate processor state information for the virtual computing system server; and

transmits the processor state information for the virtual computing system server to a data processing datastore to be processed.

5. A computer-implemented method, comprising:

under the control of one or more computer systems configured with executable instructions,

obtaining a data log for a virtual computing system server, the data log specifying data associated with one or more events encountered by the virtual computing system server;

identifying, based at least in part on the data associated with the one or more events, one or more tasks to be performed to obtain processor state data for the virtual computing system server associated with the one or more events;

generating, based at least in part on the one or more tasks to be performed, a metadata for a file, the metadata representing the one or more tasks;

storing the file;

identifying the one or more tasks based at least in part on the metadata for the file; and

performing the one or more tasks.

6. The computer-implemented method of claim 5, wherein the file is a null file having no data therein.

7. The computer-implemented method of claim 5, wherein the metadata for the file further represents one or more conditions for which the one or more tasks are to be performed.

8. The computer-implemented method of claim 5, wherein the data log for the virtual computing system server is obtained in response to a notification indicating presence of an index for the virtual computing system server that is used to identify the data log.

9. The computer-implemented method of claim 5, wherein the method further comprises updating an index of a datastore to specify the metadata for the file to enable identification of the one or more tasks based at least in part on the metadata for the file.

10. The computer-implemented method of claim 5, wherein the one or more tasks include accessing a datastore to obtain processor state data logs to obtain the processor state data for the virtual computing system server associated with the one or more events.

11. The computer-implemented method of claim 5, wherein the metadata for the file further represents information associated with the one or more events to cause data generated as a result of performance of the one or more tasks to be aggregated with the data associated with one or more events.

12. The computer-implemented method of claim 5, wherein the method further comprises specifying, within a datastore usable to decode the metadata, the one or more tasks and the metadata generated based at least in part on the one or more tasks.

13. A non-transitory computer-readable storage medium having stored thereon executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to at least:

obtain a data log for a virtual computing system server;

identify, based at least in part on data associated with one or more events specified in the data log, one or more tasks to be performed associated with the one or more events;

create a file having a metadata based at least in part on the one or more tasks to be performed, the metadata representing the one or more tasks; and

generate a task definitions file based at least in part on the one or more tasks represented in the metadata for the file.

14. The non-transitory computer-readable storage medium of claim 13, wherein the instructions further cause the computer system to:

receive a notification indicating presence of an index for the virtual computing system server; and

utilize the index for the virtual computing system server to determine a storage location of the data log for the virtual computing system server.

15. The non-transitory computer-readable storage medium of claim 14, wherein a datastore wherein the index is stored transmits the notification in response to receiving the index from the virtual computing system server.

16. The non-transitory computer-readable storage medium of claim 13, wherein the metadata for the file is a filename of the file.

17. The non-transitory computer-readable storage medium of claim 13, wherein the metadata further represents information associated with the one or more events to cause data generated as a result of performance of the one or more tasks to be processed with the data associated with one or more events.

18. The non-transitory computer-readable storage medium of claim 13, wherein the metadata further represents one or more conditions for which the one or more tasks can be performed by a data log analysis computing system instance that obtains data logs to perform the one or more tasks.

19. The non-transitory computer-readable storage medium of claim 13, wherein the one or more tasks include accessing a datastore to obtain one or more processor state data logs from a datastore to obtain processor state data for the virtual computing system server associated with the one or more events.

20. The non-transitory computer-readable storage medium of claim 13, wherein the instructions further cause the computer system to update a datastore that is used to decode the metadata to specify that the metadata corresponds to the one or more tasks.