LOG DATA SERVICE IN A VIRTUAL ENVIRONMENT

• US 20150227598A1
• Filed: 02/13/2014
• Published: 08/13/2015
• Est. Priority Date: 02/13/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Hahn, Henry David, Vassallo, Daniel, Pan, Xiaosong, Devillard, Emmanuel Pierre, Wang, Hongqi, Azad, Muhammad Umer
• Granted Patent: US 10,133,741 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

A log data service in a virtual environment that allows customers of a compute service provider to access system, application and custom log files associated with virtual machine instances that are executing. In some embodiments, log data can be received that includes events or messages from virtual machines in a multi-tenant environment. The log data can be transformed into metric data, which can be used by the customer to generate statistics, view graphs, and generally monitor the customer'"'"'s virtual machine instances. The log data can also be stored as a service so that the customer has a central repository for which to access the log data.

Claims

1. A method of implementing a log data service in a multi-tenant environment operated by a compute service provider, comprising:

receiving log data including a plurality of events and/or messages from a virtual machine in the multi-tenant environment;

using a service of the compute service provider, searching through the log data to determine which of the plurality of events and/or messages are associated with at least one predetermined metric;

transforming the log data to metric data using results of the search; and

storing the log data in association with a customer account.

2. The method of claim 1, wherein transforming the log data to metric data includes converting a matched pattern into a count associated with a number of occurrences for the matched pattern.

3. The method of claim 1, further including monitoring the metric data and transmitting a notification if the metric data exceeds a predetermined threshold.

4. The method of claim 1, wherein receiving the log data includes receiving multiple streams of log data from different sources and associating the multiple streams with a common group.

5. The method of claim 4, further including receiving an API request, having the customer account associated therewith, to access the stored log data, the API requesting including a name of the common group.

6. The method of claim 1, wherein searching includes retrieving a predetermined pattern to be searched and matching the predetermined pattern with at least one of the plurality of events and/or messages.

7. A computer-readable storage for implementing a log data service operated by a compute service provider, comprising:

at a service of the compute service provider, receiving log data from a device;

using the compute service provider, searching the log data for a predetermined pattern and matching the predetermined pattern to one or more entries in the log data; and

transforming the log data to at least one metric using the matched patterns.

8. The computer-readable storage of claim 7, wherein the log data is associated with a customer account and further including storing, by the service, the log data in association with the customer account.

9. The computer-readable storage of claim 7, wherein transforming the log data to at least one metric includes converting the matched pattern into a count associated with a number of occurrences for the matched pattern.

10. The computer-readable storage of claim 7, wherein the predetermined pattern is a literal term.

11. The computer-readable storage of claim 7, wherein the log data is streaming data or a log file and wherein transforming the log data includes using a schema that defines how to perform the transformation.

12. The computer-readable storage of claim 7, further including monitoring the at least one metric to ensure that it is within predetermined limits.

13. The computer-readable storage of claim 7, wherein the log data is a log stream from the device which is external to the compute service provider, and further including associating the log stream with a log group that includes two or more log streams.

14. The computer-readable storage of claim 7, wherein receiving the log data includes receiving an API call that identifies a customer account, a group name associated with the log data, and events together with associated event timestamps.

15. The computer-readable storage of claim 14, further including storing the log data in association with the group name and receiving an API request to retrieve the stored log data, the API request including at least the group name together with a time range.

16. The computer-readable storage of claim 7, further including generating for display the following:

the metrics, statistics associated with the metrics, and graphs associated with the metrics.

17. The computer-readable storage of claim 7, wherein the entries in the log data include a plurality of events and/or messages together with time stamps associated with the events and/or messages.

18. A system for accessing log data in a multi-tenant environment, comprising:

a plurality of host server computers coupled together through a network to form the multi-tenant environment, at least a portion of the host server computers for executing a plurality of virtual machines associated with a customer account;

a log data service for receiving a plurality of log data streams from the virtual machines and grouping the log data streams based on type; and

a database coupled to the log data service for storing the log data streams in association with the grouping and in association with the customer account.

19. The system of claim 18, wherein the log data service converts the log data streams into metrics.

20. The system of claim 19, wherein the metrics are associated with predetermined patterns or messages within the log data streams.

21. The system of claim 18, further including an agent, which is a subcomponent of an operating system running in the virtual machine, for transmitting at least one of the plurality of data streams.

22. The system of claim 18, wherein the log data streams are accessible by a customer of the multi-tenant environment through one or more API calls.