Log data service in a virtual environment

• US 10,133,741 B2
• Filed: 02/13/2014
• Issued: 11/20/2018
• Est. Priority Date: 02/13/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Hahn, Henry David, Vassallo, Daniel, Pan, Xiaosong, Devillard, Emmanuel Pierre, Wang, Hongqi, Azad, Muhammad Umer
• Primary Examiner(s): Gortayo, Dangelino N
• Application Number:
  US14/180,217
• Publication Number:
  US 20150227598A1
• Time in Patent Office: 1,741 Days
• Field of Search: 707722, 707758, 707648, 707737, 707672
• CPC Class Codes: G06F 11/3006 where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems multiprogramming arrangements G06F9/46; allocation of resources G06F9/50
• CPC Class Codes: G06F 11/3476 Data logging G06F11/14, G06F11/2205 take precedence
• CPC Class Codes: G06F 16/11 File system administration, e.g. details of archiving or snapshots file system backup G06F11/14
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

A log data service in a virtual environment that allows customers of a compute service provider to access system, application and custom log files associated with virtual machine instances that are executing. In some embodiments, log data can be received that includes events or messages from virtual machines in a multi-tenant environment. The log data can be transformed into metric data, which can be used by the customer to generate statistics, view graphs, and generally monitor the customer'"'"'s virtual machine instances. The log data can also be stored as a service so that the customer has a central repository for which to access the log data.

Claims

1. A method of implementing a log data service in a multi-tenant environment operated by a compute service provider, comprising:

receiving log data including a plurality of events and/or messages from a virtual machine in the multi-tenant environment, wherein the received log data is from the virtual machine executing an application and an operating system and the application transmits the log data directly to a log data service;

using a service of the compute service provider, searching through the log data to determine which of the plurality of events and/or messages are associated with at least one predetermined pattern based on a policy document provided by a customer;

transforming the log data based on a schema provided in the policy document that defines how to transform the log data into metric data in correspondence with the predetermined pattern using results of the search, the transforming including matching the predetermined pattern in the log data and using the matching to generate the metric data;

transmitting the metric data to a service that allows the metric data to be monitored; and

storing the log data in association with the customer account in the log data service.

2. The method of claim 1, wherein transforming the log data to metric data includes converting a matched pattern into a count associated with a number of occurrences for the matched pattern.

3. The method of claim 1, further including monitoring the metric data and transmitting a notification if the metric data exceeds a predetermined threshold.

4. The method of claim 1, wherein receiving the log data includes receiving multiple streams of log data from different sources and associating the multiple streams with a common group.

5. The method of claim 4, further including receiving an API request, having the customer account associated therewith, to access the stored log data, the API requesting including a name of the common group.

6. The method of claim 1, wherein the policy further indicates how the log data is to be transformed into metric data.

7. The method of claim 1, wherein the application and a logging agent coupled to the application each transmit log data to the log data service.

8. A computer-readable storage for implementing a log data service operated by a compute service provider, comprising:

at a service of the compute service provider, receiving log data from a device, wherein the device includes a virtual machine instance running an application and an operating system, and wherein the application generates the log data and periodically transforms it into a format needed for an Application Program Interface (API) communication;

using the compute service provider, searching the log data, received from the application, for a predetermined pattern and matching the predetermined pattern to one or more entries in the log data, the predetermined pattern being provided by a policy document associated with a customer of the compute service environment and stored in a customer account of the compute service provider;

selectively transforming the log data based on a schema provided in the policy document to at least one metric using the predetermined patterns, wherein the transforming is accomplished based on the policy;

transmitting the at least one metric to a service in the compute service provider that allows metrics to be monitored; and

storing the log data within the compute service provider so that the log data is accessible to customers.

9. The computer-readable storage of claim 8, further including storing, by the service, the log data in association with the customer account.

10. The computer-readable storage of claim 8, wherein transforming the log data to at least one metric includes converting a matched pattern into a count associated with a number of occurrences for the matched pattern.

11. The computer-readable storage of claim 8, wherein the predetermined pattern is a literal term.

12. The computer-readable storage of claim 8, wherein the log data is streaming data or a log file.

13. The computer-readable storage of claim 8, further including monitoring the at least one metric to ensure that it is within predetermined limits.

14. The computer-readable storage of claim 8, wherein the log data is a log stream from the device which is external to the compute service provider, and further including associating the log stream with a log group that includes two or more log streams.

15. The computer-readable storage of claim 8, wherein receiving the log data includes receiving an API call that identifies a customer account, a group name associated with the log data, and events together with associated event timestamps.

16. The computer-readable storage of claim 15, further including storing the log data in association with the group name and receiving an API request to retrieve the stored log data, the API request including at least the group name together with a time range.

17. The computer-readable storage of claim 8, further including generating for display the following:

the metrics, statistics associated with the metrics, and graphs associated with the metrics.

18. The computer-readable storage of claim 8, wherein the policy further indicates how the log data is to be transformed into metric data.

19. The computer-readable storage of claim 8, wherein the application transmits the log data directly to the log data service without using a logging agent.

20. A system for accessing log data in a multi-tenant environment, comprising:

a plurality of host server computers coupled together through a network to form the multi-tenant environment, at least a portion of the host server computers for executing a plurality of virtual machines associated with a customer account including a policy document specified by a customer, the policy document including a schema defining how to transform log data into metric data, wherein at least one of the plurality of virtual machines includes an application executing using support of an operating system on a first virtual machine and the application periodically reformats the log data and transmits the log data as a data stream to a log data service executing within the operating system, wherein the log data service stores the log data in association with the customer;

the log data service for receiving a plurality of log data streams from the virtual machines including the data stream from the first virtual machine and grouping the log data streams based on type, wherein the log data service selectively converts the log data streams into metrics based on the policy document using a schema stored in the customer account, wherein the policy document includes one or more patterns to match in the log data streams and defines how to perform the conversion and wherein the log data service is for transmitting the metrics so as to allow the metrics to be monitored by the customer; and

a database coupled to the log data service for storing the log data streams in association with the grouping and in association with the customer account.

21. The system of claim 20, wherein either the application or a logging agent coupled to the application may transmit log data directly to the log data service.