Retrieving log data from metric data

• US 10,664,535 B1
• Filed: 02/02/2015
• Issued: 05/26/2020
• Est. Priority Date: 02/02/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Hahn, Henry David
• Primary Examiner(s): Corrielus, Jean M
• Application Number:
  US14/611,787
• Time in Patent Office: 1,940 Days
• Field of Search: 707722
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

A log data service is described for a multi-tenant environment that allows customers to access system, application and custom log files associated with virtual machine instances that are executing. In some embodiments, log data can be received that includes events or messages from virtual machines in the multi-tenant environment. The log data can be transformed into metric data, which can be used to generate statistics, graphs, and generally monitor the customer'"'"'s virtual machine instances. Once a customer has started monitoring log files, the customer can be notified that an actionable condition can exist, such as an alarm condition wherein metrics exceeded acceptable limits. A customer can interactively select a timeframe of a metric and request that the log data that generated the metric be presented on the display. An automated system retrieves the source-level log data in real-time upon receiving a request from a user.

Claims

1. A method of retrieving source-level log data associated with metric data, comprising:

generating metric data from associated source-level log data, using a metric filter, to display as a graph on a client device, wherein the metric filter comprises a pattern of an error message;

receiving a user input to view the source-level log data associated with the graph of the metric data, wherein the user input includes at least start and end times associated with the source-level log data so that the user selects a period of time associated with the source-level log data;

retrieving metadata, wherein the metadata comprises the metric filter associated with the graph of the metric data and one or more identifiers of the source-level log data, wherein the metadata is located on a server computer in a compute service provider environment;

receiving a request from a user interface (UI) script to extract the source-level log data satisfying the metric filter over the selected time period;

searching log data using the metric filter and the start and end times;

retrieving the source-level log data based at least in part on the searching; and

transmitting the retrieved source-level log data to the client device for display to the user.

2. The method of claim 1, wherein the extracting the source-level log data includes using the pattern from the metric filter to search in the log data in order to determine events matching the metric filter.

3. The method of claim 1, wherein the source-level log data is associated with multiple log streams and retrieving the metadata includes retrieving multiple metric filters.

4. One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause the processors to perform operations comprising:

generating metric data from associated source-level log data, using a metric filter, to display as a graph on a client device, wherein the metric filter comprises a pattern of an error message;

receiving a user input to view the source-level log data associated with the graph of the metric data, wherein the user input includes at least start and end times associated with the source-level log data so that the user selects a period of time associated with the source-level log data;

retrieving metadata, wherein the metadata comprises the metric filter associated with the graph of the metric data and one or more identifiers of the source-level log data, wherein the metadata is located on a server computer in a compute service provider environment;

receiving a request from a user interface (UI) script to extract the source-level log data satisfying the metric filter over the selected time period;

searching stored log data using the metric filter and the start and end times;

retrieving the source-level log data based at least in part on the search; and

transmitting the retrieved source-level log data to the client device for display to the user.

5. The one or more non-transitory computer-readable media of claim 4, wherein the operations further comprise transmitting the metric data from the server computer to the client device.

6. The one or more non-transitory computer-readable media of claim 4, wherein the start and end times included in the user input are graphically selected on the graph of the metric data.

7. The one or more non-transitory computer-readable media of claim 4, wherein the searching the stored log data includes obtaining the pattern and comparing the pattern to different patterns in log entries in the stored log data.

8. The one or more non-transitory computer-readable media of claim 4, wherein the received request includes an Application Program Interface (API) request.

9. The one or more non-transitory computer-readable media of claim 8, wherein the API request includes a name associated with one or more log streams stored in the stored log data.

10. The one or more non-transitory computer-readable media of claim 8, wherein the operations further comprise:

providing the UI script; and

upon selection of a user interface button, executing the UI script to generate the API request to extract the source-level log data.

11. The one or more non-transitory computer-readable media of claim 4, wherein the operations further comprise:

retrieving a log group identifier associated with the metric data; and

using the log group identifier for the searching the stored log data.

12. The one or more non-transitory computer-readable media of claim 4, wherein the transmitting the retrieved source-level log data for display includes transmitting a response to the received request.

13. The one or more non-transitory computer-readable media of claim 4, wherein the operations further comprise pushing a notification to the client device if the metric data exceeds a predetermined limit.

14. A system comprising:

one or more processors with memory coupled thereto, wherein the one or more processors are configured to;

generate metric data from associated source-level log data, using a metric filter, to display as a graph of the metric data on a client device, wherein the metric filter comprises a pattern of an error message;

receive a user input to view the source-level log data associated with the graph of the metric data, wherein the user input includes at least start and end times associated with the source-level log data so that the user selects a period of time associated with the source-level log data;

retrieve metadata, wherein the metadata comprises the metric filter associated with the graph of the metric data and one or more identifiers of the source-level log data, wherein the metadata is located on a server computer in a compute service provider environment;

receive a request from a user interface (UI) script to extract the source-level log data satisfying the metric filter over the selected time period;

search stored log data using the metric filter and the start and end times;

retrieve the source-level log data based at least in part on the search; and

transmit the retrieved source-level log data to the client device for display to the user.

15. The system of claim 14, further comprising:

a first storage for storing the stored log data; and

second storage, accessible by the UI script, for storing the metadata associated with comprising the metric data.

16. The system of claim 14, further including a user interface rendering engine associated with the UI script for displaying the graph of the metric data on the client device.

17. The system of claim 14, further including a virtual machine executing on another server computer, the virtual machine including an agent for monitoring an application and generating the source-level log data.

18. The system of claim 14, wherein the stored log data comprises a plurality of log streams generated by virtual machines in a multi-tenant environment.