Centralized processing of events

• US 9,317,343 B1
• Filed: 10/07/2008
• Issued: 04/19/2016
• Est. Priority Date: 03/28/2008
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Willhoit, Donald Ryan, Hill, Peter F., Mendoza, Raymond E.
• Primary Examiner(s): Mudrick, Timothy A
• Application Number:
  US12/246,580
• Time in Patent Office: 2,751 Days
• Field of Search: 719/318
• US Class Current: 1/1
• CPC Class Codes: G06F 11/00 Error detection; Error correction; Monitoring error detection, correction or monitoring in information storage based on relative movement between record carrier and transducer G11B20/18; monitoring, i.e. supervising the progress of recording or reproducing G11B27/36; in static stores G11C29/00
• CPC Class Codes: G06F 11/006 Identification G06F11/2289 takes precedence
• CPC Class Codes: G06F 11/1446 Point-in-time backing up or restoration of persistent data
• CPC Class Codes: G06F 11/3003 Monitoring arrangements specially adapted to the computing system or computing system component being monitored
• CPC Class Codes: G06F 11/302 where the computing system component is a software system
• CPC Class Codes: G06F 11/3072 where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: G06F 2201/865 Monitoring of software
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: H04L 41/0631 using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
• CPC Class Codes: H04L 43/04 Processing captured monitoring data, e.g. for logfile generation

Abstract

Disclosed are systems, methods, and other embodiments relating to event processing. A plurality of events from a plurality of services are buffered in a server, each of the services being implemented in at least one service server. An indexed storage of the events is performed in a data store. Metrics are generated from the events, where a condition associated with an operation of the services may be determined from the metrics.

Claims

1. A computer-implemented method, comprising:

identifying, by at least one computing device comprising at least one hardware processor, at least one task to be performed in a remote processing of data by individual ones of a plurality of services implemented in at least one remotely-located processor-based system, wherein the individual ones of the plurality of services participate in the remote processing of the data;

receiving, by the at least one computing device, a plurality of events generated in at least one of the plurality of services, the plurality of events indicating a status of a performance of the at least one task with respect to the remote processing of the data, wherein individual ones of the plurality of events are idempotent and immutable, the individual ones of the plurality of events being received by the at least one computing device asynchronously from the plurality of services;

associating, by the at least one computing device, an identifier with the individual ones of the plurality of events generated to identify the plurality of events that indicate the status of the performance of the at least one task;

buffering, by the at least one computing device, the plurality of events;

performing, by the at least one computing device, an indexed storage of the plurality of events in a data store;

receiving, by the at least one computing device, a request from an external process that includes the identifier;

querying, by the at least one computing device, the data store using the identifier to identify the individual ones of the plurality of events associated with the identifier;

generating, by the at least one computing device, a plurality of metrics from the plurality of events associated with the identifier; and

sending, by the at least one computing device, at least a portion of the plurality of metrics to the external process to identify a condition associated with an operation of the plurality of services in the remote processing of the data.

2. The computer-implemented method of claim 1, further comprising determining, by the at least one computing device, a time interval between chronologically adjacent ones of the plurality of events associated with the identifier received in the request from the external process.

3. The computer-implemented method of claim 1, wherein the remote processing of the data further comprises a compression, a translation, or a transformation of the data to generate a processed file.

4. The computer-implemented method of claim 1, wherein the remote processing of the data further comprises converting the data from a first format to a second format to generate a processed file in compliance with the second format.

5. A computer-implemented method, comprising:

identifying, by at least one computing device comprising at least one hardware processor, a plurality of tasks to be performed in a processing of data by individual ones of a plurality of services to generate a processed file;

obtaining, by the at least one computing device, a plurality of events from the plurality of services, the plurality of events being generated by the plurality of services to describe a status of a performance of the plurality of tasks with respect to the processing of the data;

associating, by the at least one computing device, an identifier with individual ones of the plurality of events generated to identify the plurality of events that indicate the status of the performance of the plurality of tasks;

performing, by the at least one computing device, an indexed storage of the plurality of events in a data store; and

generating, by the at least one computing device, a plurality of metrics from the plurality of events, where a condition associated with an operation of the plurality of services may be determined from the plurality of metrics.

6. The computer-implemented method of claim 5, further comprising sending, by the at least one computing device, the plurality of metrics to an external process to identify the condition associated with the operation of the plurality of services.

7. The computer-implemented method of claim 5, wherein the individual ones of the plurality of services are configured to:

perform at least one of the plurality of tasks with respect to the data, where the at least one of the plurality of tasks is performed by a respective one of the plurality of services independent of other ones of the plurality of tasks performed by other ones of the plurality of services; and

generate at least one of the plurality of events.

8. The computer-implemented method of claim 7, further comprising retrieving, by the at least one computing device, corresponding ones of the plurality of events from the data store in response to a request being received that includes the identifier, the corresponding ones of the plurality of events indicating a progress of the data through a workflow pipeline that comprises the plurality of tasks being performed by the plurality of services.

9. The computer-implemented method of claim 7, where the data further comprises a digital file.

10. The computer-implemented method of claim 5, wherein the individual ones of the plurality of events are idempotent.

11. The computer-implemented method of claim 5, wherein the individual ones of the plurality of events are immutable.

12. The computer-implemented method of claim 5, the individual ones of the plurality of events are received by the at least one computing device asynchronously from the plurality of services.

13. The computer-implemented method of claim 5, further comprising determining, by the at least one computing device, a time interval between chronologically adjacent ones of the plurality of events.

14. The computer-implemented method of claim 13, wherein a common identifier is associated with the plurality of events.

15. The computer-implemented method of claim 5, wherein the individual ones of the plurality of services comprise an event reporting agent, the event reporting agent comprising program instructions that, when executed, cause the event reporting agent to generate at least one of the plurality of events.

16. A system, comprising:

at least one computing device;

a plurality of events from a plurality of services stored in a data store accessible to the at least one computing device; and

program instructions that, when executed by the at least one computing device, cause the at least one computing device to;

identify a plurality of tasks to be performed in a processing of data by individual ones of a plurality of remote services, the plurality of events being generated by the plurality of remote services indicating a status of a performance of the plurality of tasks with respect to the processing of the data;

perform an indexed storage of the plurality of events in the data store; and

generate a plurality of metrics from the plurality of events, where a condition associated with an operation of the plurality of remote services may be determined from the plurality of metrics.

17. The system of claim 16, wherein the individual ones of the plurality of remote services are implemented in at least one remotely-located processor-based system, the individual ones of the plurality of remote services being configured to perform at least one of the plurality of tasks during a workflow process, where an operation of the individual ones of the plurality of remote services is independent with respect to each other.

18. The system of claim 16, wherein the individual ones of the plurality of events are generated in association with at least one of the plurality of tasks performed by the plurality of services during a workflow process, the individual ones of the plurality of events being associated with an identifier.

19. The system of claim 18, further comprising program instructions that, when executed, cause the at least one computing device to retrieve a subset of the plurality of events that are associated with the identifier.

20. The system of claim 16, wherein at least one of the plurality of remote services is configured to generate a predefined set of the plurality of events in association with at least one of the plurality of tasks to be performed by the at least one of the plurality of remote services.

21. The system of claim 20, wherein the individual ones of the plurality of events indicate a predefined progress marker associated with a progress of a workflow process.

22. The system of claim 16, wherein the individual ones of the plurality of events are idempotent.

23. The system of claim 16, wherein the individual ones of the plurality of events are immutable.

24. The system of claim 16, wherein a communication of the individual ones of the plurality of events from the plurality of remote services is asynchronous.

25. The system of claim 16, where the data further comprises a digital file.

26. The system of claim 16, further comprising program instructions that, when executed, cause the at least one computing device to determine a time interval between chronologically adjacent ones of the plurality of events received from the plurality of remote services.