Filters and event schema for categorizing and processing streaming event data

• US 10,331,693 B1
• Filed: 09/12/2016
• Issued: 06/25/2019
• Est. Priority Date: 09/12/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Kothandaraman, Raghunathan, Dixit, Deep, Gupta, Gaurav, Rajaram, Sasanka, Kumar, Ankit, Aiken, Mark Aran, Parmar, Dhaval
• Primary Examiner(s): Cao, Phuong Thao
• Application Number:
  US15/262,744
• Time in Patent Office: 1,016 Days
• Field of Search: 707756
• CPC Class Codes: G06F 16/212 with details for data modelling support
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/84 Mapping; Conversion

Abstract

Disclosed are various embodiments for filters and event schema for categorizing and processing streaming event data. An event may be generated by a service that describes user interaction a client application executable on a client device. The event may be received as a data structure in a first format. A filter, such as a regular expression filter, may be applied to the data structure to identify an event type for the event. The data structure may be converted from the first format to a second format in accordance with a common event schema able to be interpreted by virtual compute engines.

Claims

1. A non-transitory computer-readable medium embodying program code executable in at least one computing device that, when executed, directs the at least one computing device to:

receive a specification of a pattern of events and an action to be performed in association with the pattern of events from an administrator device;

access an event received in a stream of events generated by a plurality of services, wherein the event describes at least one instance of user interaction with at least one client application executed on a client device, the event being received as a data structure in a first format;

identify a source of the event from the services based at least in part on a service identifier;

identify a regular expression filter from a plurality of potential regular expression filters stored in a data store, the regular expression filter being identified from the data store based at least in part on the source;

apply the regular expression filter to the data structure in the first format to identify an event type for the event;

identify a compute engine interested in the event, the compute engine identified based at least in part on the event type;

transform the data structure from the first format to a second format in accordance with a common event schema interpretable by the compute engine interested in the event; and

communicate the data structure in the second format to the compute engine, the compute engine being a virtual computing process configured to;

compare the event to the pattern of events; and

in response to all events in the pattern of events being matched, cause the action to be performed.

2. The non-transitory computer-readable medium of claim 1, wherein

transforming the data structure from the first format to the second format comprises converting the data structure from an extensible markup language (XML) format to a JavaScript object notation (JSON) format.

3. The non-transitory computer-readable medium of claim 1, wherein transforming the data structure from the first format to the second format comprises converting the data structure from a JavaScript object notation (JSON) format to an extensible markup language (XML) format.

4. The non-transitory computer-readable medium of claim 1, wherein:

the compute engine is generated in response to a receipt of the specification of the pattern of events and the action; and

the compute engine is generated to correspond to an account associated with the pattern of events.

5. A system, comprising:

at least one computing device; and

program instructions executable in the at least one computing device that, when executed by the at least one computing device, cause the at least one computing device to;

receive a specification of a pattern of events and an action to be performed in association with the pattern of events from an administrator device;

access an event received in a stream of events generated by a plurality of services, wherein the event describes at least one instance of user interaction with at least one client application executed on a client device;

identify a regular expression filter from a plurality of potential regular expression filters stored in a data store, the regular expression filter being identified from the data store based at least in part on a source of the event;

apply the regular expression filter to at least a portion of the event to classify the event;

identify a compute engine interested in the event, the compute engine identified based at least in part on the event being classified using the regular expression filter; and

communicate the event to the compute engine interested in the event as classified, wherein the compute engine comprises a virtual computing process configured to;

compare the event to the pattern of events; and

in response to all events in the pattern of events being matched, cause the action to be performed.

6. The system of claim 5, wherein the event is received as a data structure in a first format.

7. The system of claim 6, further comprising program instructions executable in the at least one computing device that, when executed by the at least one computing device, cause the at least one computing device to transform the data structure from the first format to a second format.

8. The system of claim 7, wherein transforming the data structure from the first format to the second format further comprises:

identifying a common event schema from a data store, the common event schema interpretable by the compute engine; and

generating a new data structure in the second format in accordance with the common event schema.

9. The system of claim 8, wherein generating the new data structure in the second format in accordance with the common event schema further comprises:

selecting a first portion of data accessed from the data structure for inclusion in a header of the new data structure; and

placing a second portion of data accessed from the data structure in a body of the new data structure.

10. The system of claim 5, wherein the compute engine is a virtual machine executed in the at least one computing device or a thread processed in the at least one computing device.

11. The system of claim 5, wherein:

the event is one of a plurality of events; and

individual ones of the services comprise an event reporting agent configured to generate the events.

12. The system of claim 5, wherein:

the compute engine is generated in response to a receipt of the specification of the pattern of events and the action; and

the compute engine is generated to correspond to an account associated with the pattern of events.

13. A method, comprising:

receiving, by at least one computing device comprising at least one hardware processor, a specification of a pattern of events and an action to be performed in association with the pattern of events from an administrator device;

receiving, by the at least one computing device, an event in a data stream generated by a plurality of services, wherein the event describes at least one instance of user interaction with at least one client application executed on a client device, the event being received as a data structure in a first format;

identifying, by the at least one computing device, a regular expression filter from a plurality of potential regular expression filters stored in a data store, the regular expression filter being identified from the data store based at least in part on a source of the event;

applying, by the at least one computing device, the regular expression filter to the data structure in the first format to identify an event type for the event;

accessing, by the at least one computing device, a common event schema associated with a plurality of virtual compute engines configured to process the event;

transforming, by the at least one computing device, the data structure from the first format to a second format based at least in part on the common event schema;

identifying, by the at least one computing device, at least one of the virtual compute engines interested in the event based at least in part on the event type of the event; and

communicating, by the at least one computing device, the data structure in the second format to the at least one of the virtual compute engines, wherein the at least one of the virtual compute engines comprises a virtual computing process configured to;

compare the event to the pattern of events; and

in response to all events in the pattern of events being matched, cause the action to be performed.

14. The method of claim 13, further comprising:

identifying, by the at least one computing device, the source of the event from the services based at least in part on a service identifier, wherein the regular expression filter is identified from the plurality of potential regular expression filters stored in the data store based at least in part on the service identifier.

15. The method of claim 13, wherein individual ones of the virtual compute engines comprise a virtual machine executed in the at least one computing device or a thread processed in the at least one computing device.

16. The method of claim 14, further comprising:

generating, by the at least one computing device, the at least one of the virtual compute engines; and

registering, by the at least one computing device, the at least one of the virtual compute engines in a compute engine index in association with at least one interested event type.

17. The method of claim 16, wherein the at least one of the virtual compute engines interested in the event is identified by querying, by the at least one computing device, the compute engine index to identify the at least one of the virtual compute engines associated with the at least one interested event type.

18. The method of claim 17, wherein the at least one interested event type is a same type of event as the event type identified for the event.

19. The method of claim 13, wherein transforming the data structure from the first format to the second format based at least in part on the common event schema comprises converting the data structure from a JavaScript object notation (JSON) format to an extensible markup language (XML) format.

20. The method of claim 13, further comprising generating, by the at least one computing device, the at least one of the virtual compute engines in response to a receipt of the specification of the pattern of events and the action, wherein the at least one of the virtual compute engines is generated to correspond to an account associated with the pattern of events.