Systems and methods for event stream processing

• US 9,304,809 B2
• Filed: 07/19/2012
• Issued: 04/05/2016
• Est. Priority Date: 06/26/2012
• Status: Active Grant

• Current Assignee: Walmart Apollo, LLC (WalMart Inc.)
• Original Assignee: Wal-Mart Stores Texas LLC (WalMart Inc.)
• Inventors: Lam, Wang Chee, Liu, Lu, Siripurapu, Taraka Subrahmanya Prasad, Rajaraman, Anand, Vacheri, Zoheb, Doan, AnHai
• Primary Examiner(s): NDIAYE, CHEIKH T
• Application Number:
  US13/553,652
• Publication Number:
  US 20130346625A1
• Time in Patent Office: 1,356 Days
• Field of Search: 709/207, 709/224, 709/219, 709/231
• US Class Current: 1/1
• CPC Class Codes: G06F 9/4843 by program, e.g. task dispatcher, supervisor, operating system
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications

Abstract

Disclosed are systems and methods for processing events in an event stream using a map-update application. The events may be embodied as a key-attribute pair. An event is processed by one or more instances implementing either a map or an update function. A map function receives an input event from the event stream and publishes one or more events to the event stream. An update function receives an event and updates a corresponding slate and publishes zero or more events. Systems and methods are also disclosed herein for implementing a map-update application in a multithreaded architecture and for handling overloading of a particular thread or node. Systems and methods for providing access to slates updated according to update operations are also disclosed.

Claims

1. A method for performing event processing comprising:

processing an event stream by an application executed by a computer system, the application comprising a plurality of event handlers, each event handler of the plurality of event handlers comprising;

receiving an input stream from the event stream;

performing an event handling function; and

outputting to an output stream of the event stream;

detecting an overload condition for an event handler of the plurality of event handlers, the overload condition comprising one or more excess events of the input stream in excess of a capacity of the event handler of the plurality of event handlers; and

in response to the detecting of the overload condition;

processing the one or more excess events according to a degraded handling function, the degraded handling function comprising;

protecting the one or more excess events from discard; and

a lower latency than the event handling function;

publishing events according to the processing of the one or more excess events according to the degraded handling function to the output stream of the event stream of the event handler of the plurality of event handlers;

detecting a degraded handling function overload condition for the degraded handling function, the degraded handling function overload condition comprising one or more degraded handling function excess events in excess of a capacity of the degraded handling function; and

in response to the detecting of the degraded handling function overload condition;

processing the one or more degraded handling function excess events according to a second degraded handling function, the second degraded handling function comprising protecting the one or more degraded handling function excess events from discard.

2. The method of claim 1, wherein the event handling function requires a greater number of computations than the degraded handling function.

3. The method of claim 1, wherein the event handling function accesses an external resource and the degraded handling function does not require access to the external resource.

4. The method of claim 1, wherein the event handling function makes a network function call and the degraded handling function does not make a network function call.

5. The method of claim 1, wherein the plurality of event handlers includes mappers and updaters, the mappers having an event handling function operable to perform a stateless computation and the updaters having an event handling function operable to perform a state dependent computation.

6. The method of claim 1, wherein the event stream includes a plurality of events each including an attribute and value pair.

7. A system for performing event processing comprising one or more processors and one or more memory devices operably coupled to the one or more processors, the one or memory devices storing executable and operational code effective to cause the one or more processors to:

process an event stream by an application, the application comprising a plurality of event handlers, each event handler of the plurality of event handlers comprising;

receiving an input stream from the event stream;

performing an event handling function; and

outputting to an output stream of the event stream;

detect an overload condition for an event handler of the plurality of event handlers, the overload condition comprising one or more excess events of the input stream in excess of a capacity of the event handler of the plurality of event handlers; and

in response to the detecting of the overload condition;

process the one or more excess events according to a degraded handling function, the degraded handling function comprising;

protecting the one or more excess events from discard; and

a lower latency than the event handling function;

publish events according to the processing the one or more excess events according to the degraded handling function to the output stream of the event stream of the event handler of the plurality of event handlers a lower latency than the event handling function;

detect a degraded handling function overload condition for the degraded handling function, the degraded handling function overload condition comprising one or more degraded handling function excess events in excess of a capacity of the degraded handling function; and

in response to the detecting the degraded handling function overload condition;

process the one or more degraded handling function excess events according to a second degraded handling function, the second degraded handling function comprising protecting the one or more degraded handling function excess events from discard.

8. The system of claim 7, wherein the event handling function is effective to cause the one or more processors to perform a greater number of computations than the degraded handling function.

9. The system of claim 7, wherein the event handling function is effective to cause the one or more processors to access an external resource and the degraded handling function is not operable to cause the one or more processors to access the external resource.

10. The system of claim 7, wherein the event handling function is effective to cause the one or more processors to make a network function call and the degraded handling function does not make a network function call.

11. The system of claim 7, wherein the plurality of event handlers includes mappers and updaters, the mappers having an event handling function effective to cause the one or more processors to perform a stateless computation and the updaters having an event handling function effective to cause the one or more processors to perform a state dependent computation.

12. The system of claim 7, wherein the event stream includes a plurality of events each including an attribute and value pair.

13. A computer program product for managing a transaction, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

processing an event stream by an application executed by a computer system, the application comprising a plurality of event handlers, each event handler of the plurality of event handlers comprising;

receiving an input stream from the event stream;

performing an event handling function; and

outputting to an output stream;

detecting an overload condition for an event handler of the plurality of event handlers, the overload condition comprising one or more excess events of the input stream in excess of a capacity of the event handler of the plurality of event handlers; and

in response to the detecting the overload condition;

processing at least a portion of the one or more excess events according to a degraded handling function, the degraded handling function comprising;

protecting the one or more excess events from discard; and

a lower latency than the event handling function;

publishing events according to the processing the one or more excess events according to the degraded handling function to the output stream of the event stream of the event handler of the plurality of event handlers;

detecting a degraded handling function overload condition for the degraded handling function, the degraded handling function overload condition comprising one or more degraded handling function excess events in excess of a capacity of the degraded handling function; and

in response to the detecting the degraded handling function overload condition;

processing the one or more degraded handling function excess events according to a second degraded handling function, the second degraded handling function comprising protecting the one or more degraded handling function excess events from discard.

14. The computer program product of claim 13, wherein the event handling function requires a greater number of computations than the degraded handling function.

15. The computer program product of claim 13, wherein the event handling function accesses an external resource and the degraded handling function does not require access to the external resource.

16. The computer program product of claim 13, wherein the event handling function makes a network function call and the degraded handling function does not make a network function call.

17. The computer program product of claim 13, wherein the plurality of event handlers includes mappers and updaters, the mappers having an event handling function operable to perform a stateless computation and the updaters having an event handling function operable to perform a state dependent computation.

18. The computer program product of claim 13, wherein the event stream includes a plurality of events each including an attribute and value pair.