Scalable Real Time Event Stream Processing

• US 20110016123A1
• Filed: 07/17/2009
• Published: 01/20/2011
• Est. Priority Date: 07/17/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Pandey, Vipul, Redner, Jordan, Rangaswamy, Gopi
• Granted Patent: US 8,880,524 B2
• Time in Patent Office: Days
• US Class Current: 707/737
• CPC Class Codes: G06F 16/2308 Concurrency control transaction processing G06F9/466
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues

Abstract

Scalable systems and methods for near real time processing of a substantial volume of event data streams are disclosed. A concurrent throughput receiver is coupled to an input of a processor for receiving the substantial volume of event data streams, and implementing substantially concurrent throughput of the substantial volume of event data streams. The systems and methods may provide for real time application monitoring, such as by aggregating information to a distributed cache from a plurality of entities, where the entities are structured in a nodal hierarchy, and summarizing to a summary database the information from the nodal hierarchy into summary level statistics for each of the nodes of the nodal hierarchy.

Claims

1. A computer-implemented apparatus for processing of a substantial volume of event data streams, generated by a substantial number of sensors for sensing events, the apparatus comprising:

a concurrent throughput receiver for receiving a volume of event data streams, and implementing substantially concurrent throughput of the volume of event data streams; and

a near real time event stream processor coupled with the receiver for aggregating data of the event streams into aggregated data of the events.

2. An apparatus as in claim 1 wherein the event stream processor comprises:

a distributed cache of computer-readable memory; and

a concurrent aggregator coupled with the distributed cache.

3. An apparatus as in claim 1 wherein the event stream processor comprises a concurrent aggregator.

4. An apparatus as in claim 1 wherein the event stream processor comprises a multi-threaded aggregator.

5. An apparatus as in claim 1 wherein the event stream processor comprises a host machine executing a plurality of aggregation threads.

6. An apparatus as in claim 1 wherein the event stream processor comprises an event stream processing engine.

7. An apparatus as in claim 1 wherein the event stream processor comprises a query configurable state machine.

8. An apparatus as in claim 1 wherein the event stream processor includes instructions for aggregating the data of the event data streams at levels of a logical hierarchy.

9. An apparatus as in claim 1 wherein the event stream processor includes instructions for aggregating the data of the event data streams at levels of a logical hierarchy organization of the sensors.

10. An apparatus as in claim 1 wherein the event stream processor includes instructions for aggregating the data of the event data streams at multiple levels of a logical hierarchy, the multiple levels of the logical hierarchy comprising at least two of:

a level of application instances, a level of host machines;

a level of database partitions;

a level of data centers; and

an application level.

11. An apparatus as in claim 1 wherein the event stream processor is adapted for transforming the data of the event data streams into complex event data.

12. An apparatus as in claim 1 wherein the event stream processor is adapted for transforming the data of the event data streams into complex event data, comprising at least one of:

host machine event data;

database partition event data;

data center event data; and

application event data.

13. An apparatus as in claim 1 further comprising:

a first client for providing a first user query of the aggregated data; and

a first server coupled with the event stream processor and the first client for serving the aggregated data to the first client in response to the query from the client.

14. An apparatus as in claim 13 further comprising a display coupled with the first client for displaying the near real time aggregated data of the events.

15. An apparatus as in claim 1 further comprising:

a display; and

a server coupled to the event stream processor, and operatively coupled to the display for displaying near real time graphical event visualization of the aggregated data.

16. An apparatus as in claim 10 further comprising:

a display; and

a server coupled to the event stream processor, and operatively coupled to the display for displaying near real time graphical event visualization of the aggregated data at the level of at the least one of;

the application instances;

the host machines;

the database partitions;

the data centers; and

the application.

17. An apparatus as in claim 1 further comprising a distributed publisher for replicating the volume of event data streams.

18. An apparatus as in claim 1 further comprising a host machine executing a plurality of publish threads for replicating the volume of event data streams.

19. An apparatus as in claim 1 further comprising:

a publisher; and

a publish-subscribe exchange coupled between the publisher and the concurrent receiver.

20. An apparatus as in claim 19 wherein the publish-subscribe exchange comprises a Message Oriented Middleware (MOM) publish-subscribe exchange.

21. An apparatus as in claim 19 wherein the publish-subscribe exchange comprises:

a first cluster of publish-subscribe exchange host machines provided at a location; and

a second cluster of publish-subscribe exchange host machines located geographically remote from the location of the first cluster of publish-subscribe exchange host machines.

22. An apparatus as in claim 1 wherein the concurrent receiver comprises a multi-threaded subscriber.

23. An apparatus as in claim 1 wherein the concurrent receiver comprises at least one pair of MOM queues.

24. An apparatus as in claim 1 wherein the concurrent receiver comprises an initial receiver queue sequentially arranged with a subsequent receiver queue for sequentially receiving the volume of event data streams.

25. An apparatus as in claim 1 wherein the concurrent receiver comprises an initial receiver queue having a configuration that is substantially different than a corresponding configuration of a subsequent receiver queue.

26. An apparatus as in claim 1 wherein the concurrent throughput receiver comprises an initial in-memory queue.

27. An apparatus as in claim 1 further comprising a concurrent adapter coupled to an input of the event stream processor for formatting the data of the volume of event data streams before being processed by the event stream processor.

28. An apparatus as on claim 1 further comprising a host machine executing a plurality of adapter threads coupled with the event stream processor for formatting the data of the event data streams.

29. An apparatus as in claim 1 further comprising a host machine executing a plurality of managed object adapter threads for formatting the data of the event data streams into managed objects.

30. An apparatus as in claim 1 further comprising:

database storage coupled with the event stream processor for archiving the aggregated data; and

a database server coupled with the database for providing time-based roll-up of the archived aggregated data.

31. An apparatus as in claim 1 wherein the near real time processing of the event stream processor is implemented in accordance with terms of a service level agreement.

32. An apparatus as in claim 1 wherein the throughput of the receiver is implemented in accordance with terms of a service level agreement.

33. A computer-implemented method for near real time application monitoring, the method comprising:

aggregating information to a distributed cache from a plurality of entities, wherein the entities are structured in a nodal hierarchy; and

summarizing to a summary database the nodal information into summary level statistics for each of the nodes of the hierarchy.

34. A method as in claim 33 wherein the nodal hierarchy includes application, data center, partition, host, and instance.

35. A computer readable medium including at least computer program code stored thereon for near real time application monitoring, said computer readable medium comprising:

computer program code for aggregating information to a distributed cache from a plurality of entities, wherein the entities are structured in a nodal hierarchy; and

computer program code for summarizing to a summary database the nodal information into summary level statistics for each of the nodes of the hierarchy.