Scalable and extendable stream processing
First Claim
1. A computer-implemented system, comprising:
- a streaming data processing component that receives streaming data from streaming data feeds and performs lossy per-algorithm stream throttling of streaming data feeds to enable stream processing algorithms to receive most recent data from the streaming data feeds, the streaming data processing component including an output data bus component that(i) processes algorithm output into result sets and(ii) manages expiration policies defined for each result set; and
a microprocessor that executes computer-executable instructions associated with the streaming data processing component.
2 Assignments
0 Petitions
Accused Products
Abstract
An extensible architecture that enables developers to focus solely on the domain-specific nature of the stream processing algorithm to be implemented. It is positioned as an intermediary component between streaming data feeds and stream algorithms, thereby aggregating demand on data sources and hiding the complexity involved in managing active connections to different data sources. Per-algorithm stream throttling is provided so that individual algorithms do not get overloaded; thus, ensuring that algorithms receive fresh items from the data feeds to which the algorithms subscribe. Feed items can be discarded when an algorithm is not able to process the items in realtime to ensure that feed items are sampled at the fastest processing rate of the algorithm. Thus, a single instance of an algorithm can handle an entire data stream. Moreover, redundancy can be achieved by running the same configuration on multiple machines.
-
Citations
19 Claims
-
1. A computer-implemented system, comprising:
-
a streaming data processing component that receives streaming data from streaming data feeds and performs lossy per-algorithm stream throttling of streaming data feeds to enable stream processing algorithms to receive most recent data from the streaming data feeds, the streaming data processing component including an output data bus component that (i) processes algorithm output into result sets and (ii) manages expiration policies defined for each result set; and a microprocessor that executes computer-executable instructions associated with the streaming data processing component. - View Dependent Claims (2, 3, 4, 5, 6, 7)
-
-
8. A computer-implemented system, comprising:
-
a streaming data processing component that receives streaming data from streaming data feeds and performs lossy per-algorithm stream throttling of streaming data feeds to enable stream processing algorithms to receive most recent data from the streaming data feeds, the streaming data processing component further comprising; an input data bus component that consumes the data feeds and routes the streaming data to the algorithms; a buffer management component that monitors buffer consumption for each algorithm and manages data flow to maintain realtime stream processing in one or more of the algorithms; an output data bus component that processes algorithm output into result sets and manages expiration policies defined for each result set; and a web service that enables retrieval of the results sets by consuming systems; and a microprocessor that executes computer-executable instructions associated with the streaming data processing component. - View Dependent Claims (9, 10, 11)
-
-
12. A computer-implemented method, comprising acts of:
-
receiving data for routing to stream processing algorithms; monitoring consumption of data by an algorithm for realtime stream data processing; managing, using lossy throttling, input of the data to the algorithm based on the consumption by performing lossy per-algorithm stream throttling of the received data to enable the stream processing algorithms to receive most recent data from the received data; processing algorithm output into result sets and managing expiration policies defined for each result set; and utilizing a processor that executes instructions stored in memory to perform at least one of the acts of monitoring, processing or managing. - View Dependent Claims (13, 14, 15, 16, 17, 18)
-
-
19. One or more computer-readable storage media encoded with processing instruction that, when executed by a processor, cause the execution of a method, the method comprising:
-
receiving data for routing to stream processing algorithms; monitoring consumption of data by an algorithm for realtime stream data processing; managing, using lossy throttling, input of the data to the algorithm based on the consumption by performing lossy per-algorithm stream throttling of the received data to enable the stream processing algorithms to receive most recent data from the received data; and processing algorithm output into result sets and managing expiration policies defined for each result set.
-
Specification