System and Methodology for Parallel Stream Processing

• US 20090171999A1
• Filed: 12/23/2008
• Published: 07/02/2009
• Est. Priority Date: 12/27/2007
• Status: Active Grant

• Current Assignee: Cloudscale Inc.
• Original Assignee: Cloudscale Inc.
• Inventors: McColl, William Finlay, Suijlen, Wijnand Johannes
• Granted Patent: US 8,069,190 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases

Abstract

A system and methodology for parallel processing of continuous data streams. In one embodiment, a system for parallel processing of data streams comprises: a converter receiving input streams of data in a plurality of formats and transforming the streams into a standardized data stream format comprising rows and columns in which values in a given column are of a homogeneous type; a storage system that continuously maintains a finite interval of each stream subject to specified space limits for the stream; an interface enabling a user to construct parallel stream programs for processing streams in the standardized data stream format, wherein a parallel stream program comprises a linear sequence of program building blocks for performing operations on a data stream; and a runtime computing system running multiple parallel stream programs continuously on the streams as they flow through the storage system.

Claims

1. A system for parallel processing of data streams, the system comprising:

a converter that receives input streams of data in a plurality of formats and transforms said streams into a standardized data stream format comprising rows and columns in which values in a given column are of a homogeneous type;

a storage system that continuously maintains a finite interval of each stream in the standardized data stream format subject to specified space limits for the stream;

an interface enabling a user to construct parallel stream programs for processing streams in the standardized data stream format, wherein a parallel stream program comprises a linear sequence of program building blocks for performing operations on a data stream; and

a runtime computing system running multiple parallel stream programs continuously on the streams in the standardized data stream format as they flow through the storage system.

2. The system of claim 1, wherein said input streams of data comprise live data streams that are processed by the system in real-time.

3. The system of claim 1, wherein said standardized data stream format comprises a two-dimensional cell-based data format.

4. The system of claim 1, further comprising:

a timestamp module for applying a timestamp to each row of data in a stream in the standardized data stream format.

5. The system of claim 1, wherein said interface comprises a Graphical User Interface in which each stream of data is represented in a two-dimensional cell-based data format.

6. The system of claim 5, wherein the Graphical User Interface performs continuous static dependency and type checking during construction of parallel stream programs.

7. The system of claim 5, wherein the Graphical User Interface comprises an add-in to a spreadsheet application, allowing parallel stream program construction to be performed within the spreadsheet.

8. The system of claim 5, wherein the Graphical User Interface comprises a plug-in to an Internet browser application, allowing parallel stream program construction to be performed within the Internet browser.

9. The system of claim 1, wherein said program building blocks comprise selected ones of assignment, selection, parallel and window program building blocks.

10. The system of claim 1, wherein said parallel stream programs are constructed without requiring additional nonlinear control flow.

11. The system of claim 1, wherein said interface comprises an open application programming interface in which a parallel stream program is represented as a linear sequence of Extended Markup Language (XML) building blocks.

12. The system of claim 1, wherein said plurality of formats of said input data streams include input data streams in table, text, feed and graphic formats.

13. The system of claim 1, wherein output from a stream processing program is input directly into a spreadsheet.