Method and system for dynamically initializing a view for a streaming data base system

• US 7,672,964 B1
• Filed: 12/31/2003
• Issued: 03/02/2010
• Est. Priority Date: 12/31/2003
• Status: Active Grant

• Current Assignee: Viewceler Celequest, Twitter Inc (X Holdings Corp.)
• Original Assignee: International Business Machines Corporation
• Inventors: Radalj, Maurice Mate, Jazayeri, Nasrollah, Yan, Weipeng Paul, Nesanoney, Diaz H., Younas, Fayyaz Ahmad
• Primary Examiner(s): Pham; Khanh B
• Assistant Examiner(s): Saeed; Usmaan
• Application Number:
  US10/750,507
• Time in Patent Office: 2,253 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

A method for dynamically initializing a view for a streaming database system. The method includes accessing a stream of events. An initial view is materialized from the stream, wherein the initial view comprises a dynamically defined view of the stream of events from the streaming database system. The method also includes processing a plurality of recent view snapshots from the initial view. A view state derived from each of the recent view snapshots is generated. An updated view is subsequently maintained in accordance with the view state, wherein the updated view incorporates new events of the stream.

Claims

1. A method for dynamically initializing a view for a streaming database system, comprising:

accessing, by a computing device, at least one stream of events in real time while said at least one stream of events is received by said computing device as output from a streaming database system, wherein said computing device is external to said streaming database system;

materializing, by said computing device, an initialized view from said at least one stream of events as said at least one stream of events is received from said streaming database system, wherein said initialized view is dynamically defined from said at least one stream of events, and wherein materializing said initialized view comprises receiving a first dynamic view definition and generating said initialized view according to said first dynamic view definition;

producing, by said computing device, a sequence of view snapshots from said initialized view, wherein each view snapshot in said sequence corresponds to an individual event within said at least one stream of events, wherein each individual event occurs after events of said at least one stream of events are used to materialize said initialized view;

storing said sequence of view snapshots;

receiving a second dynamic view definition, said second dynamic view definition being different than said first dynamic view definition; and

using said sequence of view snapshots to generate, by said computing device, a re-initialized view according to said second dynamic view definition, wherein generating said re-initialized view comprises incorporating events used to materialize said initialized view and events of said sequence of view snapshots, and wherein generating said re-initialized view comprises calculating a view state of said re-initialized view by retrieving and replaying said sequence of view snapshots.

2. The method as recited in claim 1 wherein said initialized view comprises a plurality of row data structures.

3. The method as recited in claim 1, wherein said initialized view comprises a stateful view resulting from a stateful stream, said stateful view having a bounded number of rows.

4. The method of claim 3, wherein each view snapshot of said stateful view comprise a state of said stateful view including events existent at said materializing of said stateful view and events accessed after said materializing of said stateful view at a particular time.

5. The method of claim 1, wherein said initialized view comprises a stateless view resulting from a stateless stream, said stateless view having an unbounded number of rows.

6. The method of claim 5, wherein each view snapshot of said stateless view comprises a state of said stateless view including events existent at said materializing of said stateless view, including a last event processed during said materializing of said stateless view.

7. The method of claim 1, further comprising:

processing said sequence of view snapshots by maintaining a sequence of a plurality of preceding current view snapshots.

8. The method of claim 7, further comprising:

applying a batch set of events to said processing of said sequence of view snapshots, wherein each event of said batch set has a corresponding view snapshot in said sequence.

9. The method of claim 1, wherein said generating of said initialized view is configured to accomplish a recovery of a view state.

10. The method of claim 1, wherein said generating of said initialized view is configured to accomplish a re-enabling of a view after a disabling of a view.

11. The method of claim 1, wherein said generating of said initialized view is configured to accomplish a load balancing of a view maintenance process.

12. The method of claim 1, wherein accessing at least one stream of events comprises:

issuing, by said computing device, a stream query to said streaming database system; and

receiving, by said computing device, each of said events of said at least one stream of events as a result of said stream query without reissuing said stream query to said streaming database system.

13. A system comprising:

a streaming database system that outputs at least one stream of events;

a computing device, external to said streaming database system, comprising a view engine configured to access said at least one stream of events in real time while said at least one stream of events is received as output from said streaming database system, to materialize an initialized view from said at least one stream of events, wherein said initialized view is dynamically defined from said at least one stream of events, said view engine further configured to receive a first dynamic view definition and generate said initialized view according to said first dynamic view definition, to produce a sequence of view snapshots from said initialized view, wherein each view snapshot in said sequence corresponds to an individual event within said at least one stream of events, wherein each individual event occurs after events of said at least one stream of events are used to materialize said initialized view, store said sequence of view snapshots, receive a second dynamic view definition, said second dynamic view definition being different than said first dynamic view definition, and use said sequence of view snapshots to generate a re-initialized view of a second view according to said second dynamic view definition, wherein said view engine is configured to incorporate said events used to materialize said initialized view and events of said view snapshots and to calculate a view state of said re-initialized view by retrieving and replaying said sequence of view snapshots in order to generate said re-initialized view,

wherein said streaming database system is configured to send said at least one stream of events to said computing device.

14. The system of claim 13, wherein said computing device issues a stream query to said streaming database system, and wherein said streaming database system sends events of said at least one stream of events to said computing device as a result of said stream query without said computing device reissuing said stream query to said streaming database system.

15. The system of claim 13, wherein said initialized view comprises a stateful view resulting from a stateful stream, said stateful view having a bounded number of rows.

16. The system of claim 15, wherein each recent view snapshot of said stateful view comprises a state of said initialized view including events existent at said materializing of said initialized view and events accessed after said materializing of said initialized view at a particular time.

17. The system of claim 13, wherein said initialized view comprises a stateless view resulting from a stateless stream, said stateless view having an unbounded number of rows.

18. The system of claim 17, wherein each recent view snapshot of said stateless view comprises a state of said initialized view including events existent at said materializing of said initialized view, including a last event processed during said materializing of said initialized view.

19. The system of claim 13, wherein said view engine is further configured to process said sequence of recent view snapshots by maintaining a sequence of a plurality of preceding current view snapshots.

20. The system of claim 19, wherein said view engine is further configured for applying a batch set of events to said processing of said sequence of recent view snapshots, wherein said event of said batch set has a corresponding recent view snapshot in said sequence.

21. A computer-readable medium comprising instructions for causing a programmable processor of a computing device external to a streaming database to:

access at least one stream of events in real time while said at least one stream of events is received by said computing device as output from a streaming database system, wherein said computing device is external to said streaming database system;

materialize an initialized view from said at least one stream of events as said at least one stream of events is received from said streaming database system, wherein said initialized view is dynamically defined from said at least one stream of events, and wherein said instructions to materialize said initialized view comprise instructions to receive a first dynamic view definition and generate said initialized view according to said first dynamic view definition;

produce a sequence of view snapshots from said initialized view, wherein each view snapshot in said sequence corresponds to an individual event within said at least one stream of events, wherein each individual event occurs after events of said at least one stream of events are used to materialize said initialized view;

store said sequence of view snapshots;

receive a second dynamic view definition, said second dynamic view definition being different than said first dynamic view definition; and

use said sequence of view snapshots to generate a re-initialized view according to said second dynamic view definition, wherein said instructions to generate said re-initialized view comprise instructions to incorporate events used to materialize said initialized view and events of said sequence of view snapshots, and wherein said instructions to generate said re-initialized view comprise instructions to calculate a view state of said re-initialized view by retrieving and replaying said sequence of view snapshots.

22. The computer-readable medium of claim 21, wherein said initialized view comprises a stateful view resulting from a stateful stream, said stateful view having a bounded number of rows.

23. The computer-readable medium of claim 22, wherein each view snapshot of said stateful view comprises a state of said stateful view including events existent at said materializing of said stateful view and events accessed after said materializing of said stateful view at a particular time.

24. The computer-readable medium of claim 21, wherein said view comprises a stateless view resulting from a stateless stream, said stateless view having an unbounded number of rows.

25. The computer-readable medium of claim 24, wherein each view snapshot of said stateless view comprises a state of said stateless view including events existent at said materializing of said stateless view, including a last event processed during said materializing of said stateless view.

26. The computer-readable medium of claim 21, further comprising instructions for causing said programmable processor of said computing device to process said sequence of view snapshots by maintaining a sequence of a plurality of preceding current view snapshots.

27. The computer-readable medium of claim 26, further comprising instructions for causing said programmable processor of said computing device to apply a batch set of events to said processing of said sequence of view snapshots, wherein each event of said batch set has a corresponding view snapshot in said sequence.