Segmentation and processing of continuous data streams using transactional semantics

• US 7,392,320 B2
• Filed: 05/14/2004
• Issued: 06/24/2008
• Est. Priority Date: 06/18/1999
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Beckerle, Michael J., Krawitz, Robert Louis, Callen, Jerry Lee, Bookman, Lawrence A., Mudambi, Shyam R., Blair, David Albert, Rosenthal, Steven M., Razdow, Allen M.
• Primary Examiner(s): DUONG, OANH L
• Application Number:
  US10/846,342
• Publication Number:
  US 20050021745A1
• Time in Patent Office: 1,502 Days
• Field of Search: 709/203, 709/217, 709/219, 709223-225, 709/229, 709/231, 707/202, 714/15, 718/101
• US Class Current: 709/231
• CPC Class Codes: G06F 40/149 Adaptation of the text data for streaming purposes, e.g. Efficient XML Interchange [EXI] format
• CPC Class Codes: G06F 9/466 Transaction processing
• CPC Class Codes: Y10S 707/99952 Coherency, e.g. same view to multiple users

Abstract

With a continuous source of data relating to transactions, the data may be segmented and processed in a data flow arrangement, optionally in parallel, and the data may be processed without storing the data in an intermediate database. Data from multiple sources may be processed in parallel. The segmentation also may define points at which aggregate outputs may be provided, and where checkpoints may be established.

Claims

1. A method for processing a continuous stream of transactional data, the method comprising:

applying transactional semantics to the continuous stream of transactional data to identify a plurality of segments of the continuous stream of transactional data, wherein the continuous stream of transactional data is from at least one transaction in progress, and wherein each of the plurality of segments contains transactional data,

processing the transactional data in each of the plurality of segments of the continuous stream of transactional data to produce results for each of the plurality of segments; and

in response to processing the transactional data in one of the plurality of segments to produce results for one of the plurality of segments, outputting the results produced for the one of the plurality of segments to a storage device,

wherein the continuous stream of transactional data is a log of information relating to a plurality of reguests issued to a server, and wherein the applying step comprises reading information relating to one of the plurality of reguests from the log of information; and

applying the transactional semantics to the information,

wherein the information relating to the one of the plurality of reguests includes a plurality of fields, and wherein the transactional semantics are defined by a function of the plurality of fields, and

wherein the transactional data in the continuous stream of transactional data has a seguence, and wherein there are multiple sources of the continuous stream of transactional data,

determining whether a particular portion of transactional data in the continuous stream of transactional data is in seguence, and

if the particular portion of transactional data is determined to be out of sequence, interrupting the processing step, inserting the particular portion of transactional data in a particular segment in the plurality of segments according to the transactional semantics, reprocessing the particular segment, and continuing the processing step.

2. The method of claim 1, wherein the transactional data in each of the plurality of segments includes a plurality of records, wherein each of the plurality of records includes a plurality of fields, and wherein the transactional semantics are defined by a function of at least one of the plurality of fields.

3. The method of claim 1, further comprising:

partitioning the continuous stream of transactional data according to the plurality of segments.

4. The method of claim 3, wherein partitioning comprises:

inserting a record in the continuous stream of transactional data indicating a boundary between two segments.

5. The method of claim 4, wherein the record is a marker record indicating only the boundary.

6. The method of claim 4, wherein the record is a semantic record including information related to the transactional semantics.

7. The method of claim 1, wherein the information relating to the one of the plurality of requests includes a time at which the one of the plurality of requests was issued to the server, and wherein the transactional semantics define a period of time.

8. The method of claim 1, wherein the continuous stream of transactional data is a log of information relating to automatic requests issued to a server, and further comprising:

filtering the log to eliminate information relating to requests associated with spiders.

9. The method of claim 1, further comprising:

filtering the continuous stream of transactional data to eliminate data from the continuous stream of transactional data.

10. The method of claim 9, wherein the step of filtering the continuous stream of transactional data to eliminate data from the continuous stream of transactional data is performed prior to the step of applying transactional semantics to the continuous stream of transactional data to identify the plurality of segments of the continuous stream of transactional data.

11. The method of claim 1, further comprising:

an additional step of processing the transactional data in each of the plurality of segments of the continuous stream of transactional data to produce additional results for each of the plurality of segments; and

in response to the additional step of processing, outputting the additional results produced for each of the plurality of segments to a storage device.

12. The method of claim 1, wherein processing comprises:

partitioning the transactional data in each of the plurality of segments into a plurality of parallel partitions; and

processing each of the plurality of parallel partitions in parallel to output intermediate results for each of the plurality of parallel partitions.

13. The method of claim 12, further comprising:

combining the intermediate results of each of the plurality of parallel partitions to produce results for each of the plurality of segments.

14. The method of claim 1, further comprising:

saving a persistent indication of a current segment that is currently being processing in the step of processing the transactional data;

in response to detecting a failure in processing the transactional data, discarding the current results and reprocessing the current segment; and

in response to successfully processing the transactional data, outputting the current results to a storage device.

15. The method of claim 1, wherein the continuous stream of transactional data represents user input associated with the at least one transaction in progress.

16. The method of claim 1, wherein the continuous stream of transactional data comprises data selected from the group consisting of a reservation system, a point-of-sale system, an automatic teller machine, a banking system, a credit card system, a search system, and an audio/video distribution system.