Method and system of processing a data stream

• US 7,730,199 B2
• Filed: 04/12/2007
• Issued: 06/01/2010
• Est. Priority Date: 05/11/2006
• Status: Expired due to Fees

• Current Assignee: Korea Advanced Institute of Science and Technology
• Original Assignee: Korea Advanced Institute of Science and Technology
• Inventors: Lee, Young-Ki, Song, June-Hwa, Park, Soun-Eil, Kang, Seung-Woo, Ihm, Sung-Hwan, Han, Jae-Sun, Kim, Su-Myeon
• Primary Examiner(s): Serrao; Ranodhi N
• Application Number:
  US11/734,399
• Publication Number:
  US 20070288649A1
• Time in Patent Office: 1,146 Days
• Field of Search: 709/230, 709/231, 710/52, 717/153, 719/331
• US Class Current: 709/231
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries

Abstract

Methods and systems of processing a data stream are disclosed. A developer may develop a service only by programming a Service Unit (SU) including a service logic (SL) and including a Service Unit Description Language (SUDL) file, so that the developer may not need to program complicated stream processing code, such as stream source connection code, network input/output (I/O) code, and buffering code. In addition, the stream source connected dynamically to the service and a service execution mode (such as push and pull modes) may be converted by simply changing setting files based on the service model. The system includes modules for managing and processing the stream in reality, such as stream source connection modules, network I/O processing modules, and stream buffering modules, the modules operating the developed program through the service model.

Claims

1. A method of processing a data stream(DS), comprising:

selecting a Stream Source Description Language (SSDL) file and a Service Unit (SU), the SSDL file and the SU being assigned by a Service Unit Activation Language (SUAL) file;

transmitting a query written by the selected SUAL file to a corresponding stream provider based on a stream provider address written by selected SSDL file;

preparing a buffer based on a schema information about a result of the transmitted query;

generating an instance of the selected SU;

intializing the selected SU into an execution mode assigned by the SUAL file;

updating, in real time, data stored in the prepared buffer by using the DS provided by the stream provider; and

processing the data stored in the buffer according to the execution mode in response to a user request to provide a result of the user request to a user by a service operator (SO) module,

wherein the SUAL file comprises the SSDL file, the SU, the query and the execution mode,

the execution mode comprises a push mode and a pull mode,

one instance of the selected SU per user request is generated and preserved during a requested time duration in the push mode, and

a predetermined number of the instances of the selected SU is generated and managed in an instance pool in the pull mode,

wherein the SO module comprises;

a user request handler (URH) that registers the user request and transmits the result of the user request to the user;

a scheduler that controls a processing schedule of the user request registered in the URH in response to a resource usage state;

a service unit instance manager (SUIM) that processes the user request by executing the SUI by processing the SUI in response to the processing schedule from the scheduler according to the execution mode and provides the result of the user request to the URH; and

a runtime moniter (RM) that monitors the resource usage state of each component and the SUI that is currently being executed, and provides the resource usage state to the scheduler,

wherein the SUIM generates one SUI per user request and the generated SUI becomes extinct after a requested time duration when the execution mode corresponds to the push mode.

2. The method of claim 1, wherein the SU comprises a service logic (SL)and a Service Unit Description Language (SUDL) file, the SL and SUDL file being developed by a service developer.

3. The method of claim 2, wherein the SL comprises an initialization function, a process function, an uninitialization function, a buffer semantic function, and a tuple handler function.

4. The method of claim 2, wherein the SUDL file comprises a service name, an input data schema, an output data schema, and a transmission element.

5. The method of claim 1, wherein the buffer comprises a first buffer for buffering recent N pieces of data, a second buffer for buffering one piece of data per data key, and a third buffer for buffering recent N pieces of data per data key.

6. The method of claim 1, wherein the requested data is updated in real time and is repeatedly provided to the user when the execution mode corresponds to the push mode.

7. The method of claim 1, wherein the requested data is provided only by the user when the execution mode corresponds to pull mode.

8. The method of claim 1, wherein the SSDL file comprises a service name, a location and a DS schema.

9. A computer middleware system of processing a data stream (DS) and transmitting the DS from a stream provider to a user terminal, comprising;

a memory;

a service deployer (SD) module configured to register a Service Unit (SU) developed by a serviced developer and an Serviced Unit Activation Language (SUAL) file written by a service provider, and configured to generate a query, a query result schema and execution mode information, the query result schema and the execution mode information being assigned by the registered SUAL file;

a stream handler (SH) module configured to transmit the query from the SD module to the stream provider and configured to update, in real time, data stored in a buffer in response to the query result schema by using the DS provided by the stream provider; and

a service operator (SO) module configured to receive a user request, configured to process the data stored in the buffer by executing a service unit instance (SUI) in response to the user request according to the execution mode information from the SD module, and configured to provide a result of the user request to the user,

wherein the SUAL file comprises a Service Source Description Language (SSDL) file, the SU, the query and the execution mode,

the execution mode comprises a push mode and a pull mode,

one SUI per user request is generated and preserved during a requested time duration in the push mode, and

a predetermined number of the SUIs is generated and managed in an instance pool in the pull mode,

wherein the SO module comprises;

a user request handler (URH) that registers the user request and transmits the result of the user request to the user;

a scheduler that controls a processing schedule of the user request registered in the URH in response to a resource usage state;

a service unit instance manager (SUIM) that processes the user request by executing the SUI by processing the SUI in response to the processing schedule from the scheduler according to the execution mode and provides the result of the user request to the URH; and

a runtime monitor (RM) that monitors the resource usage state of each component and the SUI that is currently being executed, and provides the resource usage state to the scheduler,

wherein the SUIM generates one SUI per user request and the generated SUI becomes extinct after a requested time duration when the execution mode corresponds to the push mode.

10. The computer middleware system of claim 9, wherein the SD module comprises:

a service unit registry (SUR) that stores the SU;

a service unit handler (SUH) that performs registration of the SU on the SUR and processes the SUAL file written by the service provider; and

a service unit activator (SUA) that generates the query, the query result schema, and the execution mode information, the SUAL file being provided by the SUH.

11. The computer middleware system of claim 10, wherein the SUA manages a query list, merges an input query into an existing query when the result of the input query and the result of the existing query are identical to each other, and provides a new query to the stream provider requesting a partial result of the input query when result of the input query and the result of the existing query are partially overlapped.

12. The computer middleware system of claim 9, wherein the SH module comprises:

a data stream receiver (DSR) that transmits the query from the SD module the stream provider and receives the DS from the stream provider in response to the query; and

a buffer manager (BM) that assigns a buffer in memory in the response to the query result schema provided by the SD module, assigns the buffer to the corresponding query, stores data in the assigned buffer using the DS provided by the DSR, and provides the data to the SO module according to an execution mode.

13. The computer middleware system of claim 12, wherein the BM converts the DS to data conforming to an input data schema of a service logic (SL) and provides the converted data to the SO module.

14. The computer middleware system of claim 9, wherein the SUIM selects one SUI out of an instance pool to process the user request and returns the selected SUI into the instance pool when the execution mode corresponds to the pull mode.

15. A computer-readable storage medium storing a program that performs a method of processing a data stream (DS), the method comprising:

selecting an Stream Source Description Language (SSDL) file and a Service Unit (SU), the SSDL file and the SU being assigned by an Service Unit Activation Language (SUAL) file;

transmitting a query written by the selected SUAL file to a corresponding stream provider based on a stream provider address written by the selected SSDL file;

preparing a buffer according to schema information about a result of the transmitted query;

generating an instance of the selected SU;

initializing the selected SU into an execution mode assigned by the SUAL file;

updating, in real time, data stored in the prepared buffer by using the DS provided by the stream provider; and

processing the data stored in the buffer based on the execution mode in response to a user request to provide a result of the user request to a user by a service operator (SO) module,

wherein the SUAL file comprises a Service Source Description Language (SDDL) file, the SU, the query and the execution mode,

the execution mode comprises a push mode and a pull mode,

one instance of the selected SU per user request is generated and preserved during a requested time duration in the push mode, and

a predetermined number of the instances of the selected SU is generated and managed in an instance pool in the pull mode,

wherein the SO module comprises;

a user request handler (URH) that registers the user request and transmits the result of the user request to the user;

a scheduler that controls a processing schedule of the user request registered in the URH in response to a resource usage state;

a service unit instance manager (SUIM) that processes the user request by executing the SUI by processing the SUI in response to the processing schedule from the scheduler according to the execution mode and provides the result of the user request to the URH; and

a runtime monitor (RM) that monitors the resource usage state of each component and the SUI that is currently being executed, and provides the resource usage state to the scheduler,

wherein the SUIM generates one SUI per user request and the generated SUI becomes extinct after a requested time duration when the execution mode corresponds to the push mode.