EXTRACTION DEVICE, DATA PROCESSING SYSTEM, AND EXTRACTION METHOD

• US 20150293981A1
• Filed: 06/23/2015
• Published: 10/15/2015
• Est. Priority Date: 12/13/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Imai, Haruki, Komatsu, Hideaki, Koseki, Akira, Takase, Toshiro
• Granted Patent: US 10,089,370 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database

Abstract

An extraction method for extracting a sub query to be converted to a program for processing stream data continuously inputted to a database, from a query including instructions, as sub queries, to be issued to a database management system. The extraction method includes receiving input of the query and a lower limit value of efficiency as processing time per unit memory increase amount. A calculating operation calculates a one memory increase amount and the efficiency using the memory increase amount and the processing time to be reduced. The method selects a sub query whose calculated efficiency is equal to or higher than the lower limit value and extracts the selected sub query as a conversion object on condition that the integrated memory increase amount is equal to or smaller than the maximum memory increase amount.

Claims

1. An extraction method of extracting, as a conversion object, a sub query to be converted to a program for processing stream data continuously inputted to a database, from a query including one or more instructions as sub queries to be issued to a database management system for managing the database, the method comprising the steps of:

receiving input of the query, a maximum memory increase amount of a memory whose use amount increases with the processing of the stream data, and a lower limit value of efficiency as processing time per unit memory increase amount, the processing time being reduced in a case where the stream data is processed by the program compared to a case where the database management system executes the query;

calculating by a computer processor at least one memory increase amount in a case where each sub query included in the query is converted to the program and then the stream data is processed and the processing time to be reduced in a case where the stream data is processed compared to the case of execution by the database management system, and calculating the efficiency by using the at least one calculated memory increase amount and the processing time to be reduced; and

selecting at least one sub query whose calculated efficiency is equal to or higher than the lower limit value, integrating the memory increase amount calculated for the selected sub query, and extracting the selected sub query as a conversion object on condition that the integrated memory increase amount is equal to or smaller than the maximum memory increase amount.

2. The extraction method according to claim 1, wherein, in the extraction step, the sub query is selected in a descending order of the efficiency calculated in the calculation step.

3. The extraction method according to claim 1, further comprising the steps of:

after the step of receiving the input, parsing the query and generating a graph expressed by a tree structure in which each sub query is each node; and

determining whether each node is a first node capable of pipeline processing of processing inputted data and outputting a processing result or a second node incapable of the pipeline processing on the basis of the result of parsing the query, and with reference to the generated graph, classifying the nodes into one or more first node groups in which the nodes hierarchically connected from a root node are formed only of the first nodes and one or more second node groups formed of the remaining nodes;

wherein, in the extraction step, a node whose efficiency is equal to or higher than the lower limit value is selected from the nodes classified into the first node group and is extracted as the conversion object.

4. The extraction method according to claim 3, wherein:

in the classification step, with reference to the graph, the one or more second node groups are classified into one or more third node groups in which the nodes hierarchically connected from a leaf node are formed only of the first nodes and one or more fourth node groups formed of the remaining nodes; and

in the extraction step, after selecting the nodes whose efficiency is equal to or higher than the lower limit value from the nodes classified into the first node group and extracting all the conversion objects, if the integrated memory increase amount has not reached the maximum memory increase amount, nodes whose efficiency is equal to or higher than the lower limit value are selected from the nodes classified into the third node group and are extracted as the conversion objects.

5. The extraction method according to claim 1, wherein, in the calculation step, the memory increase amount and the processing time to be reduced are calculated on the basis of a data rate of the stream data, a time period of the processing of the stream data, statistical information obtained from the result of executing the query in the past, and information about an operation of the query.