Data processing method and computer system

• US 8,583,757 B2
• Filed: 05/31/2010
• Issued: 11/12/2013
• Est. Priority Date: 05/31/2010
• Status: Expired due to Fees

• Current Assignee: Hitachi, Ltd.
• Original Assignee: Hitachi, Ltd.
• Inventors: Takaoka, Nobumitsu, Iwasaki, Masaaki, Utsunomiya, Naoki, Kano, Yoshiki
• Primary Examiner(s): Nguyen, Phuoc
• Assistant Examiner(s): SISON, JUNE Y
• Application Number:
  US12/810,707
• Publication Number:
  US 20110295968A1
• Time in Patent Office: 1,261 Days
• Field of Search: 709/213
• US Class Current: 709/213
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor

Abstract

A technique for increasing the speed of data entry into a distributed processing platform is provided. According to a computer system of the present invention, when data is entered into each node in a distributed manner, the most efficient entry method (a method with the highest processing speed) is selected from among a plurality of entry methods, so that the data is entered into each node with no overlaps in accordance with the selected method.

Claims

1. A data processing method of a computer system, the computer system including a data entry server, a plurality of nodes connected to the data entry server via a first network, and a storage device connected to the data entry server and the plurality of nodes via a second network, the storage device including a logical volume having stored therein at least data as data sources that can be read by each of the plurality of nodes, the method comprising:

causing the data entry server to estimate a processing time required to execute each of first and second data entry methods, and to select one of the first and second data entry methods based on the estimated processing time; and

causing the data entry server or the plurality of nodes to execute data entry processing based on the selected data entry method,

wherein the first data entry method is a method in which the data entry server reads data from the logical volume, divides the data into a plurality of blocks, and sends the data blocks to the plurality of nodes with no overlaps,

wherein the second data entry method is a method in which the plurality of nodes, in response to an instruction from the data entry server, reads the data from the logical volume with no overlaps by dividing the data into a plurality of blocks, and

wherein the storage device storing the logical volume is separated from the plurality of nodes so that the logical volume does not constitute a distributed file system which is composed of the plurality of nodes.

2. The data processing method according to claim 1,

wherein the first network is a local area network and the second network is a storage area network,

wherein the first data entry method is a method in which the data entry server obtains all data in a given column of a database included in the logical volume, identifies data to be read using the data in the column, obtains the identified data from the database, and distributes the obtained data to the plurality of nodes,

wherein the second data entry method includes:

(i) a second A method in which the data entry server obtains all data in a given column of the database included in the logical volume, identifies data to be read using the data in the column, and sends information on the identified data to the plurality of nodes, so that the plurality of nodes obtains the identified data from the database, and

(ii) a second B method in which the plurality of nodes obtains all data in a given column of the database included in the logical volume, identifies data to be read using the data in the column, and obtains the identified data from the database, and

wherein the data entry server estimates a processing time required to execute each of the first, second A, and second B data entry methods using throughput of the local area network, throughput of the storage area network, throughput of the logical volume, the total volume of data to be sent to the plurality of nodes, and the total volume of data to be read from the logical volume, and selects a data entry method with the shortest estimated processing time from among the first, second A, and second B data entry methods.

3. The data processing method according to claim 1,

wherein the first data entry method is a method in which the data entry server obtains all data in a given column of a database included in the logical volume, identifies data to be read using the data in the column, obtains the identified data from the database, and distributes the obtained data to the plurality of nodes,

wherein the second data entry method includes:

(i) a second A method in which the data entry server obtains all data in a given column of the database included in the logical volume, identifies data to be read using the data in the column, and sends information on the identified data to the plurality of nodes, so that the plurality of nodes obtains the identified data from the database, and

(ii) a second B method in which the plurality of nodes obtains all data in a given column of the database included in the logical volume, identifies data to be read using the data in the column, and obtains the identified data from the database, and

wherein the data entry server estimates a processing time required to execute each of the first, second A, and second B data entry methods, and selects one of the first, second A, and second B data entry methods based on the estimated processing time.

4. The data processing method according to claim 1, wherein the data entry server estimates the processing time required to execute each of the first and second data entry methods using throughput of the first and second networks, throughput of the logical volume, the total volume of data to be sent to the plurality of nodes, and the total volume of data to be read from the logical volume.

5. The data processing method according to claim 1,

wherein the second storage device includes replicas of the logical volume, and

wherein the data entry server calculates times required to read data from the logical volume and from the replicas of the logical volume in parallel, and estimates the processing time required to execute each of the first and second data entry methods using the calculated times.

6. The data processing method according to claim 1,

wherein the data stored in the logical volume includes dereference information for facilitating identification of the data to be read, and

wherein the data entry server calculates a time required to obtain the data to be read using the dereference information, and estimates the processing time required to execute each of the first and second data entry methods using the calculated time.

7. The data processing method according to claim 1,

wherein the computer system includes a plurality of storage devices,

wherein the data to be read is stored across the plurality of storage devices, and

wherein the data entry server calculates a time required to read the data from the plurality of storage devices in parallel, and estimates the processing time required to execute each of the first and second data entry methods using the calculated time.

8. A computer system comprising:

a data entry server;

a plurality of nodes connected to the data entry server via a first network; and

a storage device connected to the data entry server and the plurality of nodes via a second network, the storage device including a logical volume having stored therein at least data as data sources that can be read by each of the plurality of nodes, and the storage device being a non-transitory, tangible computer-readable storage medium,

wherein the storage device storing the logical volume is separated from the plurality of nodes so that the logical volume does not constitute a distributed file system which is composed of the plurality of nodes,

wherein the data entry server estimates a processing time required to execute each of first and second data entry methods, and selects one of the first and second data entry methods based on the estimated processing time,

wherein the data entry server or the plurality of nodes executes data entry processing based on the selected data entry method,

wherein the first data entry method is a method in which the data entry server reads data from the logical volume, divides the data into a plurality of blocks, and sends the data blocks to the plurality of nodes with no overlaps, and

wherein the second data entry method is a method in which the plurality of nodes, in response to an instruction from the data entry server, reads the data from the logical volume with no overlaps by dividing the data into a plurality of blocks.

9. The computer system according to claim 8,

wherein the first network is a local area network and the second network is a storage area network,

wherein the first data entry method is a method in which the data entry server obtains all data in a given column of a database included in the logical volume, identifies data to be read using the data in the column, obtains the identified data from the database, and distributes the obtained data to the plurality of nodes, and

wherein the second data entry method includes:

(i) a second A method in which the data entry server obtains all data in a given column of the database included in the logical volume, identifies data to be read using the data in the column, and sends information on the identified data to the plurality of nodes, so that the plurality of nodes obtains the identified data from the database, and

(ii) a second B method in which the plurality of nodes obtains all data in a given column of the database included in the logical volume, identifies data to be read using the data in the column, and obtains the identified data from the database, and

wherein the data entry server estimates a processing time required to execute each of the first, second A, and second B data entry methods using throughput of the local area network, throughput of the storage area network, throughput of the logical volume, the total volume of data to be sent to the plurality of nodes, and the total volume of data to be read from the logical volume, and selects a data entry method with the shortest estimated processing time from among the first, second A, and second B data entry methods.

10. The computer system according to claim 8,

wherein the first data entry method is a method in which the data entry server obtains all data in a given column of the database included in the logical volume, identifies data to be read using the data in the column, obtains the identified data from the database, and distributes the obtained data to the plurality of nodes,

wherein the second data entry method includes:

(i) a second A method in which the data entry server obtains all data in a given column of the database included in the logical volume, identifies data to be read using the data in the column, and sends information on the identified data to the plurality of nodes, so that the plurality of nodes obtains the identified data from the database, and

(ii) a second B method in which the plurality of nodes obtains all data in a given column of the database included in the logical volume, identifies data to be read using the data in the column, and obtains the identified data from the database, and

wherein the data entry server estimates a processing time required to execute each of the first, second A, and second B data entry methods, and selects one of the first, second A, and second B data entry methods based on the estimated processing time.

11. The computer system according to claim 8, wherein the data entry server estimates the processing time required to execute each of the first and second data entry methods using throughput of the first and second networks, throughput of the logical volume, the total volume of data to be sent to the plurality of nodes, and the total volume of data to be read from the logical volume.

12. The computer system according to claim 8,

wherein the second storage device includes replicas of the logical volume, and

wherein the data entry server calculates times required to read data from the logical volume and from the replicas of the logical volume in parallel, and estimates the processing time required to execute each of the first and second data entry methods using the calculated times.

13. The computer system according to claim 8,

wherein the data stored in the logical volume includes dereference information for facilitating identification of the data to be read, and

wherein the data entry server calculates a time required to obtain the data to be read using the dereference information, and estimates the processing time required to execute each of the first and second data entry methods using the calculated time.

14. The computer system according to claim 8,

wherein the computer system includes a plurality of storage devices,

wherein the data to be read is stored across the plurality of storage devices, and

wherein the data entry server calculates a time required to read the data from the plurality of storage devices in parallel, and estimates the processing time required to execute each of the first and second data entry methods using the calculated time.

15. The computer system according to claim 8, further comprising:

a management server that displays a user interface configured to instruct the data entry server to enter data into the plurality of nodes.