Dividing and parallel processing record sets using a plurality of sub-tasks executing across different computers

• US 9,430,285 B2
• Filed: 10/29/2013
• Issued: 08/30/2016
• Est. Priority Date: 10/31/2012
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Nishimura, Shoji
• Primary Examiner(s): WU, BENJAMIN C
• Application Number:
  US14/438,745
• Publication Number:
  US 20150254105A1
• Time in Patent Office: 1,036 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 2209/5017 Task decomposition
• CPC Class Codes: G06F 9/4881 Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
• CPC Class Codes: G06F 9/4887 involving deadlines, e.g. rate based, periodic
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451
• CPC Class Codes: G06F 9/5088 involving task migration

Abstract

Disclosed are a data processing system, etc., whereby total processing time of a data process including a parallel process and a serial process is optimized. This data processing system comprises: a task segmenting means for receiving a plurality of tasks which execute, upon a record set which is a set of a plurality of records, a data process which includes parallel processes which a plurality of computers are capable of executing in parallel and serial processes which are post-processes of the parallel processes and is executed by one computer, and, for a parallel process which is the pre-process of a serial process, among the serial processes which are included in the task, which is envisioned as requiring more time to process than another serial process, by subdividing the record set which is to be processed in the parallel process, segmenting the parallel process into a plurality of sub-tasks; and an assignment means for assigning the plurality of sub-tasks to the plurality of computers, such that the plurality of sub-tasks are executed prior to a task other than the task which includes the parallel processes being executed.

Claims

1. A data processing system comprising:

hardware, including a processor and memory;

a task dividing unit implemented by the hardware for receiving a plurality of tasks in each of which data processing, including a parallel processing that can be performed by a plurality of computers in parallel and a serial processing that is a post-processing of the parallel processing and performed by one computer, is performed to a record set that is a set of a plurality of records, and

for dividing the parallel processing that is a preprocessing of the serial processing to which an expected processing time thereof is greater than that of the serial processing included in another task into a plurality of sub tasks by subdividing the record set that is a processing target of the parallel processing; and

an assignment unit implemented by the hardware for assigning the plurality of the sub tasks to the plurality of the computers so that the plurality of the sub tasks are performed before a task other than the task including the parallel processing is performed,

wherein when an expected processing time of a task in the plurality of tasks is longer than an expected processing time of the serial processing with the longest processing time, the parallel processing included in the task is then divided.

2. The data processing system described in claim 1 further comprising:

a job reception unit implemented by the hardware for receiving a job in which the data processing is performed to the data set that is the set of a plurality of records; and

a job division unit implemented by the hardware for dividing the job into the plurality of tasks by subdividing the data set that is a processing target of the job into a plurality of record sets, wherein the task dividing unit receives the plurality of the divided tasks.

3. The data processing system described in claim 1, wherein the task dividing unit divides the parallel processing so as to differentiate the number of the records processed by each of the sub tasks, and

wherein the assignment unit assigns a sub task of which the number of the records processed therein is greater than the number of the records processed in another sub task to a computer to which the serial processing is assigned.

4. The data processing system described in claim 1, wherein the assignment unit assigns a sub task for processing a first record in the serial processing to the computer to which the serial processing is assigned.

5. The data processing system described in claim 1, wherein the assignment unit outputs schedule data in which an assigned computer is associated with a priority to each of the tasks and sub tasks, and the serial processing that is the post-processing of the sub tasks, wherein the assigned computer includes:

a schedule data acquisition unit executing on the assigned computer for acquiring the schedule data, and

a task processing unit executing on the assigned computer for performing an assigned task in a decreasing order of the priority included in the schedule data; and

when the assigned task has already been performed and when another task assigned to another computer has not been performed, the task processing unit performs the task assigned to the other computer.

6. The data processing system described in claim 1, wherein the assignment unit outputs schedule data in which an assigned computer is associated with a record that is the processing target to each of the tasks and sub tasks, and the serial processing that is the post-processing of the sub tasks, and wherein the assigned computer includes:

a record acquisition unit executing on the assigned computer for acquiring the record,

a schedule data acquisition unit executing on the assigned computer for acquiring the schedule data, and

a record transmission unit executing on the assigned computer for transmitting the acquired record to another assigned computer.

7. A data processing method including:

a division step in which a plurality of tasks in each of which data processing, including a parallel processing that can be performed by a plurality of computers in parallel and a serial processing that is a post-processing of the parallel processing and performed by one computer, is performed to a record set that is a set of a plurality of records are received, and the parallel processing that is a preprocessing of the serial processing to which an expected processing time thereof is greater than that of the serial processing included in another task is divided into a plurality of sub tasks by subdividing the record set that is a processing target of the parallel processing; and

an assignment step in which the plurality of the sub tasks are assigned to the plurality of the computers so that the plurality of the sub tasks are performed before a task other than the task including the parallel processing is performed,

wherein when an expected processing time of a task in the plurality of tasks is longer than an expected processing time of the serial processing with the longest processing time, the parallel processing included in the task is then divided.

8. A non-transitory computer readable medium storing a program that causes a computer serving as an data processing system to execute a process to:

perform a division process in which a plurality of tasks in each of which a data processing, including a parallel processing that can be performed by a plurality of computers in parallel and a serial processing that is a post-processing of the parallel processing and performed by one computer, is performed to a record set that is a set of a plurality of records are received, and the parallel processing that is a preprocessing of the serial processing to which an expected processing time thereof is greater than that of the serial processing included in another task is divided into a plurality of sub tasks by subdividing the record set that is a processing target of the parallel processing; and

an assignment process in which the plurality of the sub tasks are assigned to the plurality of the computers so that the plurality of the sub tasks are performed before a task other than the task including the parallel processing is performed,

wherein when an expected processing time of a task in the plurality of tasks is longer than an expected processing time of the serial processing with the longest processing time, the parallel processing included in the task is then divided.