Conserving computing resources during network parallel processing

• US 10,666,510 B2
• Filed: 10/30/2018
• Issued: 05/26/2020
• Est. Priority Date: 10/30/2018
• Status: Active Grant

• Current Assignee: Bank of America Corp.
• Original Assignee: Bank of America Corp.
• Inventors: Romano, Brad E., Thanikella, Shashi
• Primary Examiner(s): Zong, Ruolei
• Application Number:
  US16/174,614
• Publication Number:
  US 20200136899A1
• Time in Patent Office: 574 Days
• Field of Search: 709201
• CPC Class Codes: G06F 9/4843 by program, e.g. task dispatcher, supervisor, operating system
• CPC Class Codes: G06F 9/5072 Grid computing
• CPC Class Codes: H04L 41/0803 Configuration setting
• CPC Class Codes: H04L 41/0816 the condition being an adaptation, e.g. in response to network events
• CPC Class Codes: H04L 41/0846 based on copy from other elements
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

A parallel processing device includes a parallel processing engine implemented by a processor. The parallel processing engine is configured to execute a shell script for each particular processing job in a queue of processing jobs to run. The shell script is configured to dynamically generate a configuration file for each particular processing job. The configuration file instructs a network of computing systems to run the particular processing job using a particular number of parallel partitions corresponding to a parallel partitions parameter associated with the particular job. The configuration file includes randomized scratch directories for computing nodes within the network of computing systems and a calculated container size for the particular processing job. Each processing job is run on the network of computing systems according to the dynamically-generated configuration file of the particular processing job.

Claims

1. A parallel processing device, comprising:

one or more memory devices operable to store a queue of processing jobs to run; and

a parallel processing engine implemented b a processor communicatively coupled to the one or more memory devices, the parallel processing engine configured to;

access the queue of processing jobs to run; and

execute a shell script for each particular processing job in the queue of processing jobs to run, the shell script configured to;

access a queue size parameter associated with the particular processing job;

calculate a container size for the particular processing job based on the queue size parameter;

access a parallel partitions parameter associated, with the particular processing job;

access a configuration variable associated with the particular processing job, wherein the configuration variable indicates whether to dynamically generate a configuration file or not;

determine whether the configuration variable associated with the particular processing job matches a predetermined value;

in response to determining that the configuration variable associated with the particular processing job matches the predetermined value, dynamically generate a configuration file for the particular processing job, the configuration file configured to instruct a network of computing systems to run the particular processing job using a particular number of parallel partitions corresponding to the parallel partitions parameter, the configuration file comprising;

randomized scratch directories for computing nodes within the network of computing systems; and

the calculated container size for the particular processing job; and

trigger the particular processing job to run on the network of computing systems according to the dynamically-generated configuration file of the particular processing job,

in response to determining that the configuration variable associated with the particular processing job does not match the predetermined value, generate a static configuration file for the particular processing job to run on the network of computing systems.

2. The parallel processing device of claim 1, wherein the network of computing systems comprises a cluster of computers.

3. The parallel processing device of claim 1, wherein the shell script comprises a shell script.

4. The parallel processing device of claim 1, wherein the parallel processing engine comprises an extract, transform, load (ETL) process.

5. The parallel processing;

device of claim 1, wherein the parallel processing engine comprises a computer platform.

6. The parallel processing device of claim 1, wherein:

the number of parallel partitions for each processing job in the queue of processing jobs may be different such that the number of parallel partitions is customizable across the queue of processing jobs; and

the container size for each processing job in the queue of processing jobs may be different such that the container size is customizable, across the queue of processing jobs.

7. The parallel processing device of claim 1, wherein:

the queue size parameter comprises a particular size category chosen from a plurality of size categories; and

calculating the container size for the particular processing job based on the queue size parameter comprises determining a preassigned container size for the particular size category.

8. A parallel processing method, comprising:

accessing, by a parallel processing engine, a queue of processing jobs to run;

executing by the parallel processing engine, a shell script for each particular processing job in the queue of processing jobs to run;

accessing, by the shell Script, queue size parameter associated with the particular processing job;

calculating, by shell script, a container size for the particular processing job based on the queue size parameter;

accessing, by the shell script, parallel partitions parameter associated with the particular processing job;

accessing, by the shell script, a configuration variable associated with the particular processing job, wherein the configuration variable indicates whether to dynamically generate a configuration file or not;

determining, by the shell script, whether a configuration variable associated with each particular processing job matches a predetermined value;

in response to determining that the configuration variable associated with each particular processing job matches the predetermined value, dynamically generating a configuration file by the shell script for particular processing job, the configuration file configured to instruct a network of computing systems to run the particular processing job using a particular number of parallel partitions corresponding to the parallel partitions parameter, the configuration file comprising;

randomized scratch directories for computing nodes within the network of computing systems; and

the calculated container size for the particular processing job; and

triggering, the shell script, each particular processing JOB to run on the network of computing systems according to its associated dynamically-generated configuration file,

in response to determining that the configuration variable associated with the particular processing job does not match the predetermined value, generating a static configuration file for the particular procession job to run on the network of computing systems.

9. The parallel processing method of claim 8, wherein the network of computing systems comprises a cluster of computers.

10. The parallel processing method of claim 8, wherein the shell script comprises a shell script.

11. The parallel processing method of claim 8, wherein the parallel processing engine comprises an extract, transform, load (ETL) process.

12. The parallel processing method of claim 8, wherein the parallel processing engine comprises a computer platform.

13. The parallel processing method of claim 8, wherein:

the number of parallel partitions for each processing job in the queue of processing jobs may be different such that the number of parallel partitions is customizable across the queue of processing jobs; and

the container size for each processing job in the queue of processing jobs may be different such that the container size is customizable across the queue of processing jobs.

14. The parallel processing method of claim 8, wherein:

the queue size parameter comprises a particular size category chosen from a plurality of size categories; and

calculating the container size for the particular processing job based on the queue size parameter comprises determining a preassigned container size for the particular size category.

15. A computer program product comprising executable instructions stored in a non-transitory computer readable medium such that when executed by a processor causes the processor to implement a parallel processing engine configured to:

access a queue of processing jobs to run; and

execute a shell script for each particular processing job in the queue of processing jobs to run, the shell script configured to;

access a queue size parameter associated with the particular processing job;

calculate a container size for the particular processing job based on the queue size parameter;

access a parallel partitions parameter associated with the particular processing job;

access a configuration variable associated with the particular processing job, wherein the configuration variable indicates whether to dynamically generate a configuration file or not;

determine whether the configuration variable associated with the particular processing job matches a predetermined value;

in response to determining that the configuration variable associated with the particular processing job matches the predetermined value, dynamically generate a configuration file for the particular processing job, the configuration file configured to instruct a network of computing systems to run the particular processing job using a particular number of parallel partitions corresponding to the parallel partitions parameter, the configuration file comprising;

randomized scratch directories for computing nodes within the network of computing systems; and

the calculated container size for the particular processing job; and

trigger the particular processing job to run on the network of computing systems according to the dynamically-generated configuration file of the particular processing job,

in response to determining that the configuration variable associated with the particular processing job does not match the predetermined value, generate a static configuration file for the particular procession job to run on the network of computing systems.

16. The computer program product of claim 15, wherein the network of computing systems comprises a cluster of computers.

17. The computer program product of claim 15, wherein the shell script comprises a shell script.

18. The computer program product of claim 15, wherein the parallel processing engine comprises an extract, transform, load (ETL) process.

19. The computer program product of claim 15, wherein:

the number of parallel partitions for each processing job m the queue of processing jobs may be different such that the number of parallel partitions is customizable across the queue of processing lobs; and

the container size for each processing job in the queue of processing jobs may be different such that the container size is customizable across the queue of processing jobs.

20. The computer program product of claim 15, wherein:

the queue size parameter comprises a particular size category chosen from a plurality of size categories; and

calculating the container size for the particular processing job based on the queue size parameter comprises determining a preassigned container size for the particular size category.