×

Conserving Computing Resources during Network Parallel Processing

  • US 20200136899A1
  • Filed: 10/30/2018
  • Published: 04/30/2020
  • Est. Priority Date: 10/30/2018
  • Status: Active Grant
First Claim
Patent Images

1. A parallel processing device, comprising:

  • one or more memory devices operable to store a queue of processing jobs to run; and

    a parallel processing engine implemented by a processor communicatively coupled to the one or more memory devices, the parallel processing engine configured to;

    access the queue of processing jobs to run; and

    execute a shell script for each particular processing job in the queue of processing jobs to run, the shell script configured to;

    access a queue size parameter associated with the particular processing job;

    calculate a container size for the particular processing job based on the queue size parameter;

    access a parallel partitions parameter associated with the particular processing job;

    access a configuration variable associated with the particular processing job;

    determine whether the configuration variable associated with the particular processing job matches a predetermined value;

    in response to determining that the configuration variable associated with the particular processing job matches the predetermined value, dynamically generate a configuration file for the particular processing job, the configuration file configured to instruct a network of computing systems to run the particular processing job using a particular number of parallel partitions corresponding to the parallel partitions parameter, the configuration file comprising;

    randomized scratch directories for computing nodes within the network of computing systems; and

    the calculated container size for the particular processing job; and

    trigger the particular processing job to run on the network of computing systems according to the dynamically-generated configuration file of the particular processing job.

View all claims
  • 1 Assignment
Timeline View
Assignment View
    ×
    ×