×

Dynamic node group allocation

  • US 9,762,672 B2
  • Filed: 06/15/2015
  • Issued: 09/12/2017
  • Est. Priority Date: 06/15/2015
  • Status: Expired due to Fees
First Claim
Patent Images

1. A method, comprising:

  • connecting a parallel application server to a data source structure, wherein the data source structure contains a big data distributed file system, wherein the big data distributed file system contains multiple nodes and data blocks;

    in response to the parallel application operating the data source structure within the multiple nodes of the big data distributed file system, the parallel application server and the data source structure performing read and write operations on the data blocks in a local mode setting;

    in response to the parallel application operating the data source structure outside of the multiple nodes of the big data distributed file system, the parallel application server and the data source structure performing read and write operations on the data blocks in a remote mode setting;

    in response to a consumer job starting to read one or more files in the big data distributed file system, retrieving node group information for the one or more files to be read, wherein the node group information identifies nodes from the multiple nodes on which a producer job wrote the one or more files;

    implementing a node grouping mechanism to read and write the data blocks within the local mode setting over the remote mode setting;

    assigning the consumer job to the nodes identified by the node group information to allow for reading of the one or more files by the consumer job within the local mode setting, wherein the local mode setting reads and writes the data blocks;

    in response to assigning the consumer job to the nodes identified by the node group information, generating a configuration file, wherein the configuration file comprises a dynamically generated configuration file and a non-dynamically generated configuration file;

    wherein the dynamically generated configuration file corresponds to the consumer job and the dynamically generated configuration file is dynamically assigned to the node group for the consumer job;

    in response to retrieving the node group information, requesting logical resources;

    executing the consumer job with the configuration file identifying the nodes on which the consumer job is to run; and

    in response to determining that logical resources cannot be allocated in the nodes identified by the node group information, attempting to allocate logical resources in nodes close to the nodes identified by the node group information.

View all claims
  • 1 Assignment
Timeline View
Assignment View
    ×
    ×