×

PEER-TO-PEER ARCHITECTURE FOR PROCESSING BIG DATA

  • US 20150312335A1
  • Filed: 04/28/2015
  • Published: 10/29/2015
  • Est. Priority Date: 04/28/2014
  • Status: Active Application
First Claim
Patent Images

1. A system for managing large datasets comprising:

  • a physical network comprising a plurality of computing devices and a plurality of processors;

    a peer-to-peer (P2P) network, the P2P network comprising a plurality of nodes and a logical network derived from the physical network;

    a distributed file system for distributing data and jobs randomly across the plurality of nodes in the P2P network, by;

    receiving a file by an originating data node of the plurality of nodes, with a first processor of the plurality of processors assigned to the originating data node and being configured to;

    divide the file into a plurality of pages,assign a value to a first page of the plurality of pages, andtransfer the first page of the plurality of pages to an initial responsible data node, the initial responsible node including a name that closely matches the value of the first page of the plurality of pages;

    replicating the first page of the plurality of page from the initial responsible data node to a first responsible data node and a second responsible data node;

    receiving a job at an originating job node and dividing the job into a plurality of tasks;

    routing a first task of the plurality of tasks to an initial responsible job node; and

    assigning the first task of the plurality of tasks to a first processing node using the initial responsible job node; and

    a task scheduler for delegating the first task of the plurality of tasks as necessary to optimize load distribution, by;

    assigning the first task of the plurality of tasks to a first queue of at least two queues in the first processing node, andforwarding the first task of the plurality of tasks to a second processing node remote from the first processing node.

View all claims
  • 2 Assignments
Timeline View
Assignment View
    ×
    ×