×

Techniques for Reading From and Writing to Distributed Data Stores

  • US 20180288154A1
  • Filed: 07/12/2017
  • Published: 10/04/2018
  • Est. Priority Date: 04/02/2017
  • Status: Active Grant
First Claim
Patent Images

1. A system for writing files to a distributed file system, comprising:

  • one or more processors; and

    a non-transitory computer readable storage medium including instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including;

    receiving a request to write a file to a distributed file system, wherein the distributed file system corresponds to a plurality of data blocks distributed across a plurality of nodes;

    partitioning the file into a plurality of file-parts;

    assigning each of the plurality of file-parts to a file-part queue;

    instantiating, at each of multiple nodes, a plurality of write tasks for completing the request to write the file to the distributed file system, wherein write tasks correspond to processes for writing data blocks to the distributed file system using pluralities of threads, and wherein data blocks include multiple data records; and

    processing, in parallel, each plurality of write tasks, wherein processing each write task includes;

    instantiating, for the write task, a plurality of threads for writing file-parts to the distributed file system; and

    processing each of the plurality of threads in parallel, wherein processing each thread includes;

    retrieving a file-part assignment from the file-part queue, wherein the file-part assignment corresponds to a particular file-part;

    obtaining a data record from a data buffer associated with the file, wherein the data record corresponds to a portion of the particular file-part; and

    writing the data record to a data block associated with local storage of a particular node on which the thread is processing.

View all claims
  • 1 Assignment
Timeline View
Assignment View
    ×
    ×