Techniques for Reading From and Writing to Distributed Data Stores

• US 20180288154A1
• Filed: 07/12/2017
• Published: 10/04/2018
• Est. Priority Date: 04/02/2017
• Status: Active Grant

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: Ghazaleh, David Abu
• Granted Patent: US 10,803,024 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/1441 Resetting or repowering
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/2023 Failover techniques
• CPC Class Codes: G06F 11/2041 with more than one idle spare processing component
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 12/084 with a shared cache
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1858 Parallel file systems, i.e. file systems supporting multiple processors
• CPC Class Codes: G06F 2212/1024 Latency reduction
• CPC Class Codes: G06F 2212/154 Networked environment
• CPC Class Codes: G06F 2212/608 Details relating to cache mapping
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/0661 Format or protocol conversion arrangements
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/568 : Storing data temporarily at an intermediate stage, e.g. caching
• CPC Class Codes: H04L 67/5681 : Pre-fetching or pre-delivering data based on network characteristics

Abstract

Described herein are techniques for reading data from a distributed storage system and for writing data to a distributed storage system. The disclosed techniques make use of efficient computing task and thread usage to minimize or reduce overhead and improve read or write efficiency. For example, read or write tasks may handle multiple read or write operations instead of just a single operation, which may reduce overhead associated with task creation and termination. Additionally, operations within a single task may be processed in parallel. For example, the disclosed techniques provide MapReduce implementations useful in Apache Hadoop that perform better than previous MapReduce implementations.

Claims

1. A system for writing files to a distributed file system, comprising:

one or more processors; and

a non-transitory computer readable storage medium including instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including;

receiving a request to write a file to a distributed file system, wherein the distributed file system corresponds to a plurality of data blocks distributed across a plurality of nodes;

partitioning the file into a plurality of file-parts;

assigning each of the plurality of file-parts to a file-part queue;

instantiating, at each of multiple nodes, a plurality of write tasks for completing the request to write the file to the distributed file system, wherein write tasks correspond to processes for writing data blocks to the distributed file system using pluralities of threads, and wherein data blocks include multiple data records; and

processing, in parallel, each plurality of write tasks, wherein processing each write task includes;

instantiating, for the write task, a plurality of threads for writing file-parts to the distributed file system; and

processing each of the plurality of threads in parallel, wherein processing each thread includes;

retrieving a file-part assignment from the file-part queue, wherein the file-part assignment corresponds to a particular file-part;

obtaining a data record from a data buffer associated with the file, wherein the data record corresponds to a portion of the particular file-part; and

writing the data record to a data block associated with local storage of a particular node on which the thread is processing.

2. The system of claim 1, wherein the operations further comprise updating a data block distribution map for the distributed file system to indicate which data blocks are locally stored by each node.

3. The system of claim 2, wherein the data block distribution map corresponds to a split distribution map, and wherein each data block is associated with a split.

4. The system of claim 1, wherein processing each thread includes repeating, until all data records associated with the file-part of the file-part assignment are written to the data block associated with the local storage of the particular node on which the thread is processing:

obtaining a next data record from the data buffer associated with the file, wherein the next data record is associated with the file-part of the file-part assignment; and

writing the next data record to the data block associated with the local storage of the particular node on which the thread is processing.

5. The system of claim 1, wherein processing each thread includes repeating, without terminating the thread until all file-part assignments from the file-part queue are retrieved:

retrieving a next file-part assignment from the file-part queue;

obtaining a next data record from the data buffer associated with the file, wherein the next data record is associated with a next file-part of the next file-part assignment; and

writing the next data record to a next data block associated with the local storage of the particular node on which the thread is processing.

6. The system of claim 1, wherein a maximum number of write tasks instantiated at each node is configurable.

7. The system of claim 1, wherein a maximum number of threads instantiated by each write task is configurable.

8. The system of claim 1, wherein the distributed file system corresponds to a Hadoop Distributed File System.

9. The system of claim 1, wherein each write task corresponds to a custom MapReduce task.

10. The system of claim 1, wherein writing the data record to storage of the distributed file system includes instantiating an HCatalog writer object, calling the HCatalog writer object, and writing the data record to the data block using the HCatalog writer object.

11. A computer-program product for writing files to a distributed file system, the computer-program product tangibly embodied in a non-transitory computer readable storage medium comprising instructions configured to, when executed by one or more processors, cause the one or more processors to perform operations including:

receiving a request to write a file to a distributed file system, wherein the distributed file system corresponds to a plurality of data blocks distributed across a plurality of nodes;

partitioning the file into a plurality of file-parts;

assigning each of the plurality of file-parts to a file-part queue;

instantiating, at each of multiple nodes, a plurality of write tasks for completing the request to write the file to the distributed file system, wherein write tasks correspond to processes for writing data blocks to the distributed file system using pluralities of threads, and wherein data blocks include multiple data records; and

processing, in parallel, each plurality of write tasks, wherein processing each write task includes;

instantiating, for the write task, a plurality of threads for writing file-parts to the distributed file system; and

processing each of the plurality of threads in parallel, wherein processing each thread includes;

retrieving a file-part assignment from the file-part queue, wherein the file-part assignment corresponds to a particular file-part;

obtaining a data record from a data buffer associated with the file, wherein the data record corresponds to a portion of the particular file-part; and

writing the data record to a data block associated with local storage of a particular node on which the thread is processing.

12. The computer-program product of claim 11, wherein the operations further comprise updating a data block distribution map for the distributed file system to indicate which data blocks are locally stored by each node.

13. The computer-program product of claim 12, wherein the data block distribution map corresponds to a split distribution map, and wherein each data block is associated with a split.

14. The computer-program product of claim 11, wherein processing each thread includes repeating, until all data records associated with the file-part of the file-part assignment are written to the data block associated with the local storage of the particular node on which the thread is processing:

obtaining a next data record from the data buffer associated with the file, wherein the next data record is associated with the file-part of the file-part assignment; and

writing the next data record to the data block associated with the local storage of the particular node on which the thread is processing.

15. The computer-program product of claim 11, wherein processing each thread includes repeating, without terminating the thread until all file-part assignments from the file-part queue are retrieved:

retrieving a next file-part assignment from the file-part queue;

obtaining a next data record from the data buffer associated with the file, wherein the next data record is associated with a next file-part of the next file-part assignment; and

writing the next data record to a next data block associated with the local storage of the particular node on which the thread is processing.

16. The computer-program product of claim 11, wherein a maximum number of write tasks instantiated at each node is configurable.

17. The computer-program product of claim 11, wherein a maximum number of threads instantiated by each write task is configurable.

18. The computer-program product of claim 11, wherein the distributed file system corresponds to a Hadoop Distributed File System.

19. The computer-program product of claim 11, wherein each write task corresponds to a custom MapReduce task.

20. The computer-program product of claim 11, wherein writing the data record to storage of the distributed file system includes instantiating an HCatalog writer object, calling the HCatalog writer object, and writing the data record to the data block using the HCatalog writer object.

21. A computer implemented method for writing files to a distributed file system, comprising:

receiving a request to write a file to a distributed file system, wherein the distributed file system corresponds to a plurality of data blocks distributed across a plurality of nodes;

partitioning the file into a plurality of file-parts;

assigning each of the plurality of file-parts to a file-part queue;

instantiating, at each of multiple nodes, a plurality of write tasks for completing the request to write the file to the distributed file system, wherein write tasks correspond to processes for writing data blocks to the distributed file system using pluralities of threads, and wherein data blocks include multiple data records; and

processing, in parallel, each plurality of write tasks, wherein processing each write task includes;

instantiating, for the write task, a plurality of threads for writing file-parts to the distributed file system; and

processing each of the plurality of threads in parallel, wherein processing each thread includes;

retrieving a file-part assignment from the file-part queue, wherein the file-part assignment corresponds to a particular file-part;

obtaining a data record from a data buffer associated with the file, wherein the data record corresponds to a portion of the particular file-part; and

writing the data record to a data block associated with local storage of a particular node on which the thread is processing.

22. The method of claim 21, further comprising updating a data block distribution map for the distributed file system to indicate which file parts data blocks are locally stored by each node.

23. The method of claim 22, wherein the data block distribution map corresponds to a split distribution map, and wherein each data block is associated with a split.

24. The method of claim 21, wherein processing each thread includes repeating, until all data records associated with the file-part of the file-part assignment are written to the data block associated with the local storage of the particular node on which the thread is processing:

obtaining a next data record from the data buffer associated with the file, wherein the next data record is associated with the file-part of the file-part assignment; and

writing the next data record to the data block associated with the local storage of the particular node on which the thread is processing.

25. The method of claim 21, wherein processing each thread includes repeating, without terminating the thread until all file-part assignments from the file-part queue are retrieved:

retrieving a next file-part assignment from the file-part queue;

obtaining a next data record from the data buffer associated with the file, wherein the next data record is associated with a next file-part of the next file-part assignment; and

writing the next data record to a next data block associated with the local storage of the particular node on which the thread is processing.

26. The method of claim 21, wherein a maximum number of write tasks instantiated at each node is configurable.

27. The method of claim 21, wherein a maximum number of threads instantiated by each write task is configurable.

28. The method of claim 21, wherein the distributed file system corresponds to a Hadoop Distributed File System.

29. The method of claim 21, wherein each write task corresponds to a custom MapReduce task.

30. The method of claim 21, wherein writing the data record to storage of the distributed file system includes instantiating an HCatalog writer object, calling the HCatalog writer object, and writing the data record to the data block using the HCatalog writer object.