Memory optimized data shuffle

• US 10,642,520 B1
• Filed: 04/18/2017
• Issued: 05/05/2020
• Est. Priority Date: 04/18/2017
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Zhao, Junping, Taylor, Kenneth J., Shain, Randall, Wang, Kun
• Primary Examiner(s): Rojas, Midys
• Application Number:
  US15/490,430
• Time in Patent Office: 1,113 Days
• Field of Search: None
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0638 Organizing or formatting or addressing of data
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0655 Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
• CPC Class Codes: G06F 3/0656 Data buffering arrangements
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0683 Plurality of storage devices

Abstract

In a distributed data processing system with a set of multiple nodes, a first data shuffle memory pool is maintained at a data shuffle writer node, and a second data shuffle memory pool is maintained at a data shuffle reader node. The data shuffle writer node and the data shuffle reader node are part of the set of multiple nodes of the distributed data processing system. In-memory compression is performed on at least a portion of a data set from the first data shuffle memory pool. At least a portion of the compressed data is transmitted from the first data shuffle memory pool to the second data shuffle memory pool in a peer-to-peer manner. Each of the first data shuffle memory pool and the second data shuffle memory pool may include a hybrid memory configuration.

Claims

1. A method comprising:

maintaining a first data shuffle memory pool at a data shuffle writer node and a second data shuffle memory pool at a data shuffle reader node, wherein the data shuffle writer node and the data shuffle reader node are part of a set of multiple nodes of a distributed data processing system;

performing an in-memory compression on at least a portion of a data set from the first data shuffle memory pool;

performing a data shuffle operation on the at least a portion of the compressed data from the first shuffle memory pool, wherein the data shuffle operation maps different parts of the compressed data for transmission to different nodes of the distributed data processing system; and

transmitting, in response to the data shuffle operation, the at least a portion of the compressed data from the first data shuffle memory pool to the second data shuffle memory pool in a peer-to-peer manner;

wherein the distributed data processing system is implemented via one or more processing devices.

2. The method of claim 1, wherein the performance of the in-memory compression is triggered by a given policy.

3. The method of claim 2, wherein the given policy is based on a memory pressure associated with a memory capacity threshold.

4. The method of claim 2, wherein the given policy is based on a ratio of incoming write speed to outgoing read speed.

5. The method of claim 1, wherein performing the in-memory compression further comprises:

obtaining one or more records;

compressing the obtained records into a work buffer;

copying the compressed records to an original location of the one or more compressed records; and

updating an active location, the active location indicating the beginning location of a next non-compressed record.

6. The method of claim 1, wherein each of the first data shuffle memory pool and the second data shuffle memory pool comprise a hybrid memory configuration.

7. The method of claim 6, wherein the hybrid memory configuration comprises a dynamic random access memory tier and one or more nonvolatile random access memory tiers.

8. The method of claim 7, wherein the one or more nonvolatile random access memory tiers comprise a local nonvolatile random access memory tier and a remote nonvolatile random access memory tier.

9. The method of claim 7, wherein the one or more nonvolatile random access memory tiers comprise a global nonvolatile random access memory tier.

10. The method of claim 1, wherein each of the first data shuffle memory pool and the second data shuffle memory pool comprise a global shared memory tier.

11. The method of claim 1, wherein data in the data set is characterized by the data shuffle writer node as one or more of:

data to be transmitted to the data shuffle reader node;

data to be infrequently locally accessed; and

data to be sorted and aggregated as one or more extents.

12. The method of claim 1, wherein the portion of the data set transmitted from the first data shuffle memory pool to the second data shuffle memory pool is transferred using remote direct memory access.

13. The method of claim 1, wherein the portion of the data set to be transmitted is sorted in the data shuffle writer node based on one or more memory ranges associated with the data shuffle reader node.

14. The method of claim 1, wherein the distributed data processing system comprises a MapReduce processing framework, and wherein the data shuffle writer node comprises a mapper node and the data shuffle read node comprises a reducer node.

15. A distributed data processing system, comprising:

a plurality of nodes wherein at least one node is configured as a data shuffle writer node and at least another node is configured as a data shuffle reader node, wherein;

the data shuffle writer node is configured to maintain a first data shuffle memory pool;

the data shuffle reader node is configured to maintain a second data shuffle memory pool; and

further wherein;

the data shuffle writer node is configured to perform an in-memory compression on at least a portion of a data set from the first data shuffle memory pool;

the data shuffle writer node is configured to perform a data shuffle operation on at least a portion of the compressed data from the first shuffle memory pool, wherein the data shuffle operation maps different parts of the compressed data for transmission to different nodes of the distributed data processing system; and

the data shuffle writer node, in response to the data shuffle operation, is configured to transmit the at least a portion of the compressed data from the first data shuffle memory pool to the second data shuffle memory pool in a peer-to-peer manner;

wherein the distributed data processing system is implemented via one or more processing devices.

16. The system of claim 15, wherein performance of the in-memory compression is triggered by a given policy, and wherein the given policy is based on a memory pressure associated with a memory capacity threshold or a ratio of incoming write speed to outgoing read speed.

17. The system of claim 15, wherein each of the first data shuffle memory pool and the second data shuffle memory pool comprise a hybrid memory configuration.

18. The system of claim 15, wherein the portion of the data set transmitted from the first data shuffle memory pool to the second data shuffle memory pool is transferred using remote direct memory access.

19. The system of claim 15, wherein the distributed data processing system comprises a MapReduce processing framework, and wherein the data shuffle writer node comprises a mapper node and the data shuffle read node comprises a reducer node.

20. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by one or more processing devices of a distributed data processing system causes said one or more processing devices to:

maintain a first data shuffle memory pool at a data shuffle writer node and a second data shuffle memory pool at a data shuffle reader node, wherein the data shuffle writer node and the data shuffle reader node are part of a set of multiple nodes of the distributed data processing system;

perform an in-memory compression on at least a portion of a data set from the first data shuffle memory pool; and

perform a data shuffle operation on the at least a portion of the compressed data from the first shuffle memory pool, wherein the data shuffle operation maps different parts of the compressed data for transmission to different nodes of the distributed data processing system; and

transmit, in response to the data shuffle operation, the at least a portion of the compressed data from the first data shuffle memory pool to the second data shuffle memory pool in a peer-to-peer manner.