DYNAMIC SHARDING FOR STATE-BASED PROCESSING

• US 20170118281A1
• Filed: 10/26/2015
• Published: 04/27/2017
• Est. Priority Date: 10/26/2015
• Status: Active Grant

• Current Assignee: McAfee, LLC
• Original Assignee: McAfee, Inc. (McAfee, LLC)
• Inventors: Lehmann, John
• Granted Patent: US 10,841,374 B2
• Time in Patent Office: Days
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/5682 Policies or rules for updating, deleting or replacing the stored data

Abstract

Enabling a distributed data processing system to process a data set from local storage devices by dynamically reallocating portions of the data set.

Claims

1. A system, comprising:

a first node to process data stored in a first local storage device;

a second node including a second local storage device; and

a distributed cache in network communication with the first node and the second node, wherein the system is configured to;

determine whether a size of the data stored in the first local storage device is greater than or equal to a predetermined threshold; and

transfer a portion of the data stored in the first local storage device to the second local storage device via the distributed cache in response to a determination that the size of the data stored in the first local storage device is greater than or equal to the predetermined threshold.

2. The system of claim 1, wherein the predetermined threshold is a ratio of the size of the data stored in the first local storage device relative to a capacity of the first local storage device.

3. The system of claim 1, wherein the first local storage device is in direct communication with the first node.

4. The system of claim 1, wherein the first local storage device includes volatile memory.

5. The system of claim 1, wherein the distributed cache includes nonvolatile memory.

6. The system of claim 1, further comprising a controller that outputs instructions to the first node, the second node, and the distributed cache.

7. The system of claim 1, wherein the first node periodically outputs information indicative of the data stored in the first local storage device to the distributed cache.

8. The system of claim 7, wherein the system is further configured to:

determine whether the first node has experienced a failure; and

transfer the information indicative of the data stored in the first local storage device from the distributed cache to the second local storage device in response to a determination that the first node has experienced a failure.

9. A method, comprising:

storing data in a first local storage device in communication with a first node;

determining whether a size of the data stored in the first local storage device is greater than or equal to a predetermined threshold; and

transferring a portion of the data stored in the first local storage device to a second local storage device in direct communication with a second node via a distributed cache in response to a determination that the size of the data stored in the first local storage device is greater than or equal to the predetermined threshold.

10. The method of claim 9, wherein the predetermined threshold is a ratio of the size of the data stored in the first local storage device relative to a capacity of the first local storage device.

11. The method of claim 9, wherein the first local storage device includes volatile memory.

12. The method of claim 9, wherein the distributed cache includes nonvolatile memory.

13. The method of claim 9, further comprising:

periodically transferring information indicative of the data stored in the first local storage device to the distributed cache.

14. The method of claim 13, further comprising:

transferring the information indicative of the data stored in the first local storage device from the distributed cache to a third local storage device in direct communication with a third node in response to a determination that the first node has experienced a failure.

15. A machine readable medium, on which are stored instructions, comprising instructions that when executed cause a machine to:

determine whether a size of data stored in a first local storage device in communication with a first node is greater than or equal to a predetermined threshold; and

transfer a portion of the data stored in the first local storage device to a second local storage device in direct communication with a second node via a distributed cache in response to a determination that the size of the data stored in the first local storage device is greater than or equal to the predetermined threshold.

16. The machine readable medium of claim 15, wherein the predetermined threshold is a ratio of the size of the data stored in a first local storage device relative to a capacity of the first local storage device.

17. The machine readable medium of claim 15, wherein the first local storage device includes volatile memory.

18. The machine readable medium of claim 15, wherein the distributed cache includes nonvolatile memory.

19. The machine readable medium of claim 15, further comprising instructions that when executed cause the machine to:

periodically transfer information indicative of the data stored in the first local storage device to the distributed cache.

20. The machine readable medium of claim 15, further comprising instructions that when executed cause the machine to:

transfer the information indicative of the data stored in the first local storage device from the distributed cache to a third local storage device in direct communication with a third node in response to a determination that the first node has experienced a failure.