Chaining computes in a distributed computing system

• US 10,146,621 B2
• Filed: 01/12/2017
• Issued: 12/04/2018
• Est. Priority Date: 12/12/2011
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Baptist, Andrew D., Dhuse, Greg R., Gladwin, S. Christopher, Grube, Gary W., Leggette, Wesley B., Motwani, Manish, Resch, Jason K., Shirley, Jr., Thomas F., Volvovski, Ilya
• Primary Examiner(s): Kim, Tae K
• Application Number:
  US15/405,130
• Publication Number:
  US 20170123912A1
• Time in Patent Office: 691 Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/2058 using more than 2 mirrored copies
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 21/602 Providing cryptographic facilities or services
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 9/5077 Logical partitioning of resources; Management or configuration of virtualized resources specific details on emulation or internal functioning of virtual machines G06F9/455
• CPC Class Codes: G06F 9/5083 Techniques for rebalancing the load in a distributed system
• CPC Class Codes: H03M 13/09 Error detection only, e.g. using cyclic redundancy check [CRC] codes or single parity bit
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/3761 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/06 for supporting key management in a packet data network cryptographic mechanisms or cryptographic arrangements for key management H04L9/08
• CPC Class Codes: H04L 63/08 for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 63/10 for controlling access to devices or network resources
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1017 : based on a round robin mechanism
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A computing device includes an interface configured to interface and communicate with a dispersed storage network (DSN), a memory that stores operational instructions, and a processing module operably coupled to the interface and memory such that the processing module, when operable within the computing device based on the operational instructions, is configured to perform various operations. The computing device receives distributed computing data and a distributed computing request that includes a trigger pattern and then identifies a portion of the distributed computing data. The computing device then executes a distributed computing task on the portion of the distributed computing data based on the distributed computing request to generate a partial result, determines whether the partial result compares favorably to the trigger pattern of the distributed computing request, and then either dispersed error encodes the partial result to generate sets of encoded data slices (EDSs) or facilitates execution of further processing.

Claims

1. A computing device comprising:

an interface configured to interface and communicate with a dispersed or distributed storage network (DSN);

memory that stores operational instructions; and

a processing module operably coupled to the interface and to the memory, wherein the processing module, when operable within the computing device based on the operational instructions, is configured to;

receive distributed computing data via the DSN;

receive a distributed computing request that includes a trigger pattern via the DSN;

identify a portion of the distributed computing data;

execute a distributed computing task on the portion of the distributed computing data based on the distributed computing request to generate a partial result;

determine whether the partial result compares favorably to the trigger pattern of the distributed computing request;

when the partial result compares favorably to the trigger pattern of the distributed computing request, dispersed error encode (DSE) the partial result to generate sets of encoded data slices (EDSs) to be stored within a plurality of storage units (SUs) of the DSN; and

when the partial result compares unfavorably to the trigger pattern of the distributed computing request, facilitate execution of at least one other distributed computing task on at least one of the partial result, the portion of the distributed computing data, or another portion of the distributed computing data via the DSN.

2. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured, when the partial result compares favorably to the trigger pattern of the distributed computing request, to:

segment the partial result to generate a plurality of partial result segments;

DSE the plurality of partial result segments to generate sets of EDSs; and

distributedly store the sets of EDSs within the plurality of SUs of the DSN, wherein a decode threshold number of EDSs are needed to recover a partial result segment of the plurality of partial result segments, wherein a read threshold number of EDSs provides for reconstruction of the partial result segment of the plurality of partial result segments, and wherein a write threshold number of EDSs provides for a successful transfer of a set of the sets of the EDSs from a first at least one location in the DSN to a second at least one location in the DSN.

3. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured, when the partial result compares unfavorably to the trigger pattern of the distributed computing request, to:

determine the at least one other distributed computing task;

determine processing parameters of the at least one of the partial result, the portion of the distributed computing data, or another portion of the distributed computing data based on a number of a plurality of other computing devices to perform execution of at least one other distributed computing task on at least one of the partial result, the portion of the distributed computing data, or another portion of the distributed computing data;

determine task partitioning based on the plurality of other computing devices and the processing parameters;

process the at least one of the partial result, the portion of the distributed computing data, or another portion of the distributed computing data based on processing parameters to generate data slice groupings;

partition the at least one other distributed computing task based on the task partitioning to generate partial tasks; and

transmit the partial tasks and the data slice groupings respectively to the plurality of other computing devices to be executed respectively by the plurality of other computing devices to generate a plurality of processed partial results.

4. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured, when the partial result compares unfavorably to the trigger pattern of the distributed computing request, to:

select the at least one other distributed computing task on at least one of the partial result, the portion of the distributed computing data, or another portion of the distributed computing data based on at least one of the partial result, a partial result threshold, a comparison of the partial result of the partial result threshold, an aspect of the partial result, a trigger associated with the aspect of the partial result, or a comparison of the aspect of the partial result with the trigger associated with the aspect of the partial result.

5. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:

determine whether the partial result compares favorably to the trigger pattern of the distributed computing request by at least one of comparing the partial result to a partial result threshold or determining whether the partial result includes a keyword search aspect.

6. The computing device of claim 1, wherein the computing device is located at a first premises that is remotely located from at least one SU of the plurality of SUs within the DSN.

7. The computing device of claim 1 further comprising:

a SU of the plurality of SUs within the DSN, a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device.

8. The computing device of claim 1, wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).

9. A computing device comprising:

an interface configured to interface and communicate with a dispersed or distributed storage network (DSN);

memory that stores operational instructions; and

a processing module operably coupled to the interface and to the memory, wherein the processing module, when operable within the computing device based on the operational instructions, is configured to;

receive distributed computing data via the DSN;

receive a distributed computing request that includes a trigger pattern via the DSN;

identify a portion of the distributed computing data;

execute a distributed computing task on the portion of the distributed computing data based on the distributed computing request to generate a partial result;

determine whether the partial result compares favorably to the trigger pattern of the distributed computing request by at least one of comparing the partial result to a partial result threshold or determining whether the partial result includes a keyword search aspect;

when the partial result compares favorably to the trigger pattern of the distributed computing request, dispersed error encode (DSE) the partial result to generate sets of encoded data slices (EDSs) to be stored within a plurality of storage units (SUs) of the DSN including to;

segment the partial result to generate a plurality of partial result segments;

DSE the plurality of partial result segments to generate sets of EDSs; and

distributedly store the sets of EDSs within the plurality of SUs of the DSN, wherein a decode threshold number of EDSs are needed to recover a partial result segment of the plurality of partial result segments, wherein a read threshold number of EDSs provides for reconstruction of the partial result segment of the plurality of partial result segments, and wherein a write threshold number of EDSs provides for a successful transfer of a set of the sets of the EDSs from a first at least one location in the DSN to a second at least one location in the DSN; and

when the partial result compares unfavorably to the trigger pattern of the distributed computing request, facilitate execution of at least one other distributed computing task on at least one of the partial result, the portion of the distributed computing data, or another portion of the distributed computing data via the DSN.

10. The computing device of claim 9, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured, when the partial result compares unfavorably to the trigger pattern of the distributed computing request, to:

determine the at least one other distributed computing task;

determine processing parameters of the at least one of the partial result, the portion of the distributed computing data, or another portion of the distributed computing data based on a number of a plurality of other computing devices to perform execution of at least one other distributed computing task on at least one of the partial result, the portion of the distributed computing data, or another portion of the distributed computing data;

determine task partitioning based on the plurality of other computing devices and the processing parameters;

process the at least one of the partial result, the portion of the distributed computing data, or another portion of the distributed computing data based on processing parameters to generate data slice groupings;

partition the at least one other distributed computing task based on the task partitioning to generate partial tasks; and

transmit the partial tasks and the data slice groupings respectively to the plurality of other computing devices to be executed respectively by the plurality of other computing devices to generate a plurality of processed partial results.

11. The computing device of claim 9, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured, when the partial result compares unfavorably to the trigger pattern of the distributed computing request, to:

select the at least one other distributed computing task on at least one of the partial result, the portion of the distributed computing data, or another portion of the distributed computing data based on at least one of the partial result, a partial result threshold, a comparison of the partial result of the partial result threshold, an aspect of the partial result, a trigger associated with the aspect of the partial result, or a comparison of the aspect of the partial result with the trigger associated with the aspect of the partial result.

12. The computing device of claim 9 further comprising:

a SU of the plurality of SUs within the DSN, a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device.

13. The computing device of claim 9, wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).

14. A method for execution by a computing device, the method comprising:

receiving, via an interface of the computing device that is configured to interface and communicate with a dispersed or distributed storage network (DSN), distributed computing data;

receiving, via the interface of the computing device, a distributed computing request that includes a trigger pattern;

identifying a portion of the distributed computing data;

executing a distributed computing task on the portion of the distributed computing data based on the distributed computing request to generate a partial result;

determining whether the partial result compares favorably to the trigger pattern of the distributed computing request;

when the partial result compares favorably to the trigger pattern of the distributed computing request, dispersed error encoding the partial result to generate sets of encoded data slices (EDSs) to be stored within a plurality of storage units (SUs) of the DSN; and

when the partial result compares unfavorably to the trigger pattern of the distributed computing request, facilitating execution of at least one other distributed computing task on at least one of the partial result, the portion of the distributed computing data, or another portion of the distributed computing data via the DSN.

15. The method of claim 14, when the partial result compares favorably to the trigger pattern of the distributed computing request, further comprising:

segmenting the partial result to generate a plurality of partial result segments;

dispersed error encoding the plurality of partial result segments to generate sets of EDSs; and

distributedly storing the sets of EDSs within the plurality of SUs of the DSN, wherein a decode threshold number of EDSs are needed to recover a partial result segment of the plurality of partial result segments, wherein a read threshold number of EDSs provides for reconstruction of the partial result segment of the plurality of partial result segments, and wherein a write threshold number of EDSs provides for a successful transfer of a set of the sets of the EDSs from a first at least one location in the DSN to a second at least one location in the DSN.

16. The method of claim 14, when the partial result compares favorably to the trigger pattern of the distributed computing request, further comprising:

determining the at least one other distributed computing task;

determining processing parameters of the at least one of the partial result, the portion of the distributed computing data, or another portion of the distributed computing data based on a number of a plurality of other computing devices to perform execution of at least one other distributed computing task on at least one of the partial result, the portion of the distributed computing data, or another portion of the distributed computing data;

determining task partitioning based on the plurality of other computing devices and the processing parameters;

processing the at least one of the partial result, the portion of the distributed computing data, or another portion of the distributed computing data based on processing parameters to generate data slice groupings;

partitioning the at least one other distributed computing task based on the task partitioning to generate partial tasks; and

transmitting, via the interface of the computing device, the partial tasks and the data slice groupings respectively to the plurality of other computing devices to be executed respectively by the plurality of other computing devices to generate a plurality of processed partial results.

17. The method of claim 14, when the partial result compares unfavorably to the trigger pattern of the distributed computing request, further comprising:

selecting the at least one other distributed computing task on at least one of the partial result, the portion of the distributed computing data, or another portion of the distributed computing data based on at least one of the partial result, a partial result threshold, a comparison of the partial result of the partial result threshold, an aspect of the partial result, a trigger associated with the aspect of the partial result, or a comparison of the aspect of the partial result with the trigger associated with the aspect of the partial result.

18. The method of claim 14 further comprising:

determining whether the partial result compares favorably to the trigger pattern of the distributed computing request by at least one of comparing the partial result to a partial result threshold or determining whether the partial result includes a keyword search aspect.

19. The method of claim 14, wherein the computing device is a SU of the plurality of SUs within the DSN, a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device.

20. The method of claim 14, wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).