Internet based shared memory in a distributed computing system

• US 10,176,045 B2
• Filed: 02/21/2017
• Issued: 01/08/2019
• Est. Priority Date: 12/12/2011
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Baptist, Andrew D., Dhuse, Greg R., Gladwin, S. Christopher, Grube, Gary W., Leggette, Wesley B., Motwani, Manish, Resch, Jason K., Shirley, Jr., Thomas F., Volvovski, Ilya
• Primary Examiner(s): Kanaan, Simon P
• Application Number:
  US15/437,862
• Publication Number:
  US 20170160943A1
• Time in Patent Office: 686 Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/2058 using more than 2 mirrored copies
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 21/602 Providing cryptographic facilities or services
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0659 Command handling arrangements, e.g. command buffers, queues, command scheduling
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 9/5066 Algorithms for mapping a plurality of inter-dependent sub-tasks onto a plurality of physical CPUs mappping at compile time, see G06F8/451
• CPC Class Codes: H03M 13/09 Error detection only, e.g. using cyclic redundancy check [CRC] codes or single parity bit
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/3761 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H04L 67/04 specially adapted for terminals or networks with limited capabilities; specially adapted for terminal portability
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1017 based on a round robin mechanism
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/306 User profiles
• CPC Class Codes: H04L 69/04 Protocols for data compression, e.g. ROHC

Abstract

A method includes receiving, by a computing device of a dispersed storage and task (DST) network, raw data. The method further includes dispersed storage (DS) error encoding the raw data in accordance with initial DS error encoding parameters to produce a plurality of sets of encoded data slices (EDSs). The method further includes determining index generation task information and sending the plurality of sets of EDSs and the index generation task information to a set of DST execution units. The method further includes generating, by the set of DST execution units, partial data indexes from the plurality of sets of EDSs based on the index generation task information. The method further includes generating a data index based on the partial data indexes and an operational task, partitioning the operational task into a set of partial tasks, and sending the set of partial tasks to the set of DST execution units.

Claims

1. A method comprises:

receiving, by a computing device of a distributed storage and task (DST) network, raw data;

dispersed storage error encoding, by the computing device, the raw data in accordance with initial dispersed storage error encoding parameters to produce a plurality of sets of encoded data slices, wherein the initial dispersed storage error encoding parameters include one or more of;

initial data segmenting information, initial segment security information, initial error coding information, initial slicing information, and initial slice security information;

determining, by the computing device, index generation task information regarding the raw data;

sending, by the computing device, the plurality of sets of encoded data slices and the index generation task information to a set of DST execution units;

generating, by the set of DST execution units, partial data indexes from the plurality of sets of encoded data slices based on the index generation task information;

generating, by the computing device, a data index based on the partial data indexes;

determining, by the computing device, an operational task from a list of operational tasks that includes storing the plurality of sets of encoded data slices, storing the data index, long term storage of the raw data, execute a data processing function on the sets of encoded data slices and execute a data processing function on the data index;

partitioning, by the computing device, the operational task into a set of partial tasks; and

sending, by the computing device, the set of partial tasks to the set of DST execution units.

2. The method of claim 1 further comprises:

executing, by at least some of the set of DST execution units, the set of partial tasks on at least some of the plurality of sets of encoded data slices.

3. The method of claim 1, wherein the determining the index generation task information comprises:

determining, by the computing device, the index generation task information based on a storage profile regarding the raw data.

4. The method of claim 3, wherein the index generation task information includes one or more of:

a search parameter;

a keyword;

pattern recognition information; and

timing information.

5. The method of claim 3, wherein the storage profile includes one or more of:

a security requirement;

a performance requirement;

an estimated retrieval frequency;

an estimated distributed processing level;

a data visibility profile;

a data owner;

a data index storage indicator; and

a data deletion policy.

6. The method of claim 1, wherein the data index includes one or more of:

identified keywords;

identified dates;

identified internet protocol addresses;

partial content;

a word count;

raw data statistics;

a summary of the raw data; and

one or more DST network addresses.

7. The method of claim 1 further comprises:

when the operational task is long term storage of the raw data, selecting, by the computing device, new dispersed storage error encoding parameters;

dispersed storage error encoding, by the computing device, the raw data in accordance with the new dispersed storage error encoding parameters to produce a new plurality of sets of encoded data slices; and

sending, by the computing device, the new plurality of encoded data slices and the set of partial tasks to the set of DST execution units, such that the set of DST execution units stores the new plurality of encoded data slices.

8. A computer readable memory comprises:

a first memory element that stores operational instructions that, when executed by a computing device of a distributed storage and task (DST) network, causes the computing device to;

receive raw data;

dispersed storage error encode the raw data in accordance with initial dispersed storage error encoding parameters to produce a plurality of sets of encoded data slices, wherein the initial dispersed storage error encoding parameters include one or more of;

initial data segmenting information, initial segment security information, initial error coding information, initial slicing information, and initial slice security information;

determine index generation task information regarding the raw data; and

send the plurality of sets of encoded data slices and the index generation task information to a set of DST execution units;

a second memory element that stores operational instructions that, when executed by the set of DST execution units, causes the set of DST execution units to;

generate partial data indexes from the plurality of sets of encoded data slices based on the index generation task information;

a third memory element that stores operational instructions that, when executed by the computing device, causes the computing device to;

generate a data index based on the partial data indexes;

determine an operational task from a list of operational tasks that includes storing the plurality of sets of encoded data slices, storing the data index, long term storage of the raw data, execute a data processing function on the sets of encoded data slices and execute a data processing function on the data index;

partition the operational task into a set of partial tasks; and

send the set of partial tasks to the set of DST execution units.

9. The computer readable memory of claim 8 further comprises:

a fourth memory element that stores operational instructions that, when executed by the set of DST execution units, causes at least some of the set of DST execution units to;

execute the set of partial tasks on at least some of the plurality of sets of encoded data slices.

10. The computer readable memory of claim 8, wherein the first memory element further stores operational instructions that, when executed by the computing device, causes the computing device to determine the index generation task information by:

determining the index generation task information based on a storage profile regarding the raw data.

11. The computer readable memory of claim 10, wherein the index generation task information includes one or more of:

a search parameter;

a keyword;

pattern recognition information; and

timing information.

12. The computer readable memory of claim 10, wherein the storage profile includes one or more of:

a security requirement;

a performance requirement;

an estimated retrieval frequency;

an estimated distributed processing level;

a data visibility profile;

a data owner;

a data index storage indicator; and

a data deletion policy.

13. The computer readable memory of claim 8, wherein the data index includes one or more of:

identified keywords;

identified dates;

identified internet protocol addresses;

partial content;

a word count;

raw data statistics;

a summary of the raw data; and

one or more DST network addresses.

14. The computer readable memory of claim 8, wherein the third memory element further stores operational instructions that, when executed by the computing device, causes the computing device to:

when the operational task is long term storage of the raw data select new dispersed storage error encoding parameters;

dispersed storage error encodes the raw data in accordance with the new dispersed storage error encoding parameters to produce a new plurality of sets of encoded data slices; and

send the new plurality of encoded data slices and the set of partial tasks to the set of DST execution units, such that the set of DST execution units stores the new plurality of encoded data slices.