Analyzing found data in a distributed storage and task network

• US 9,304,858 B2
• Filed: 12/06/2012
• Issued: 04/05/2016
• Est. Priority Date: 12/12/2011
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Leggette, Wesley, Baptist, Andrew, Dhuse, Greg, Resch, Jason K., Volvovski, Ilya, Motwani, Manish, Gladwin, S. Christopher, Grube, Gary W., Shirley, Thomas Franklin Jr.
• Primary Examiner(s): JACOBS, LASHONDA T
• Application Number:
  US13/707,525
• Publication Number:
  US 20130151581A1
• Time in Patent Office: 1,216 Days
• Field of Search: 709/223, 709/224, 709/217, 709/219, 709/229, 709/203
• US Class Current: 1/1
• CPC Class Codes: G06F 11/10 Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1092 Rebuilding, e.g. when physically replacing a failing disk
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0689 Disk arrays, e.g. RAID, JBOD
• CPC Class Codes: H03M 13/05 using block codes, i.e. a predetermined number of check bits joined to a predetermined number of information bits H03M13/2906 takes precedence
• CPC Class Codes: H03M 13/611 Specific encoding aspects, e.g. encoding by means of decoding
• CPC Class Codes: H04L 61/00 Network arrangements, protocols or services for addressing or naming
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A method begins by a dispersed storage (DS) processing module establishing data identifying criteria for searching data on a network, establishing data analyzing criteria for analyzing found data of the data on the network, and establishing distributed computing criteria. The method continues with the DS processing module distributing the data identifying criteria and the data analyzing criteria to a set of distributed storage and task (DST) units. The method continues with the DS processing module receiving a set of network data partial resultants from the set of DST units, wherein the set of DST units generates the set of network data partial results in accordance with the data identifying criteria to produce found data and analyzing the found data in accordance with the data analyzing criteria. The method continues with the DS processing module processing the set of network data partial resultants to produce a network data resultant.

Claims

1. A method for execution by a computing device in a distributed storage and computing network that stores a large volume of data objects as pluralities of sets of encoded data slices in a plurality of distributed storage and task execution (DST) units, the method comprises:

establishing data identifying criteria for searching for like data objects of the large volume of data objects;

establishing data analyzing criteria for analyzing found data objects;

establishing distributed computing criteria based on the data identifying criteria, the data analyzing criteria, and a slice grouping storage type indication, wherein, for at least one of the found data objects, a DST unit of a set of DST units of the plurality of DST units stores a contiguous data chunk or error code data partition, and wherein the slice grouping storage type indication indicates processing of a contiguous data chunk or processing of the error coded data partition;

distributing the data identifying criteria and the data analyzing criteria to a decode threshold number of DST units in accordance with the distributed computing criteria, wherein the decode threshold number corresponds to a minimum number of encoded data slices of a set of encoded data slices of the pluralities of sets of encoded data slices that is needed to recover a data segment of a corresponding one of the data objects, and wherein the decode threshold number of DST units is less than a number of the DST units storing the pluralities of sets of encoded data slices;

receiving a set of network data partial resultants from the decode threshold number of DST units, wherein the decode threshold number of DST units generates the set of network data partial resultants based on searching at least some of the large volume of data objects in accordance with the data identifying criteria, the data analyzing criteria, and the distributed computing criteria; and

processing the set of network data partial resultants to produce a network data resultant regarding the data on the network.

2. The method of claim 1, wherein the establishing data identifying criteria comprises at least one of:

determining data content search criteria;

determining file name search criteria;

determining data source identifying criteria;

determining data destination identifying criteria;

determining data type searching criteria;

determining data routing searching criteria; and

compiling at least one of the data content search criteria, the file name search criteria, the data source identifying criteria, the data destination identifying criteria, the data type searching criteria, and the data routing searching criteria to produce the data identifying criteria.

3. The method of claim 1, wherein the establishing data analyzing criteria comprises at least one of:

performing organization analysis on the found data objects;

performing statistical analysis on the found data objects; and

performing interpretive analysis on the found data objects.

4. The method of claim 1, wherein the establishing the distributed computing criteria comprises at least one of:

estimating computational resource requirements for searching the data on the network in accordance with the data identifying criteria to produce the found data objects and analyzing the found data objects in accordance with the data analyzing criteria;

determining computational capabilities of the set of DST units; and

establishing the distributed computing criteria based on the estimated computational resource requirements and the computational capabilities.

5. The method of claim 1 further comprises:

estimating computational resource requirements for searching the data on the network in accordance with the data identifying criteria to produce the found data objects and analyzing the found data objects in accordance with the data analyzing criteria; and

selecting the set of DST units from the plurality of DST units based on the computational resource requirements.

6. The method of claim 1 further comprises:

determining, by a DST unit of the set of DST units, whether to sub-distribute at least one of an allocated portion of the network data identifying criteria and an allocated portion of the network data analyzing criteria; and

when the DST unit determines to sub-distribute the at least one of the allocated portion of the network data identifying criteria and the allocated portion of the network data analyzing criteria;

establishing, by the DST unit, at least one of;

local data identifying criteria for searching the data on the network based on an allocation portion of the data identifying criteria; and

local data analyzing criteria for analyzing local found data of the found data objects based on an allocation portion of the data analyzing criteria;

establishing local distributed computing criteria based on at least one of the local data identifying criteria and the local data analyzing criteria; and

distributing the at least one of the local data identifying criteria and the local data analyzing criteria to a second set of DST units of the plurality of DST units in accordance with the local distributed computing criteria.

7. The method of claim 6 further comprises:

distributing the local data identifying criteria to the second set of DST units in accordance with the local distributed computing criteria;

receiving a set of sub-partial found data from the second set of DST units;

compiling the set of sub-partial found data into partial found data; and

performing the allocation portion of the data analyzing criteria on the partial found data to produce one of the set of network data partial resultants.

8. The method of claim 6 further comprises:

performing the allocation portion of the data identifying criteria to produce partial found data;

distributing the local data analyzing criteria to the second set of DST units in accordance with the local distributed computing criteria;

dividing the partial found data into a set of sub-partial found data in accordance with the local distributed computing criteria;

distributing the set of sub-partial found data to the second set of DST units in accordance with the local distributed computing criteria;

receiving a set of data sub-partial results from the second set of DST units; and

compiling the set of data sub-partial results to produce one of the set of network data partial resultants.

9. The method of claim 1 further comprises:

establishing data storage criteria for storing at least one of the found data objects and the data on the network, wherein the data storage criteria identifies the at least one of the found data objects and the data on the network and includes dispersed storage error coding parameters; and

storing, by the set of DST units, the at least one of the found data objects and the data on the network in accordance with the dispersed storage error coding parameters.

10. A dispersed storage (DS) module operable in a computing device of a distributed storage and computing network that stores a large volume of data objects as pluralities of sets of encoded data slices in a plurality of distributed storage and task execution (DST) units, the DS module comprises:

a first module, when operable within a computing device, causes the computing device to;

establish data identifying criteria for searching for like data objects of the large volume of data objects;

a second module, when operable within the computing device, causes the computing device to;

establish data analyzing criteria for analyzing found data objects;

a third module, when operable within the computing device, causes the computing device to;

establish distributed computing criteria based on the data identifying criteria, the data analyzing criteria, and a slice grouping storage type indication, wherein, for at least one of the found data objects, a DST unit of a set of DST units of the plurality of DST units stores a contiguous data chunk or error code data partition, and wherein the slice grouping storage type indication indicates processing of a contiguous data chunk or processing of the error coded data partition; and

a fourth module, when operable within the computing device, causes the computing device to;

distribute the data identifying criteria and the data analyzing criteria to a decode threshold number of DST units in accordance with the distributed computing criteria, wherein the decode threshold number corresponds to a minimum number of encoded data slices of a set of encoded data slices of the pluralities of sets of encoded data slices that is needed to recover a data segment of a corresponding one of the data objects, and wherein the decode threshold number of DST units is less than a number of the DST units storing the pluralities of sets of encoded data slices;

receive a set of network data partial resultants from the decode threshold number of DST units, wherein the decode threshold number of DST units generates the set of network data partial resultants based on searching at least some of the large volume of data objects in accordance with the data identifying criteria, the data analyzing criteria, and the distributed computing criteria; and

process the set of network data partial resultants to produce a network data resultant regarding the data on the network.

11. The DS module of claim 10, wherein the first module functions to establish the data identifying criteria by at least one of:

determining data content search criteria;

determining file name search criteria;

determining data source identifying criteria;

determining data destination identifying criteria;

determining data type searching criteria;

determining data routing searching criteria; and

compiling at least one of the data content search criteria, the file name search criteria, the data source identifying criteria, the data destination identifying criteria, the data type searching criteria, and the data routing searching criteria to produce the data identifying criteria.

12. The DS module of claim 10, wherein the second module functions to establish data analyzing criteria by at least one of:

performing organization analysis on the found data objects;

performing statistical analysis on the found data objects; and

performing interpretive analysis on the found data objects.

13. The DS module of claim 10, wherein the third module functions to establish the distributed computing criteria by at least one of:

estimating computational resource requirements for searching the data on the network in accordance with the data identifying criteria to produce the found data objects and analyzing the found data objects in accordance with the data analyzing criteria;

determining computational capabilities of the set of DST units; and

establishing the distributed computing criteria based on the estimated computational resource requirements and the computational capabilities.

14. The DS module of claim 10 further comprises:

the fourth module further functions to;

estimate computational resource requirements for searching the data on the network in accordance with the data identifying criteria to produce the found data objects and analyzing the found data objects in accordance with the data analyzing criteria; and

select the set of DST units from the plurality of DST units based on the computational resource requirements.

15. The DS module of claim 10 further comprises:

a DST unit of the set of DST units further functions to;

determine whether to sub-distribute at least one of an allocated portion of the network data identifying criteria and an allocated portion of the network data analyzing criteria; and

when the DST unit determines to sub-distribute the at least one of the allocated portion of the network data identifying criteria and the allocated portion of the network data analyzing criteria;

establish at least one of;

local data identifying criteria for searching the data on the network based on an allocation portion of the data identifying criteria; and

local data analyzing criteria for analyzing local found data of the found data objects based on an allocation portion of the data analyzing criteria;

establish local distributed computing criteria based on at least one of the local data identifying criteria and the local data analyzing criteria; and

distribute the at least one of the local data identifying criteria and the local data analyzing criteria to a second set of DST units of the plurality of DST units in accordance with the local distributed computing criteria.

16. The DS module of claim 15 further comprises:

the DST unit of the set of DST units further functions to;

distribute the local data identifying criteria to the second set of DST units in accordance with the local distributed computing criteria;

receive a set of sub-partial found data from the second set of DST units;

compile the set of sub-partial found data into partial found data; and

perform the allocation portion of the data analyzing criteria on the partial found data to produce one of the set of network data partial resultants.

17. The DS module of claim 15 further comprises:

the DST unit of the set of DST units further functions to;

perform the allocation portion of the data identifying criteria to produce partial found data;

distribute the local data analyzing criteria to the second set of DST units in accordance with the local distributed computing criteria;

divide the partial found data into a set of sub-partial found data in accordance with the local distributed computing criteria;

distribute the set of sub-partial found data to the second set of DST units in accordance with the local distributed computing criteria;

receive a set of data sub-partial results from the second set of DST units; and

compile the set of data sub-partial results to produce one of the set of network data partial resultants.

18. The DS module of claim 10 further comprises:

the fourth module further functions to establish data storage criteria for storing at least one of the found data objects and the data on the network, wherein the data storage criteria identifies the at least one of the found data objects and the data on the network and includes dispersed storage error coding parameters; and

the set of DST units further functions to store the at least one of the found data objects and the data on the network in accordance with the dispersed storage error coding parameters.