Dispersed storage network with metadata generation and methods for use therewith

• US 9,430,336 B2
• Filed: 06/26/2014
• Issued: 08/30/2016
• Est. Priority Date: 09/30/2005
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Motwani, Manish, Resch, Jason K.
• Primary Examiner(s): Zele, Krista
• Assistant Examiner(s): FORMAN, JAMES Q
• Application Number:
  US14/315,842
• Publication Number:
  US 20140310492A1
• Time in Patent Office: 796 Days
• Field of Search: 726/2
• US Class Current: 1/1
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 11/2038 with a single idle spare processing component
• CPC Class Codes: G06F 11/2048 where the redundant components share neither address space nor persistent storage
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/172 Caching, prefetching or hoarding of files
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2211/1028 Distributed, i.e. distributed RAID systems with parity
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 63/10 for controlling access to devices or network resources
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A technique by a processing module of a computing device of a dispersed storage network (DSN) to generate metadata associated with a stored data object. Once the data object is identified, a data segment of the data object is determined and a number of data slices corresponding to the data segment is retrieved. Then, the data segment is regenerated from the retrieved data slices. Once the data segment is regenerated, metadata that is associated with the data segment is generated and stored.

Claims

1. A method for execution by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), the method comprises:

a) identifying a data object stored in the DSN, in which the data object is segmented into a plurality of data segments and respective data segments are error encoded to generate a plurality of data slices and the plurality of data slices are stored in a plurality of storage devices of the DSN, and in which a threshold number of data slices of a particular data segment are needed to reconstruct the particular data segment, the threshold number of data slices being less than the plurality of data slices generated for the particular data segment;

b) determining a data segment of the data object stored in the plurality of storage devices of the DSN;

c) retrieving the threshold number of data slices corresponding to the data segment;

d) regenerating the data segment from the threshold number of data slices retrieved corresponding to the data segment;

e) generating metadata from the regenerated data segment;

f) storing the metadata associated with the data segment;

g) comparing an amount of the stored metadata to a completeness threshold to determine if additional metadata needs to be generated and, when additional metadata is required, determining that the metadata does not include data in at least one of a plurality of categories;

h) determining another data segment of the data object when the amount of the stored metadata does not meet the completeness threshold; and

i) repeating c), d), e), f), g) and h) for one or more additional data segment or segments until a determination is made that the stored metadata meets the completeness threshold.

2. The method of claim 1 wherein the metadata corresponding to the data segment includes at least one of:

a type of data, a keyword, a phrase, a pattern, a priority indicator, a relationship to other data objects, a security indicator, a user identifier, or a timestamp.

3. The method of claim 1 further comprising:

determining when additional metadata is no longer required.

4. A dispersed storage (DS) processing unit, having a hardware processor and memory, for use in a dispersed storage network (DSN) comprises:

at least one module, operable with the hardware processor, causes the hardware processor to;

a) identify a data object stored in the DSN, in which the data object is segmented into a plurality of data segments and respective data segments are error encoded to generate a plurality of data slices and the plurality of data slices are stored in a plurality of storage devices of the DSN, and in which a threshold number of data slices of a particular data segment are needed to reconstruct the particular data segment, the threshold number of data slices being less than the plurality of data slices generated for the particular data segment;

b) determine a data segment of the data object stored in the plurality of storage devices of the DSN;

c) retrieve the threshold number of data slices corresponding to the data segment;

d) regenerate the data segment from the threshold number of data slices retrieved corresponding to the data segment;

e) generate metadata from the regenerated data segment;

f) store the metadata associated with the data segment;

g) compare an amount of the stored metadata to a completeness threshold to determine if additional metadata needs to be generated and, when additional metadata is required, determine that the metadata does not include data in at least one of a plurality of categories;

h) determine another data segment of the data object when the amount of the stored metadata does not meet the completeness threshold; and

i) repeat c), d), e), f), g) and h) for one or more additional data segment or segments until a determination is made that the stored metadata meets the completeness threshold.

5. The DS processing unit of claim 4 wherein the metadata corresponding to the data segment includes at least one of:

a type of data, a keyword, a phrase, a pattern, a priority indicator, a relationship to other data objects, a security indicator, a user identifier, or a timestamp.

6. The DS processing unit of claim 4 wherein the at least one module, operable with the hardware processor, further causes the hardware processor to:

determine when additional metadata is no longer required.

7. A non-transitory computer readable storage medium comprises:

at least one memory section that stores operational instructions that, when executed by one or more processing modules of one or more computing devices of a dispersed storage network (DSN), causes the one or more computing devices to;

a) identify a data object previously stored in the DSN, in which the data object is segmented into a plurality of data segments and respective data segments are error encoded to generate a plurality of data slices and the plurality of data slices are stored in a plurality of storage devices of the DSN, and in which a threshold number of data slices of a particular data segment are needed to reconstruct the particular data segment, the threshold number of data slices being less than the plurality of data slices generated for the particular data segment;

b) determine a data segment of the data object stored in the plurality of storage devices of the DSN;

c) retrieve the threshold number of data slices corresponding to the data segment;

d) regenerate the data segment from the threshold number of data slices retrieved corresponding to the data segment;

e) generate metadata from the regenerated data segment;

f) store the metadata associated with the data segment;

g) compare an amount of the stored metadata to a completeness threshold to determine if additional metadata needs to be generated and, when additional metadata is required, determine that the metadata does not include data in at least one of a plurality of categories;

h) determine another data segment of the data object when the amount of the stored metadata does not meet the completeness threshold; and

i) repeat c), d), e), f), g) and h) for one or more additional data segment or segments until a determination is made that the stored metadata meets the completeness threshold.

8. The non-transitory computer readable storage medium of claim 7 wherein the metadata corresponding to the data segment includes at least one of:

a type of data, a keyword, a phrase, a pattern, a priority indicator, a relationship to other data objects, a security indicator, a user identifier, or a timestamp.

9. The non-transitory computer readable storage medium of claim 7 wherein the operational instructions, when executed by the one or more processing modules of the one or more computing devices of the dispersed storage network (DSN), further causes the one or more computing devices to:

determine when additional metadata is no longer required.

10. The non-transitory computer readable storage medium of claim 7 wherein the operational instructions, when executed by the one or more processing modules of the one or more computing devices of the dispersed storage network (DSN), further causes the one or more computing devices to:

determine if the metadata does not include data in at least one of a plurality of categories.