Generation collapse

• US 10,001,923 B2
• Filed: 03/02/2017
• Issued: 06/19/2018
• Est. Priority Date: 12/29/2009
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Baptist, Andrew D., Motwani, Manish, Resch, Jason K.
• Primary Examiner(s): Parry, Chris
• Assistant Examiner(s): Stiltner, Weiwei
• Application Number:
  US15/448,148
• Publication Number:
  US 20170177228A1
• Time in Patent Office: 474 Days
• Field of Search: 709213
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0772 Means for error signaling, e.g. using interrupts, exception flags, dedicated error registers
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1464 for networked environments
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 21/10 Protecting distributed programs or content, e.g. vending or licensing of copyrighted material protection in video systems or pay television H04N7/16 ; Digital rights management [DRM]
• CPC Class Codes: G06F 21/80 in storage media based on magnetic or optical technology, e.g. disks with sectors preventing unauthorised reproduction or copying of disc-type recordable media G11B20/00
• CPC Class Codes: G06F 2221/2129 Authenticate client device independently of the user
• CPC Class Codes: G06F 2221/2151 Time stamp
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0611 in relation to response time
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0625 Power saving in storage systems
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/0653 : Monitoring storage devices or systems
• CPC Class Codes: G06F 3/067 : Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0683 : Plurality of storage devices
• CPC Class Codes: H03M 13/1515 : Reed-Solomon codes
• CPC Class Codes: H03M 13/3761 : using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H03M 13/616 : Matrix operations, especially for generator matrices or check matrices, e.g. column or row permutations
• CPC Class Codes: H04L 1/16 : in which the return channel carries supervisory signals, e.g. repetition request signals
• CPC Class Codes: H04L 2209/34 : Encoding or coding, e.g. Huffman coding or error correction
• CPC Class Codes: H04L 43/16 : Threshold monitoring
• CPC Class Codes: H04L 63/101 : Access control lists [ACL]
• CPC Class Codes: H04L 63/102 : Entity profiles
• CPC Class Codes: H04L 67/06 : specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 9/085 : Secret sharing or secret splitting, e.g. threshold schemes
• CPC Class Codes: H04N 21/2181 : comprising remotely distributed storage units, e.g. when movies are replicated over a plurality of video servers distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: H04N 21/23116 : involving data replication, e.g. over plural servers synchronization of replicated data G06F11/1658; error detection or correction by means of data replication G06F11/2053; replication in distributed file systems G06F16/10; replication in distributed file systems G06F16/27; replication or mirroring of data in data networks H04L67/1095
• CPC Class Codes: H04N 21/232 : Content retrieval operation locally within server, e.g. reading video streams from disk arrays storage management G06F3/0604; details of querying and searching of video data from a database G06F16/739
• CPC Class Codes: H04N 21/274 : Storing end-user multimedia data in response to end-user request , e.g. network recorder

Abstract

A computing device includes an interface configured to interface and communicate with a dispersed storage network (DSN), a memory that stores operational instructions, and a processing module that is configured to perform various operations. The computing device determines and converts sets of encoded data slices (EDSs) associated with a data object stored within a generation-based storage pool set of storage units (SUs) to a Decentralized, or Distributed, Agreement Protocol (DAP)-based storage pool set of SUs. The computing device convert a first storage generation set of EDSs associated with the data object to pre-establish DAP scoring information. Then, for additional generation sets of EDSs associated with the data object and stored within the generation-based storage pool set of SUs, the computing device uses additional DAP scoring information to determine migration of EDSs from the generation-based storage pool set of SUs to the DAP-based storage pool set of SUs.

Claims

1. A computing device comprising:

an interface configured to interface and communicate with a dispersed or distributed storage network (DSN);

memory that stores operational instructions; and

a processing module operably coupled to the interface and to the memory, wherein the processing module, when operable within the computing device based on the operational instructions, is configured to;

determine to convert sets of encoded data slices (EDSs) associated with a data object stored within a generation-based storage pool set of storage units (SUs) to a Decentralized, or Distributed, Agreement Protocol (DAP)-based storage pool set of SUs, wherein the generation-based storage pool set of SUs stores a plurality of sets of EDSs associated with the data object, wherein the data object is segmented into a plurality of data segments, wherein a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce a set of the sets of encoded data slices (EDSs);

convert a first storage generation set of the sets of EDSs associated with the data object and stored within the generation-based storage pool set of SUs into a first DAP-based set of the sets of EDSs in accordance with the DAP and stored within a first DAP-based storage pool of SUs of the DAP-based storage pool set of SUs to pre-establish DAP scoring information that correspond to slice names of EDSs of the first storage generation set of the sets of EDSs associated with the data object;

for additional generation sets of the sets of EDSs associated with the data object and stored within the generation-based storage pool set of SUs;

perform a DAP function on a slice name to produce first DAP scoring information for DAP-based storage pools of SUs of the DAP-based storage pool set of SUs and to produce second DAP scoring information for sets of the EDSs associated with the data object stored within the generation-based storage pool set of SUs having the slice name; and

migrate an EDS of the sets of EDSs associated with the data object stored within the generation-based storage pool set of SUs associated with the slice name and having highest DAP scoring information based on the second DAP scoring information to a DAP-based storage pool of SUs among the DAP-based storage pool set of SUs having highest DAP scoring information based on the first DAP scoring information.

2. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:

determine to convert the sets of EDSs associated with the data object stored within a generation-based storage pool set of SUs to the DAP-based storage pool set of SUs based on detection of detect an unfavorable storage efficiency of the sets of EDSs associated with the data object within the generation-based storage pool set of SUs.

3. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:

facilitate migration of a second storage generation set of the sets of EDSs associated with the data object and stored within the generation-based storage pool set of SUs into a second DAP-based set of the sets of EDSs in accordance with the DAP to be stored in a second DAP-based storage pool of SUs of the DAP-based storage pool set of SUs;

facilitate migration of a third storage generation set of the sets of EDSs associated with the data object and stored within the generation-based storage pool set of SUs into a third DAP-based set of the sets of EDSs in accordance with the DAP to be stored in a third DAP-based storage pool of SUs of the DAP-based storage pool set of SUs;

perform the DAP function on the second storage generation set of the sets of EDSs associated with the data object stored within the second DAP-based storage pool of SUs of the DAP-based storage pool set of SUs to generate third DAP scoring information;

perform the DAP function on the third storage generation set of the sets of EDSs associated with the data object stored within the third DAP-based storage pool of SUs of the DAP-based storage pool set of SUs to generate fourth DAP scoring information;

select the second DAP-based storage pool of SUs as the DAP-based storage pool of SUs when the third DAP scoring information is higher than the fourth DAP scoring information; and

select the third DAP-based storage pool of SUs as the DAP-based storage pool of SUs when the fourth DAP scoring information is higher than the third DAP scoring information.

4. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:

for the data segment, migrate at least a write threshold number of EDSs to the DAP-based storage pool of SUs among the DAP-based storage pool set of SUs, wherein the write threshold number of EDSs provides for a successful transfer of a set of the sets of EDSs from a first at least one location in the DSN to a second at least one location in the DSN; and

after migration of the at least a write threshold number of EDSs, re-provision at least one pool of SUs of the generation-based storage pool set of SUs to store at least one EDSs associated with another data object.

5. The computing device of claim 1, wherein:

a decode threshold number of EDSs are needed to recover the data segment associated with the set of the sets of EDSs;

a read threshold number of EDSs provides for reconstruction of the data segment associated with the set of the sets of EDSs; and

for the data segment, a write threshold number of EDSs provides for a successful transfer of the set of the sets of EDSs from a first at least one location in the DSN to a second at least one location in the DSN.

6. The computing device of claim 1, wherein the computing device is located at a first premises that is remotely located from at least one SU of the DAP-based storage pool set of SUs or the generation-based storage pool set of SUs within the DSN.

7. The computing device of claim 1 further comprising:

a SU of the DAP-based storage pool set of SUs or the generation-based storage pool set of SUs within the DSN, a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device.

8. The computing device of claim 1, wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).

9. A computing device comprising:

an interface configured to interface and communicate with a dispersed or distributed storage network (DSN);

memory that stores operational instructions; and

a processing module operably coupled to the interface and to the memory, wherein the processing module, when operable within the computing device based on the operational instructions, is configured to;

determine to convert sets of encoded data slices (EDSs) associated with a data object stored within a generation-based storage pool set of storage units (SUs) to a Decentralized, or Distributed, Agreement Protocol (DAP)-based storage pool set of SUs based on detection of detect an unfavorable storage efficiency of the sets of EDSs associated with the data object within the generation-based storage pool set of SUs, wherein the generation-based storage pool set of SUs stores a plurality of sets of EDSs associated with the data object, wherein the data object is segmented into a plurality of data segments, wherein a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce a set of the sets of encoded data slices (EDSs), wherein a decode threshold number of EDSs are needed to recover the data segment associated with the set of the sets of EDSs, wherein a read threshold number of EDSs provides for reconstruction of the data segment associated with the set of the sets of EDSs, and wherein a write threshold number of EDSs provides for a successful transfer of the set of the sets of EDSs from a first at least one location in the DSN to a second at least one location in the DSN for the data segment;

convert a first storage generation set of the sets of EDSs associated with the data object and stored within the generation-based storage pool set of SUs into a first DAP-based set of the sets of EDSs in accordance with the DAP and stored within a first DAP-based storage pool of SUs of the DAP-based storage pool set of SUs to pre-establish DAP scoring information that correspond to slice names of EDSs of the first storage generation set of the sets of EDSs associated with the data object;

for additional generation sets of the sets of EDSs associated with the data object and stored within the generation-based storage pool set of SUs;

perform a DAP function on a slice name to produce first DAP scoring information for DAP-based storage pools of SUs of the DAP-based storage pool set of SUs and to produce second DAP scoring information for sets of the EDSs associated with the data object stored within the generation-based storage pool set of SUs having the slice name; and

migrate an EDS of the sets of EDSs associated with the data object stored within the generation-based storage pool set of SUs associated with the slice name and having highest DAP scoring information based on the second DAP scoring information to a DAP-based storage pool of SUs among the DAP-based storage pool set of SUs having highest DAP scoring information based on the first DAP scoring information.

10. The computing device of claim 9, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:

facilitate migration of a second storage generation set of the sets of EDSs associated with the data object and stored within the generation-based storage pool set of SUs into a second DAP-based set of the sets of EDSs in accordance with the DAP to be stored in a second DAP-based storage pool of SUs of the DAP-based storage pool set of SUs;

facilitate migration of a third storage generation set of the sets of EDSs associated with the data object and stored within the generation-based storage pool set of SUs into a third DAP-based set of the sets of EDSs in accordance with the DAP to be stored in a third DAP-based storage pool of SUs of the DAP-based storage pool set of SUs;

perform the DAP function on the second storage generation set of the sets of EDSs associated with the data object stored within the second DAP-based storage pool of SUs of the DAP-based storage pool set of SUs to generate third DAP scoring information;

perform the DAP function on the third storage generation set of the sets of EDSs associated with the data object stored within the third DAP-based storage pool of SUs of the DAP-based storage pool set of SUs to generate fourth DAP scoring information;

select the second DAP-based storage pool of SUs as the DAP-based storage pool of SUs when the third DAP scoring information is higher than the fourth DAP scoring information; and

select the third DAP-based storage pool of SUs as the DAP-based storage pool of SUs when the fourth DAP scoring information is higher than the third DAP scoring information.

11. The computing device of claim 9, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:

for the data segment, migrate at least a write threshold number of EDSs to the DAP-based storage pool of SUs among the DAP-based storage pool set of SUs, wherein the write threshold number of EDSs provides for a successful transfer of a set of the sets of EDSs from a first at least one location in the DSN to a second at least one location in the DSN; and

after migration of the at least a write threshold number of EDSs, re-provision at least one pool of SUs of the generation-based storage pool set of SUs to store at least one EDSs associated with another data object.

12. The computing device of claim 9 further comprising:

a SU of the DAP-based storage pool set of SUs or the generation-based storage pool set of SUs within the DSN, a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device.

13. The computing device of claim 9, wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).

14. A method for execution by a computing device, the method comprising:

determining to convert sets of encoded data slices (EDSs) associated with a data object stored within a generation-based storage pool set of storage units (SUs) to a Decentralized, or Distributed, Agreement Protocol (DAP)-based storage pool set of SUs within a dispersed or distributed storage network (DSN), wherein the generation-based storage pool set of SUs stores a plurality of sets of EDSs associated with the data object, wherein the data object is segmented into a plurality of data segments, wherein a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce a set of the sets of encoded data slices (EDSs);

converting a first storage generation set of the sets of EDSs associated with the data object and stored within the generation-based storage pool set of SUs into a first DAP-based set of the sets of EDSs in accordance with the DAP and stored within a first DAP-based storage pool of SUs of the DAP-based storage pool set of SUs to pre-establish DAP scoring information that correspond to slice names of EDSs of the first storage generation set of the sets of EDSs associated with the data object;

for additional generation sets of the sets of EDSs associated with the data object and stored within the generation-based storage pool set of SUs;

performing a DAP function on a slice name to produce first DAP scoring information for DAP-based storage pools of SUs of the DAP-based storage pool set of SUs and to produce second DAP scoring information for sets of the EDSs associated with the data object stored within the generation-based storage pool set of SUs having the slice name; and

migrating, via an interface configured to interface and communicate with the DSN, an EDS of the sets of EDSs associated with the data object stored within the generation-based storage pool set of SUs associated with the slice name and having highest DAP scoring information based on the second DAP scoring information to a DAP-based storage pool of SUs among the DAP-based storage pool set of SUs having highest DAP scoring information based on the first DAP scoring information.

15. The method of claim 14 further comprising:

determining to convert the sets of EDSs associated with the data object stored within a generation-based storage pool set of SUs to the DAP-based storage pool set of SUs based on detection of detect an unfavorable storage efficiency of the sets of EDSs associated with the data object within the generation-based storage pool set of SUs.

16. The method of claim 14 further comprising:

facilitating migration of a second storage generation set of the sets of EDSs associated with the data object and stored within the generation-based storage pool set of SUs into a second DAP-based set of the sets of EDSs in accordance with the DAP to be stored in a second DAP-based storage pool of SUs of the DAP-based storage pool set of SUs;

facilitating migration of a third storage generation set of the sets of EDSs associated with the data object and stored within the generation-based storage pool set of SUs into a third DAP-based set of the sets of EDSs in accordance with the DAP to be stored in a third DAP-based storage pool of SUs of the DAP-based storage pool set of SUs;

performing the DAP function on the second storage generation set of the sets of EDSs associated with the data object stored within the second DAP-based storage pool of SUs of the DAP-based storage pool set of SUs to generate third DAP scoring information;

performing the DAP function on the third storage generation set of the sets of EDSs associated with the data object stored within the third DAP-based storage pool of SUs of the DAP-based storage pool set of SUs to generate fourth DAP scoring information;

selecting the second DAP-based storage pool of SUs as the DAP-based storage pool of SUs when the third DAP scoring information is higher than the fourth DAP scoring information; and

selecting the third DAP-based storage pool of SUs as the DAP-based storage pool of SUs when the fourth DAP scoring information is higher than the third DAP scoring information.

17. The method of claim 14 further comprising:

for the data segment, migrating at least a write threshold number of EDSs to the DAP-based storage pool of SUs among the DAP-based storage pool set of SUs, wherein the write threshold number of EDSs provides for a successful transfer of a set of the sets of EDSs from a first at least one location in the DSN to a second at least one location in the DSN; and

after migration of the at least a write threshold number of EDSs, re-provisioning at least one pool of SUs of the generation-based storage pool set of SUs to store at least one EDSs associated with another data object.

18. The method of claim 14, wherein:

a decode threshold number of EDSs are needed to recover the data segment associated with the set of the sets of EDSs;

a read threshold number of EDSs provides for reconstruction of the data segment associated with the set of the sets of EDSs; and

for the data segment, a write threshold number of EDSs provides for a successful transfer of the set of the sets of EDSs from a first at least one location in the DSN to a second at least one location in the DSN.

19. The method of claim 14, wherein the computing device includes a SU of the DAP-based storage pool set of SUs or the generation-based storage pool set of SUs within the DSN, a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device.

20. The method of claim 14, wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).