Scheduling migration related traffic to be non-disruptive and performant

• US 10,031,669 B2
• Filed: 02/28/2017
• Issued: 07/24/2018
• Est. Priority Date: 12/29/2009
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Kazi, Asimuddin, Resch, Jason K.
• Primary Examiner(s): Parry, Chris
• Assistant Examiner(s): Stiltner, Weiwei
• Application Number:
  US15/445,056
• Publication Number:
  US 20170168720A1
• Time in Patent Office: 511 Days
• Field of Search: 709213
• CPC Class Codes: G06F 1/3296 by lowering the supply or operating voltage
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0772 Means for error signaling, e.g. using interrupts, exception flags, dedicated error registers
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/261 by simulating additional hardware, e.g. fault simulation
• CPC Class Codes: G06F 21/10 Protecting distributed programs or content, e.g. vending or licensing of copyrighted material protection in video systems or pay television H04N7/16 ; Digital rights management [DRM]
• CPC Class Codes: G06F 21/31 User authentication
• CPC Class Codes: G06F 21/62 Protecting access to data via a platform, e.g. using keys or access control rules
• CPC Class Codes: G06F 21/80 in storage media based on magnetic or optical technology, e.g. disks with sectors preventing unauthorised reproduction or copying of disc-type recordable media G11B20/00
• CPC Class Codes: G06F 2221/2129 Authenticate client device independently of the user
• CPC Class Codes: G06F 2221/2151 Time stamp
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0625 Power saving in storage systems
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0644 : Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/0647 : Migration mechanisms
• CPC Class Codes: G06F 3/065 : Replication mechanisms
• CPC Class Codes: G06F 3/0653 : Monitoring storage devices or systems
• CPC Class Codes: G06F 3/067 : Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06F 3/0683 : Plurality of storage devices
• CPC Class Codes: H03M 13/1515 : Reed-Solomon codes
• CPC Class Codes: H03M 13/3761 : using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H04L 1/16 : in which the return channel carries supervisory signals, e.g. repetition request signals
• CPC Class Codes: H04L 2209/34 : Encoding or coding, e.g. Huffman coding or error correction
• CPC Class Codes: H04L 63/101 : Access control lists [ACL]
• CPC Class Codes: H04L 63/102 : Entity profiles
• CPC Class Codes: H04L 67/06 : specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 9/085 : Secret sharing or secret splitting, e.g. threshold schemes
• CPC Class Codes: H04N 21/2181 : comprising remotely distributed storage units, e.g. when movies are replicated over a plurality of video servers distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: H04N 21/23116 : involving data replication, e.g. over plural servers synchronization of replicated data G06F11/1658; error detection or correction by means of data replication G06F11/2053; replication in distributed file systems G06F16/10; replication in distributed file systems G06F16/27; replication or mirroring of data in data networks H04L67/1095
• CPC Class Codes: H04N 21/232 : Content retrieval operation locally within server, e.g. reading video streams from disk arrays storage management G06F3/0604; details of querying and searching of video data from a database G06F16/739
• CPC Class Codes: H04N 21/274 : Storing end-user multimedia data in response to end-user request , e.g. network recorder

Abstract

A computing device includes an interface configured to interface and communicate with a dispersed storage network (DSN), a memory that stores operational instructions, and a processing module operably coupled to the interface and memory such that the processing module, when operable within the computing device based on the operational instructions, is configured to perform various operations. The computing device determines to facilitate migration of encoded data slices (EDSs) from a first storage unit (SU) pool to a second SU pool and identifies storage resources associated with the EDSs to be migrated. The computing device then generates a migration schedule for the EDSs based on performance information associated with storage resources and facilitates the migration of the plurality of EDSs based on the migration schedule using the storage resources based on an aggression factor and adapts the aggression factor as deemed necessary based on the performance information.

Claims

1. A computing device comprising:

an interface configured to interface and communicate with a dispersed or distributed storage network (DSN);

memory that stores operational instructions; and

a processing module operably coupled to the interface and to the memory, wherein the processing module, when operable within the computing device based on the operational instructions, is configured to;

determine to facilitate migration of a plurality of encoded data slices (EDSs) from a first storage unit (SU) pool to a second SU pool within the DSN, wherein a data object is segmented into a plurality of data segments, wherein a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce a set of EDSs that includes at least one of the plurality of EDSs;

identify storage resources associated with the plurality of EDSs to be migrated;

generate a migration schedule for the plurality of EDSs based on performance information associated with storage resources associated with the plurality of EDSs to be migrated;

facilitate, via the interface, the migration of the plurality of EDSs from the first SU pool to the second SU pool within the DSN based on the migration schedule using the storage resources based on a first aggression factor, wherein the first aggression factor corresponds to an acceptable level of impact to non-migration traffic within the DSN;

during and prior to completion of the migration of the plurality of EDSs based on the migration schedule, obtain performance information regarding the migration of the plurality of EDSs based on the migration schedule; and

when the performance information includes impact information that compares unfavorably with an aggression factor threshold;

modify the first aggression factor to generate a second aggression factor; and

continue to facilitate, via the interface, the migration of the plurality of EDSs from the first SU pool to the second SU pool within the DSN based on the migration schedule using the storage resources based on second aggression factor, wherein the aggression factor threshold corresponds to a maximum degree of tolerable acceptable level of impact to the non-migration traffic within the DSN.

2. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:

when the performance information includes migration rate information that compares unfavorably with a migration rate threshold;

modify the migration schedule to generate another migration schedule; and

continue to facilitate the migration of the plurality of EDSs based on the another migration schedule, wherein the migration rate threshold corresponds to an acceptable migration rate to effectuate migration of the plurality of EDSs based on the migration schedule.

3. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:

during and prior to the completion of the migration of the plurality of EDSs based on the migration schedule, obtain the performance information includes DSN resource availability information regarding the migration of the plurality of EDSs based on the migration schedule;

when the DSN resource availability information compares unfavorably with a DSN resource availability information threshold, increase a migration rate of the migration of the plurality of EDSs based on the migration schedule; and

when the DSN resource availability information compares favorably with the DSN resource availability information threshold, decrease or maintain the migration rate of the migration of the plurality of EDSs based on the migration schedule.

4. The computing device of claim 1, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:

determine to facilitate the migration of the plurality of EDSs from the first SU pool to the second SU pool based on at least one of detection of a SU retirement, detection of a SU replacement, detection of new storage resources, detection of new location weights of a Decentralized, or Distributed, Agreement Protocol (DAP) function associated with SU of the at least one of the first SU pool or the second SU pool, interpretation of a request received from another computing device, or interpretation of another migration schedule.

5. The computing device of claim 1, wherein:

the set of EDSs is of pillar width;

the set of EDSs are distributedly stored among a plurality of SUs;

a decode threshold number of EDSs are needed to recover the data segment;

a read threshold number of EDSs provides for reconstruction of the data segment; and

a write threshold number of EDSs provides for a successful transfer of the set of EDSs from a first at least one location in the DSN to a second at least one location in the DSN.

6. The computing device of claim 1, wherein the computing device is located at a first premises that is remotely located from at least one SU of the first SU pool or the second SU pool within the DSN.

7. The computing device of claim 1 further comprising:

a SU of the first SU pool or the second SU pool within the DSN, a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device.

8. The computing device of claim 1, wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).

9. A computing device comprising:

an interface configured to interface and communicate with a dispersed or distributed storage network (DSN);

memory that stores operational instructions; and

a processing module operably coupled to the interface and to the memory, wherein the processing module, when operable within the computing device based on the operational instructions, is configured to;

determine to facilitate migration of a plurality of encoded data slices (EDSs) from a first storage unit (SU) pool to a second SU pool within the DSN, wherein a data object is segmented into a plurality of data segments, wherein a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce a set of EDSs that includes that includes at least one of the plurality of EDSs, wherein the set of EDSs is of pillar width, the set of EDSs are distributedly stored among a plurality of SUs, a decode threshold number of EDSs are needed to recover the data segment, a read threshold number of EDSs provides for reconstruction of the data segment, and a write threshold number of EDSs provides for a successful transfer of the set of EDSs from a first at least one location in the DSN to a second at least one location in the DSN;

identify storage resources associated with the plurality of EDSs to be migrated;

generate a migration schedule for the plurality of EDSs based on performance information associated with storage resources associated with the plurality of EDSs to be migrated;

facilitate, via the interface, the migration of the plurality of EDSs based from the first SU pool to the second SU pool within the DSN on the migration schedule using the storage resources based on a first aggression factor, wherein the first aggression factor corresponds to an acceptable level of impact to non-migration traffic within the DSN;

during and prior to completion of the migration of the plurality of EDSs based on the migration schedule, obtain performance information regarding the migration of the plurality of EDSs based on the migration schedule; and

when the performance information includes impact information that compares unfavorably with an aggression factor threshold;

modify the first aggression factor to generate a second aggression factor; and

continue to facilitate, via the interface, the migration of the plurality of EDSs from the first SU pool to the second SU pool within the DSN based on the migration schedule using the storage resources based on second aggression factor, wherein the aggression factor threshold corresponds to a maximum degree of tolerable acceptable level of impact to the non-migration traffic within the DSN;

when the performance information includes migration rate information that compares unfavorably with a migration rate threshold;

modify the migration schedule to generate another migration schedule; and

continue to facilitate, via the interface, the migration of the plurality of EDSs from the first SU pool to the second SU pool within the DSN based on the another migration schedule, wherein the migration rate threshold corresponds to an acceptable migration rate to effectuate migration of the plurality of EDSs based on the migration schedule.

10. The computing device of claim 9, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:

during and prior to the completion of the migration of the plurality of EDSs based on the migration schedule, obtain the performance information includes DSN resource availability information regarding the migration of the plurality of EDSs based on the migration schedule;

when the DSN resource availability information compares unfavorably with a DSN resource availability information threshold, increase a migration rate of the migration of the plurality of EDSs based on the migration schedule; and

when the DSN resource availability information compares favorably with the DSN resource availability information threshold, decrease or maintain the migration rate of the migration of the plurality of EDSs based on the migration schedule.

11. The computing device of claim 9, wherein the processing module, when operable within the computing device based on the operational instructions, is further configured to:

determine to facilitate the migration of the plurality of EDSs from the first SU pool to the second SU pool based on at least one of detection of a SU retirement, detection of a SU replacement, detection of new storage resources, detection of new location weights of a Decentralized, or Distributed, Agreement Protocol (DAP) function associated with SU of the at least one of the first SU pool or the second SU pool, interpretation of a request received from another computing device, or interpretation of another migration schedule.

12. The computing device of claim 9 further comprising:

a SU of the first SU pool or the second SU pool within the DSN, a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device.

13. The computing device of claim 9, wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).

14. A method for execution by a computing device, the method comprising:

determining to facilitate migration of a plurality of encoded data slices (EDSs) from a first storage unit (SU) pool to a second SU pool within a dispersed or distributed storage network (DSN), wherein a data object is segmented into a plurality of data segments, wherein a data segment of the plurality of data segments is dispersed error encoded in accordance with dispersed error encoding parameters to produce a set of EDSs that includes that includes at least one of the plurality of EDSs;

identifying storage resources associated with the plurality of EDSs to be migrated;

generating a migration schedule for the plurality of EDSs based on performance information associated with storage resources associated with the plurality of EDSs to be migrated;

facilitating, via an interface of the computing device configured to interface and communicate with the DSN, the migration of the plurality of EDSs from the first SU pool to the second SU pool within the DSN based on the migration schedule using the storage resources based on a first aggression factor, wherein the first aggression factor corresponds to an acceptable level of impact to non-migration traffic within the DSN;

during and prior to completion of the migration of the plurality of EDSs based on the migration schedule, obtaining, via the interface of the computing device, performance information regarding the migration of the plurality of EDSs based on the migration schedule; and

when the performance information includes impact information that compares unfavorably with an aggression factor threshold;

modifying the first aggression factor to generate a second aggression factor; and

continuing to facilitate, via the interface, the migration of the plurality of EDSs from the first SU pool to the second SU pool within the DSN based on the migration schedule using the storage resources based on second aggression factor, wherein the aggression factor threshold corresponds to a maximum degree of tolerable acceptable level of impact to the non-migration traffic within the DSN.

15. The method of claim 14, when the performance information includes migration rate information that compares unfavorably with a migration rate threshold, further comprising:

modifying the migration schedule to generate another migration schedule; and

continuing to facilitate the migration of the plurality of EDSs based on the another migration schedule, wherein the migration rate threshold corresponds to an acceptable migration rate to effectuate migration of the plurality of EDSs based on the migration schedule.

16. The method of claim 14 further comprising:

during and prior to the completion of the migration of the plurality of EDSs based on the migration schedule, obtaining the performance information includes DSN resource availability information regarding the migration of the plurality of EDSs based on the migration schedule;

when the DSN resource availability information compares unfavorably with a DSN resource availability information threshold, increasing a migration rate of the migration of the plurality of EDSs based on the migration schedule; and

when the DSN resource availability information compares favorably with the DSN resource availability information threshold, decreasing or maintaining the migration rate of the migration of the plurality of EDSs based on the migration schedule.

17. The method of claim 14 further comprising:

determining to facilitate the migration of the plurality of EDSs from the first SU pool to the second SU pool based on at least one of detection of a SU retirement, detection of a SU replacement, detection of new storage resources, detection of new location weights of a Decentralized, or Distributed, Agreement Protocol (DAP) function associated with SU of the at least one of the first SU pool or the second SU pool, interpretation of a request received from another computing device, or interpretation of another migration schedule.

18. The method of claim 14, wherein:

the set of EDSs is of pillar width;

the set of EDSs are distributedly stored among a plurality of SUs;

a decode threshold number of EDSs are needed to recover the data segment;

a read threshold number of EDSs provides for reconstruction of the data segment; and

a write threshold number of EDSs provides for a successful transfer of the set of EDSs from a first at least one location in the DSN to a second at least one location in the DSN.

19. The method of claim 14, wherein the computing device includes a SU of the first SU pool or the second SU pool within the DSN, a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device.

20. The method of claim 14, wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).