Determining completion of migration in a dispersed storage network

• US 10,133,632 B2
• Filed: 04/11/2017
• Issued: 11/20/2018
• Est. Priority Date: 12/29/2009
• Status: Active Grant

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Leggette, Wesley B., Motwani, Manish, Resch, Jason K.
• Primary Examiner(s): Parry, Chris
• Assistant Examiner(s): Stiltner, Weiwei
• Application Number:
  US15/485,160
• Publication Number:
  US 20170220418A1
• Time in Patent Office: 588 Days
• Field of Search: 709213
• CPC Class Codes: G06F 1/3296 by lowering the supply or operating voltage
• CPC Class Codes: G06F 11/0727 in a storage system, e.g. in a DASD or network based storage system drivers for digital recording or reproducing units G06F3/06; circuits for error detection or correction within digital recording or reproducing units G11B20/18; for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS], H04L67/1097
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0772 Means for error signaling, e.g. using interrupts, exception flags, dedicated error registers
• CPC Class Codes: G06F 11/079 Root cause analysis, i.e. error or fault diagnosis in a hardware test environment G06F11/22; in a software test environment G06F11/36
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 21/10 Protecting distributed programs or content, e.g. vending or licensing of copyrighted material protection in video systems or pay television H04N7/16 ; Digital rights management [DRM]
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 21/80 in storage media based on magnetic or optical technology, e.g. disks with sectors preventing unauthorised reproduction or copying of disc-type recordable media G11B20/00
• CPC Class Codes: G06F 2221/2129 Authenticate client device independently of the user
• CPC Class Codes: G06F 2221/2151 Time stamp
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0635 by changing the path, e.g. traffic rerouting, path reconfiguration
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/0647 Migration mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/3761 : using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H04L 1/16 : in which the return channel carries supervisory signals, e.g. repetition request signals
• CPC Class Codes: H04L 2209/34 : Encoding or coding, e.g. Huffman coding or error correction
• CPC Class Codes: H04L 43/16 : Threshold monitoring
• CPC Class Codes: H04L 63/101 : Access control lists [ACL]
• CPC Class Codes: H04L 63/102 : Entity profiles
• CPC Class Codes: H04L 67/06 : specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 9/085 : Secret sharing or secret splitting, e.g. threshold schemes
• CPC Class Codes: H04L 9/0894 : Escrow, recovery or storing of secret information, e.g. secret key escrow or cryptographic key storage

Abstract

A method for determining completion of a data migration that results from a distributed agreement protocol (DAP) change within a distributed storage network (DSN). The method begins by transferring, in accordance with the DAP change, encoded data slices to one or more other storage units within the DSN. The method continues by maintaining a storage unit migration tracking repository that tracks migration of the encoded data slices. The method continues by maintaining a storage pool migration tracking repository based on the storage unit migration tracking repositories of the plurality of storage units. The method continues by maintaining a DSN migration tracking repository based on the storage pool migration tracking repositories of the plurality of storage pools. The method continues by indicating completion of the data migration as a result of the DAP change based on information within the DSN tracking repository.

Claims

1. A method for determining completion of a data migration that results from a distributed agreement protocol (DAP) change within a distributed storage network (DSN) that includes a plurality of storage pools, wherein a storage pool of the plurality of storage pools includes a plurality of storage units, the method comprises:

in accordance with the DAP change, transferring, by a storage unit of the plurality of storage units, encoded data slices to one or more other storage units within the DSN;

while transferring the encoded data slices, maintaining, by the storage unit, a storage unit migration tracking repository that tracks migration of the encoded data slices;

maintaining, by a storage pool monitoring module affiliated with the storage pool, a storage pool migration tracking repository based on the storage unit migration tracking repositories of the plurality of storage units, wherein the storage pool migration tracking repository tracks migration of a group of encoded data slices by the plurality of storage units;

maintaining, by a DSN monitoring module affiliated with the DSN, a DSN migration tracking repository based on the storage pool migration tracking repositories of the plurality of storage pools, wherein the DSN migration tracking repository tracks migration of a plurality of groups of encoded data slices by pluralities of storage units of the plurality of storage pools; and

indicating, by the DSN monitoring module, completion of the data migration as a result of the DAP change based on information within the DSN tracking repository.

2. The method of claim 1, wherein the maintaining the storage unit migration tracking repository comprises:

receiving a rebuild listing request that includes a range of DSN addresses;

identifying one or more encoded data slices of the encoded data slices being transferred as a result of the DAP change that have one or more DSN addresses within the range of DSN addresses to produce identified encoded data slices;

determining whether the identified encoded data slices have been transferred; and

when one or more of the identified encoded data slices have not been transferred;

determining one or more source names for the one or more encoded data slices; and

adding the one or more source names to the storage unit migration tracking repository.

3. The method of claim 2 further comprises:

when one or more of the identified encoded data slices have been transferred;

determining whether the one or more source names are in the storage unit migration tracking repository;

when the one or more source names are in the storage unit migration tracking repository; and

deleting the one or more source names from the storage unit migration tracking repository.

4. The method of claim 1, wherein the maintaining by the storage pool migration tracking repository comprises:

updating a first entry in the storage pool migration tracking repository for a first storage unit of the plurality of storage units based on a first status of migration by the first storage unit; and

updating a second entry in the storage pool migration tracking repository for a second storage unit of the plurality of storage units based on a second status of migration by the second storage unit.

5. The method of claim 4 further comprises:

determining the first status as non-complete when the storage unit migration tracking repository of the first storage unit includes at least one source name; and

determining the first status as complete when the storage unit migration tracking repository of the first storage unit is void of source names.

6. The method of claim 4 further comprises:

determining the first status as non-complete when the storage unit migration tracking repository of the first storage unit includes an indication that migration is incomplete for at least one source name; and

determining the first status as complete when the storage unit migration tracking repository of the first storage unit includes an indication that migration is complete for all source names listed in the storage unit migration tracking repository of the first storage unit.

7. The method of claim 1, wherein the maintaining the DSN migration tracking repository comprises:

updating a first entry in the DSN migration tracking repository for a first storage pool of the plurality of storage pools based on a first status of migration by the first storage pool; and

updating a second entry in the DSN migration tracking repository for a second storage pool of the plurality of storage pools based on a second status of migration by the second storage pool.

8. The method of claim 7 further comprises:

determining the first status as non-complete when the storage pool migration tracking repository of the first storage pool includes an indication that migration is incomplete for at least one storage unit of a first plurality of storage units associated with the first storage pool; and

determining the first status as complete when the storage pool migration tracking repository of the first storage pool includes an indication that migration is complete for all storage units of the first plurality of storage units.

9. The method of claim 1, wherein the indicating completion of the data migration comprises:

determining a status of each storage pool of the plurality of storage pools indicates completion of data migration for the respective storage pool.

10. A computer readable storage device for determining completion of a data migration that results from a distributed agreement protocol (DAP) change within a distributed storage network (DSN) that includes a plurality of storage pools, wherein a storage pool of the plurality of storage pools includes a plurality of storage units, the computer readable storage device comprises:

a first memory section that stores operational instructions that, when executed by a storage unit of the plurality of storage units of the DSN, causes the storage unit to;

transfer in accordance with the DAP change, encoded data slices to one or more other storage units within the DSN;

a second memory section that stores operational instructions that, when executed by the storage unit, causes the storage unit to;

while transferring the encoded data slices, maintain a storage unit migration tracking repository that tracks migration of the encoded data slices;

a third memory section that stores operational instructions that, when executed by a storage pool monitoring module affiliated with the storage pool, causes the storage pool monitoring module to;

maintain a storage pool migration tracking repository based on the storage unit migration tracking repositories of the plurality of storage units, wherein the storage pool migration tracking repository tracks migration of a group of encoded data slices by the plurality of storage units;

a fourth memory section that stores operational instructions that, when executed by a DSN monitoring module affiliated with the DSN, causes the DSN monitoring module to;

maintain a DSN migration tracking repository based on the storage pool migration tracking repositories of the plurality of storage pools, wherein the DSN migration tracking repository tracks migration of a plurality of groups of encoded data slices by pluralities of storage units of the plurality of storage pools; and

a fifth memory section that stores operational instructions that, when executed by the DSN monitoring module, causes the DSN monitoring module to;

indicate completion of the data migration as a result of the DAP change based on information within the DSN tracking repository.

11. The computer readable storage device of claim 10, wherein the second memory section stores further operational instructions that, when executed by the storage unit, causes the storage unit to maintaining the storage unit migration tracking repository by:

receiving a rebuild listing request that includes a range of DSN addresses;

identifying one or more encoded data slices of the encoded data slices being transferred as a result of the DAP change that have one or more DSN addresses within the range of DSN addresses to produce identified encoded data slices;

determining whether the identified encoded data slices have been transferred; and

when one or more of the identified encoded data slices have not been transferred;

determining one or more source names for the one or more encoded data slices; and

adding the one or more source names to the storage unit migration tracking repository.

12. The computer readable storage device of claim 11, wherein the second memory section stores further operational instructions that, when executed by the storage unit, causes the storage unit to:

determine whether the one or more source names are in the storage unit migration tracking repository;

when the one or more source names are in the storage unit migration tracking repository; and

delete the one or more source names from the storage unit migration tracking repository.

13. The computer readable storage device of claim 10, wherein the third memory section stores further operational instructions that, when executed by the storage pool monitoring module, causes the storage pool monitoring module to maintain the storage pool migration tracking repository by:

updating a first entry in the storage pool migration tracking repository for a first storage unit of the plurality of storage units based on a first status of migration by the first storage unit; and

updating a second entry in the storage pool migration tracking repository for a second storage unit of the plurality of storage units based on a second status of migration by the second storage unit.

14. The computer readable storage device of claim 13, wherein the third memory section stores further operational instructions that, when executed by the storage pool monitoring module, causes the storage pool monitoring module to:

determine the first status as non-complete when the storage unit migration tracking repository of the first storage unit includes at least one source name; and

determine the first status as complete when the storage unit migration tracking repository of the first storage unit is void of source names.

15. The computer readable storage device of claim 13, wherein the third memory section stores further operational instructions that, when executed by the storage pool monitoring module, causes the storage pool monitoring module to:

determine the first status as non-complete when the storage unit migration tracking repository of the first storage unit includes an indication that migration is incomplete for at least one source name; and

determine the first status as complete when the storage unit migration tracking repository of the first storage unit includes an indication that migration is complete for all source names listed in the storage unit migration tracking repository of the first storage unit.

16. The computer readable storage device of claim 10, wherein the fourth memory section stores further operational instructions that, when executed by the DSN monitoring module, causes the DSN monitoring module to maintain the DSN migration tracking repository by:

updating a first entry in the DSN migration tracking repository for a first storage pool of the plurality of storage pools based on a first status of migration by the first storage pool; and

updating a second entry in the DSN migration tracking repository for a second storage pool of the plurality of storage pools based on a second status of migration by the second storage pool.

17. The computer readable storage device of claim 16, wherein the fourth memory section stores further operational instructions that, when executed by the DSN monitoring module, causes the DSN monitoring module to:

determine the first status as non-complete when the storage pool migration tracking repository of the first storage pool includes an indication that migration is incomplete for at least one storage unit of a first plurality of storage units associated with the first storage pool; and

determine the first status as complete when the storage pool migration tracking repository of the first storage pool includes an indication that migration is complete for all storage units of the first plurality of storage units.

18. The computer readable storage device of claim 10, wherein the fifth memory section stores further operational instructions that, when executed by the DSN monitoring module, causes the DSN monitoring module to indicate completion of the data migration by:

determining a status of each storage pool of the plurality of storage pools indicates completion of data migration for the respective storage pool.