Accelerating slice transfers utilizing multiple interfaces

• US 10,055,290 B2
• Filed: 06/21/2016
• Issued: 08/21/2018
• Est. Priority Date: 06/30/2015
• Status: Expired due to Fees

• Current Assignee: Pure Storage, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Iljazi, Ilir
• Primary Examiner(s): Lamarre, Guy
• Application Number:
  US15/187,923
• Publication Number:
  US 20170004042A1
• Time in Patent Office: 791 Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 2201/805 Real-time
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/064 Management of blocks
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H03M 13/1515 Reed-Solomon codes
• CPC Class Codes: H03M 13/33 Synchronisation based on error coding or decoding
• CPC Class Codes: H03M 13/3761 using code combining, i.e. using combining of codeword portions which may have been transmitted separately, e.g. Digital Fountain codes, Raptor codes or Luby Transform [LT] codes
• CPC Class Codes: H04L 65/611 for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/51 Discovery or management thereof, e.g. service location protocol [SLP] or web services

Abstract

Based on a detected data transfer instruction, a computing device within a dispersed storage network (DSN) determines a data transfer synchronization protocol that substantially maintains synchronization of at least the write threshold number of first associated slices (e.g., a first row of encoded data slices) to be transferred from the first set of storage units (SUs) to a second set of SUs based on a substantially same first transfer rate and substantially maintains synchronization of at least the write threshold number of second associated slices (e.g., a second row of encoded data slices) to be transferred from the first set of SUs to the second set of SUs based on a substantially same second transfer rate. The computing device then executes the data transfer synchronization protocol to perform substantially synchronized transfer of respective sets of the plurality of sets of encoded data slices from the first set of SUs to the second set of SUs.

Claims

1. A computing device comprising:

an interface for interfacing with a dispersed storage network (DSN);

memory that stores operational instructions; and

a processing module operably coupled to the interface and to the memory, wherein the processing module, when operable within the computing device based on the operational instructions, is configured to;

detect a data transfer instruction that directs transfer of a plurality of sets of encoded data slices from a first set of storage units (SUs) to a second set of SUs via at least one of a plurality of DSN interfaces between the first set of SUs and the second set of SUs within the DSN, wherein;

a data object is segmented into a plurality of data segments that includes a first data segment corresponding to a first row and a second data segment corresponding to a second row;

the first data segment is dispersed error encoded to produce a first row of encoded data slices, a read threshold number of the first row of encoded data slices provides for reconstruction of the first data segment, and a write threshold number of the first row of encoded data slices provides for a successful transfer of the first row of encoded data slices;

the second data segment is dispersed error encoded to produce a second row of encoded data slices, the read threshold number of the second row of encoded data slices provides for reconstruction of the second data segment, and the write threshold number of the second row of encoded data slices provides for the successful transfer of the first row of encoded data slices;

the first row of encoded data slices is distributedly stored in the first set of SUs such that a first encoded data slice of the first row of encoded data slices is stored in a first SU of the first set of SUs, and a second encoded data slice of the first row of encoded data slices is stored in a second SU of the first set of SUs;

the second row of encoded data slices is distributedly stored in the first set of SUs such that a first encoded data slice of the second row of encoded data slices is stored in the first SU of the first set of SUs, and a second encoded data slice of the second row of encoded data slices is stored in the second SU of the first set of SUs;

determine a data transfer synchronization protocol that maintains synchronization of at least the write threshold number of the first row of encoded data slices to be transferred from the first set of SUs to the second set of SUs based on a same first transfer rate and maintains synchronization of at least the write threshold number of the second row of encoded data slices to be transferred from the first set of SUs to the second set of SUs based on a same second transfer rate; and

execute the data transfer synchronization protocol to perform synchronized transfer of respective sets of the plurality of sets of encoded data slices from the first set of SUs to the second set of SUs including to perform synchronized transfer of a first set of the plurality of sets of encoded data slices during a first time based on the same first transfer rate and to perform synchronized transfer of a second set of the plurality of sets of encoded data slices during a second time based on the same second transfer rate.

2. The computing device of claim 1, wherein the processing module is further configured to:

determine the data transfer synchronization protocol including to identify a first DSN interface of the plurality of DSN interfaces that supports synchronized transfer of the at least the write threshold number of the first row of encoded data slices from the first set of SUs to the second set of SUs based on the same first transfer rate;

execute the data transfer synchronization protocol to transfer perform synchronized transfer of the at least the write threshold number of the first row of encoded data slices from the first set of SUs to the second set of SUs during the first time based on the same first transfer rate via the first DSN interface;

determine the data transfer synchronization protocol including to identify a second DSN interface of the plurality of DSN interfaces that supports synchronized transfer of the at least the write threshold number of the second row of encoded data slices from the first set of SUs to the second set of SUs based on the same second transfer rate; and

execute the data transfer synchronization protocol to perform synchronized transfer of the at least the write threshold number of the second row of encoded data slices from the first set of SUs to the second set of SUs during the second time based on the same second transfer rate via the second DSN interface.

3. The computing device of claim 1, wherein the processing module is further configured to:

determine the data transfer synchronization protocol including to identify a first DSN interface of the plurality of DSN interfaces to transfer the first encoded data slice of the first row of encoded data slices that is stored in the first SU of the first set of SUs to a first SU of the second set of SUs and to identify a second DSN interface of the plurality of DSN interfaces to transfer the second encoded data slice of the first row of encoded data slices that is stored in the second SU of the first set of SUs to a first SU of the second set of SUs such that the first DSN interface and the second DSN interface cooperatively support synchronized transfer of the first encoded data slice and the second encoded data slice during the first time based on the same first transfer rate; and

execute the data transfer synchronization protocol to perform synchronized transfer of the first encoded data slice via the first DSN interface and transfer of the second encoded data slice via the second DSN interface during the first time based on the same first transfer rate.

4. The computing device of claim 1, wherein the processing module is further configured to:

receive, from another computing device within the DSN, the data transfer instruction that directs transfer of the plurality of sets of encoded data slices from the first set of SUs to the second set of SUs via the at least one of the plurality of DSN interfaces between the first set of SUs and the second set of SUs within the DSN.

5. The computing device of claim 1, wherein the processing module is further configured to:

when execution of the data transfer synchronization protocol includes fewer than all of the plurality of sets of encoded data slices, perform transfer of at least one remaining encoded data slice of the plurality of sets of encoded data slices after execution of the data transfer synchronization protocol.

6. The computing device of claim 1, wherein the computing device is located at a first premises that is remotely located the first set of SUs and the second set of SUs within the DSN.

7. The computing device of claim 1, wherein the plurality of DSN interfaces includes at least one of a wireless local area network (WLAN) interface, a local area network (LAN) interface, a wired interface, a satellite interface, a wireless interface, a fiber-optic interface, or a mobile interface.

8. The computing device of claim 1, wherein the computing device includes a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device.

9. The computing device of claim 1, wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).

10. A method for execution by a computing device, the method comprising:

detecting a data transfer instruction that directs transfer, within a dispersed storage network (DSN), of a plurality of sets of encoded data slices from a first set of storage units (SUs) to a second set of SUs via at least one of a plurality of DSN interfaces between the first set of SUs and the second set of SUs within the DSN, wherein;

a data object is segmented into a plurality of data segments that includes a first data segment corresponding to a first row and a second data segment corresponding to a second row;

the first data segment is dispersed error encoded to produce a first row of encoded data slices, a read threshold number of the first row of encoded data slices provides for reconstruction of the first data segment, and a write threshold number of the first row of encoded data slices provides for a successful transfer of the first row of encoded data slices;

the second data segment is dispersed error encoded to produce a second row of encoded data slices, the read threshold number of the second row of encoded data slices provides for reconstruction of the second data segment, and the write threshold number of the second row of encoded data slices provides for the successful transfer of the first row of encoded data slices;

the first row of encoded data slices is distributedly stored in the first set of SUs such that a first encoded data slice of the first row of encoded data slices is stored in a first SU of the first set of SUs, and a second encoded data slice of the first row of encoded data slices is stored in a second SU of the first set of SUs;

the second row of encoded data slices is distributedly stored in the first set of SUs such that a first encoded data slice of the second row of encoded data slices is stored in the first SU of the first set of SUs, and a second encoded data slice of the second row of encoded data slices is stored in the second SU of the first set of SUs;

determining a data transfer synchronization protocol that maintains synchronization of at least the write threshold number of the first row of encoded data slices to be transferred from the first set of SUs to the second set of SUs based on a same first transfer rate and maintains synchronization of at least the write threshold number of the second row of encoded data slices to be transferred from the first set of SUs to the second set of SUs based on a same second transfer rate; and

executing the data transfer synchronization protocol to perform synchronized transfer of respective sets of the plurality of sets of encoded data slices from the first set of SUs to the second set of SUs including performing synchronized transfer of a first set of the plurality of sets of encoded data slices during a first time based on the same first transfer rate and performing synchronized transfer of a second set of the plurality of sets of encoded data slices during a second time based on the same second transfer rate.

11. The method of claim 10 further comprising:

determining the data transfer synchronization protocol including to identify a first DSN interface of the plurality of DSN interfaces that supports synchronized transfer of the at least the write threshold number of the first row of encoded data slices from the first set of SUs to the second set of SUs based on the same first transfer rate;

executing the data transfer synchronization protocol to transfer perform synchronized transfer of the at least the write threshold number of the first row of encoded data slices from the first set of SUs to the second set of SUs during the first time based on the same first transfer rate via the first DSN interface;

determining the data transfer synchronization protocol including to identify a second DSN interface of the plurality of DSN interfaces that supports synchronized transfer of the at least the write threshold number of the second row of encoded data slices from the first set of SUs to the second set of SUs based on the same second transfer rate; and

executing the data transfer synchronization protocol to perform synchronized transfer of the at least the write threshold number of the second row of encoded data slices from the first set of SUs to the second set of SUs during the second time based on the same second transfer rate via the second DSN interface.

12. The method of claim 10 further comprising:

determining the data transfer synchronization protocol including to identify a first DSN interface of the plurality of DSN interfaces to transfer the first encoded data slice of the first row of encoded data slices that is stored in the first SU of the first set of SUs to a first SU of the second set of SUs and to identify a second DSN interface of the plurality of DSN interfaces to transfer the second encoded data slice of the first row of encoded data slices that is stored in the second SU of the first set of SUs to a first SU of the second set of SUs such that the first DSN interface and the second DSN interface cooperatively support synchronized transfer of the first encoded data slice and the second encoded data slice during the first time based on the same first transfer rate; and

executing the data transfer synchronization protocol to perform synchronized transfer of the first encoded data slice via the first DSN interface and transfer of the second encoded data slice via the second DSN interface during the first time based on the same first transfer rate.

13. The method of claim 10 further comprising:

receiving, from another computing device within the DSN and via an interface of the computing device, the data transfer instruction that directs transfer of the plurality of sets of encoded data slices from the first set of SUs to the second set of SUs via the at least one of the plurality of DSN interfaces between the first set of SUs and the second set of SUs within the DSN.

14. The method of claim 10 further comprising:

when execution of the data transfer synchronization protocol includes fewer than all of the plurality of sets of encoded data slices, performing transfer of at least one remaining encoded data slice of the plurality of sets of encoded data slices after execution of the data transfer synchronization protocol.

15. The method of claim 10, wherein the computing device is located at a first premises that is remotely located the first set of SUs and the second set of SUs within the DSN.

16. The method of claim 10, wherein the plurality of DSN interfaces includes at least one of a wireless local area network (WLAN) interface, a local area network (LAN) interface, a wired interface, a satellite interface, a wireless interface, a fiber-optic interface, or a mobile interface.

17. The method of claim 10, wherein the computing device includes a wireless smart phone, a laptop, a tablet, a personal computers (PC), a work station, or a video game device.

18. The method of claim 10, wherein the DSN includes at least one of a wireless communication system, a wire lined communication system, a non-public intranet system, a public internet system, a local area network (LAN), or a wide area network (WAN).