Data migration service

• US 9,960,979 B1
• Filed: 08/12/2013
• Issued: 05/01/2018
• Est. Priority Date: 03/12/2013
• Status: Active Grant

• Current Assignee: Western Digital Technologies Incorporated (Western Digital Corporation)
• Original Assignee: Western Digital Technologies Incorporated (Western Digital Corporation)
• Inventors: Blahaerath, Grant C., Hooper, Gary J.
• Primary Examiner(s): Won, Michael Y
• Application Number:
  US13/965,066
• Time in Patent Office: 1,723 Days
• Field of Search: 709212, 709213, 709223, 709688, 709692, 709694, 709802, 707668, 707692, 707694, 707802, 707688
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 3/0604 Improving or facilitating administration, e.g. storage management
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0665 at area level, e.g. provisioning of virtual or logical volumes
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 43/08 Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/562 Brokering proxy services

Abstract

Systems and methods for brokering storage services are disclosed. In one embodiment, the brokering system stores a mapping of stored data sets to online storage providers. The system can monitor one or more metrics associated with a plurality of online storage providers, identify one or more provider criteria associated with a first customer account, and obtain one or more metrics of a first storage provider storing a first data set associated with the first customer account. The system can determine whether to migrate the first data set based in part on the monitored metrics, the metrics of the first storage provider and the provider criteria. The system can auto-initiating migration of the first data set from the first storage provider to one or more selected storage providers of the plurality of online storage providers and update the mapping for the first data set.

Claims

1. A system for brokering a plurality of online storage services, the system comprising:

a memory that stores a mapping of stored data sets to online storage providers, each stored data set associated with a customer account; and

one or more processors configured to;

monitor one or more metrics associated with a plurality of separate online storage providers, each online storage provider associated with a different storage system;

identify one or more provider criteria associated with a first customer account;

obtain one or more metrics of a first online storage provider storing a first data set associated with the first customer account;

determine whether to migrate the first data set based in part on the monitored one or more metrics, the one or more metrics of the first online storage provider, and the provider criteria;

store the mapping in a temporary manifest;

in response to determining to migrate the first data set, auto-initiate migration of the first data set from the first online storage provider to one or more selected online storage providers of the plurality of online storage providers in one or more data transactions, based on a first access token associated with the first online storage provider and a second access token associated with the one or more selected online storage providers, wherein the mapping comprises the first access token;

update the mapping stored in the temporary manifest to replace the first access token with the second access token and to indicate the one or more data transactions;

determine whether the migration completed successfully or failed;

in response to determining that the migration completed successfully;

store the mapping in a permanent manifest;

delete the temporary manifest; and

respond to a request from a client device by providing the permanent manifest to the client device, wherein the permanent manifest enables the client device to access the first data set from the one or more selected online storage providers; and

in response to determining that the migration failed;

identify orphaned data transactions based on the temporary manifest; and

execute a cleaning service on the orphaned data transactions.

2. The system of claim 1, wherein the determination whether to migrate the first data set is performed periodically, without action from a customer.

3. The system of claim 1, wherein the one or more processors are further configured to select a time to auto-initiate the migration of the first data set based at least partly on a load of an interconnecting network between the first online storage provider and the one or more selected online storage providers.

4. The system of claim 1, wherein the one or more processors are further configured to determine the one or more selected online storage providers based at least partly on the monitored one or more metrics and the provider criteria.

5. The system of claim 1, wherein the monitored one or more metrics include at least one of storage pricing, connectivity costs, capacity, and availability.

6. The system of claim 1, further comprising a network interface in communication with a plurality of network attached storage (NAS) devices associated with a plurality of customers, the plurality of NAS devices configured to access respective data sets associated with the plurality of customers and stored by the online storage providers using the mapping.

7. The system of claim 1, wherein the one or more processors is further configured to:

identify a set of online storage providers having data centers in two or more geographically dispersed locations; and

replicate the first data set and parity data across the two or more geographically dispersed locations of the set of online storage providers such that the first data set is reconstructable from portions distributed among the set of online storage providers even if a first data center is offline.

8. A method for brokering a plurality of online storage services, the method comprising:

monitoring, by a computing system, one or more metrics associated with a plurality of separate online storage providers, each online storage provider associated with a different storage system;

identifying, by the computing system, one or more provider criteria associated with a first customer account;

obtaining, by the computing system, one or more metrics of a first online storage provider storing a first data set associated with the first customer account;

determining whether to migrate the first data set based in part on the monitored one or more metrics, the one or more metrics of the first online storage provider, and the provider criteria;

storing a mapping of stored data sets to online storage providers in a temporary manifest;

in response to determining to migrate the first data set, auto-initiating, by the computing system, migration of the first data set from the first online storage provider to one or more selected online storage providers in one or more data transactions, based on a first access token associated with the first online storage provider and a second access token associated with the one or more selected online storage providers;

updating the mapping by disassociating the first data set from the first online storage provider, associating the first data set with the one or more selected online storage providers, and replacing the first access token with the second access token, wherein the mapping comprises the first access token prior to the updating and the mapping is stored in the temporary manifest;

determining whether the migration completed successfully or failed;

responsive to determining that the migration completed successfully;

storing the mapping in a permanent manifest;

deleting the temporary manifest; and

responding to a request from a client device by providing the permanent manifest to the client device based on the request, wherein the permanent manifest enables the client device to access the first data set from the one or more selected online storage providers; and

responsive to determining that the migration failed;

identifying orphaned data transactions based on the temporary manifest; and

executing a cleaning service on the orphaned data transactions.

9. The method of claim 8, wherein the determination whether to migrate the first data set is performed periodically, without action from a customer.

10. The method of claim 8, further comprising selecting a time to auto-initiate the migration of the first data set based at least partly on a load of an interconnecting network between the first online storage provider and the one or more selected online storage providers.

11. The method of claim 8, further comprising determining the one or more selected online storage providers based at least partly on the monitored one or more metrics and the provider criteria.

12. The method of claim 8, wherein the monitored one or more metrics include at least one of storage pricing, connectivity costs, capacity, and availability.

13. The method of claim 8, further comprising providing access to the mapping, over a communications network, to a plurality of network attached storage (NAS) devices associated with a plurality of customers, the plurality of NAS devices configured to access respective data sets associated with the plurality of customers and stored by the online storage providers using the mapping.

14. The method of claim 8, wherein auto-initiating migration of the first data set comprises:

copying the first data set from the first online storage provider to the one or more selected online storage providers; and

deleting the first data set from the first online storage provider.

15. The method of claim 8, wherein auto-initiating migration of the first data set comprises:

copying the first data set from the first online storage provider to the one or more selected online storage providers; and

maintaining a copy of the first data set on the first online storage provider.

16. The method of claim 8, further comprising:

identifying a set of online storage providers having data centers in two or more geographically dispersed locations; and

replicating the first data set and parity data across the two or more geographically dispersed locations of the set of online storage providers such that the first data set is reconstructable from portions distributed among the set of online storage providers even if a first data center is offline.

17. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, direct a computing system to perform operations comprising:

monitoring one or more metrics associated with a plurality of separate online storage providers, each online storage provider associated with a different storage system;

identifying one or more provider criteria associated with a first customer account;

obtaining one or more metrics of a first online storage provider storing a first data set associated with the first customer account;

determining whether to migrate the first data set based in part on the monitored one or more metrics, the one or more metrics of the first online storage provider, and the provider criteria;

storing a mapping of stored data sets to online storage providers in a temporary manifest;

in response to determining to migrate the first data set, auto-initiating migration of the first data set from the first online storage provider to one or more selected online storage providers, based on a first access token associated with the first online storage provider and a second access token associated with the one or more selected online storage providers;

updating the mapping by disassociating the first data set from the first online storage provider, associating the first data set with the one or more selected online storage providers, and replacing the first access token with the second access token, wherein the mapping comprises the first access token prior to the updating and the mapping is stored in the temporary manifest;

determining whether the migration completed successfully or failed;

responsive to determining that the migration completed successfully;

storing the mapping in a permanent manifest;

deleting the temporary manifest; and

responding to a request from a client device by providing the permanent manifest to the client device, wherein the permanent manifest enables the client device to access the first data set from the one or more selected online storage providers; and

responsive to determining that the migration failed;

identifying orphaned data transactions based on the temporary manifest; and

executing a cleaning service on the orphaned data transactions.

18. The non-transitory computer-readable storage medium of claim 17, wherein the determination whether to migrate the first data set is performed periodically, without action from a customer.

19. The non-transitory computer-readable storage medium of claim 17, wherein the operations further comprise:

selecting a time to auto-initiate the migration of the first data set based at least partly on a load of an interconnecting network between the first online storage provider and the one or more selected online storage providers.

20. The non-transitory computer-readable storage medium of claim 17, wherein the operations further comprise:

providing access to the mapping, over a communications network, to a plurality of network attached storage (NAS) devices associated with a plurality of customers, the plurality of NAS devices configured to access respective data sets associated with the plurality of customers and stored by the online storage providers using the mapping.