PERFORMING DATA STORAGE OPERATIONS WITH A CLOUD STORAGE ENVIRONMENT, INCLUDING AUTOMATICALLY SELECTING AMONG MULTIPLE CLOUD STORAGE SITES

• US 20100332401A1
• Filed: 03/31/2010
• Published: 12/30/2010
• Est. Priority Date: 06/30/2009
• Status: Abandoned Application

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Gokhale, Parag, Prahlad, Anand, Kottomtharayil, Rajiv, Kavuri, Srinivas, Muller, Marcus S.
• Application Number:
  US12/751,651
• Publication Number:
  US 20100332401A1
• Time in Patent Office: Days
• US Class Current: 705/80
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0626 Reducing size or complexity of storage systems
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0649 Lifecycle management
• CPC Class Codes: G06F 3/0667 at data level, e.g. file, record or object virtualisation
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0206 Price or cost determination based on market factors
• CPC Class Codes: G06Q 50/188 Electronic negotiation
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/06 : specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/535 : Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04L 67/56 : Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54
• CPC Class Codes: H04L 67/5682 : Policies or rules for updating, deleting or replacing the stored data
• CPC Class Codes: H04L 69/08 : Protocols for interworking; Protocol conversion
• CPC Class Codes: Y04S 40/20 : Information technology specific aspects, e.g. CAD, simulation, modelling, system security

Abstract

Systems and methods are disclosed for performing data storage operations, including content-indexing, containerized deduplication, and policy-driven storage, within a cloud environment. The systems support a variety of clients and cloud storage sites that may connect to the system in a cloud environment that requires data transfer over wide area networks, such as the Internet, which may have appreciable latency and/or packet loss, using various network protocols, including HTTP and FTP. Methods are disclosed for content indexing data stored within a cloud environment to facilitate later searching, including collaborative searching. Methods are also disclosed for performing containerized deduplication to reduce the strain on a system namespace, effectuate cost savings, etc. Methods are disclosed for identifying suitable storage locations, including suitable cloud storage sites, for data files subject to a storage policy. Further, systems and methods for providing a cloud gateway and a scalable data object store within a cloud environment are disclosed, along with other features.

Claims

1. A method for identifying storage locations for a set of data files subject to a storage policy, wherein the set of data files is generated within a storage operation cell that has multiple client computers, and wherein the storage operation cell is coupled to multiple cloud storage sites via a network, the method comprising:

grouping the data files into at least one logical group of data files using a storage policy, wherein the storage policy defines classes of storage locations on which the set of data files may be stored;

determining aggregate storage requirements of a logical group of data files based at least in part on the storage policy;

identifying two or more candidate cloud storage sites to store a copy of the logical group of data files,

wherein each of the two or more candidate cloud storage sites are operated by independent organizations;

generating a request for quotes for storing a copy of the logical group of data files on one of the candidate cloud storage sites,

wherein the request for quotes includes the aggregate storage requirements of the logical group of data files;

transmitting the request for quotes to the two or more candidate cloud storage sites;

receiving one or more quotes from each of the two or more candidate cloud storage sites,

wherein the received quotes include a price;

identifying a target cloud storage site from the two or more candidate cloud storage sites by evaluating, based at least in part on the received quotes, storage costs of storing a copy of the logical group of data files;

wherein the storage costs include estimated monetary expenses associated with storing the logical group of data objects; and

transmitting for storage at least some of the logical group of data files from a client computer to the target cloud storage site.

2. The method of claim 1, wherein the storage policy defines classes of storage locations of cloud storage sites at least in part by a fault-tolerance and an access speed.

3. The method of claim 1, wherein the storage policy defines different categories of data files and classes of storage locations on which the different categories of data files may be stored.

4. The method of claim 1, wherein grouping the data files comprises grouping the set of data files based at least in part on a particular class of storage location.

5. The method of claim 1, wherein a quote received from a candidate cloud storage site further comprises at least two of:

a first pricing rate for an initial upload to the candidate cloud storage site;

a second pricing rate for downloads from the candidate cloud storage site;

a third pricing rate for searching or accessing the candidate cloud storage site; and

a fourth pricing rate for continued storage and maintenance of data on the candidate cloud storage site.

6. The method of claim 1, wherein a quote received from a candidate cloud storage site further comprises a time window during which the logical group of data files are to be transmitted to the candidate cloud storage site.

7. The method of claim 1, further comprising transmitting the request for quotes to at least one of:

a data storage dealer-broker, and a site that aggregates information related to cloud storage costs.

8. The method of claim 1, further comprising accessing information related to local storage devices within the storage operation cell, including information related to at least two of:

current or projected power consumption of the storage devices, current or projected power rates, acquisition cost of the storage devices, mean operating time of the storage devices, mean repair time of the storage devices, and mean data access rates of the storage devices.

9. The method of claim 1 further comprising transmitting another round of counteroffer quote requests.

10. A non-transitory computer-readable medium storing instructions that when , executed by a processor perform a method, the method comprising:

requesting, from two or more cloud storage sites, a quote for storing a copy of a set of data objects on a cloud storage site;

receiving two or more quotes from respective two or more cloud storage sites, wherein each quote reflects a pricing rate for storing a copy of the set of data objects on a cloud storage site;

calculating estimated costs of storing a copy of the set of data objects on two or more cloud storage sites using the received two or more quotes;

choosing a target cloud storage site that has a lowest calculated estimated cost; and

storing a copy of the set of data objects on the target cloud storage site.

11. The computer-readable medium of claim 10, wherein calculating estimated costs of storing a copy of the set of data objects on two or more cloud storage sites further comprises accessing historical or projected cost information related to the two or more cloud storage sites.

12. The computer-readable medium of claim 10, wherein requesting a quote includes specifying at least three of the following:

an amount of storage space required;

a unique identifier associated with a request;

a desired data transfer rate;

an upload time window or deadline;

an estimated storage lifetime;

a suggested pricing rate; and

a type of storage medium desired.

13. The computer-readable medium of claim 10, wherein calculating estimated costs of storing a copy of the set of data objects on two or more cloud storage sites further comprises evaluating a multi-variable numerical cost function that utilizes one or more of the following:

a quoted pricing rate;

a total amount of storage required;

a network load associated with uploading or downloading data;

a projected data access cost;

information related to site reliability, quality or reputation;

a geographical location of a cloud storage site; and

a mean data access rate.

14. The computer-readable medium of claim 10, wherein storing a copy of the set of data objects on the target cloud storage site comprises transmitting:

instructions regarding an identity of the target cloud storage site;

an identifier associated with a quote received from the target cloud storage site; and

information regarding a schedule for storing a copy of the set of data objects.

15. A method for scheduling storage operations on a cloud storage site, comprising:

determining a current capacity of the cloud storage site by accessing information relating to at least one of;

a capacity policy, a scheduled job, a quoted job, one or more queued requests, and a quotation policy that includes a set of preferences and criteria associated with generating a quote in response to auction client requests;

receiving multiple new requests for cloud storage from one or more auction clients;

identifying one or more winning requests that will receive responsive quotes by evaluating pending requests by applying preferences and criteria specified in the accessed quotation policy, wherein pending requests comprise the received new requests and the one or more queued requests;

generating one or more responsive quotes for winning requests by applying preferences and criteria specified in the accessed quotation policy, wherein the responsive quotes include one or more pricing values;

sending the one or more responsive quotes to one or more auction clients; and

receiving from one or more auction clients an indication of acceptance of one or more responsive quotes.

16. The method of claim 15, wherein, the capacity policy specifies system resources available for auction during specified periods, scheduled maintenance windows and current storage capacity available on servers.

17. The method of claim 15, further comprising determining system resources required for storage operations already scheduled or quoted.

18. The method of claim 15, wherein the quotation policy specifies at least three of:

a revenue function;

a pricing function;

a pricing rate table;

information associated with marketing promotions;

a list of preferred auction clients;

a list of disfavored auction clients;

classes of storage;

retention policies;

upload time periods;

data characteristics;

compression or encryption requirements; and

estimated or historic cost of storage, including a cost of power.

19. The method of claim 15, wherein the quotation policy specifies a revenue function that describes a method for numerically evaluating a projected revenue generated by the received requests.

20. The method of claim 15, wherein the quotation policy specifies a pricing function that describes a method for generating various pricing values for a responsive quote.

21. The method of claim 15, wherein identifying one or more winning requests further comprises identifying received requests that either do not satisfy minimum requirements specified by the quotation policy or cannot be accommodated due to a lack of system resources.

22. The method of claim 15, further comprising at least one of the following:

sending a responsive quote having at least one term that is different from a term in a received request;

sending an explicit rejection of a received request; and

queuing a received request for later evaluation.

23. The method of claim 15, wherein identifying one or more winning requests further comprises identifying a set of requests that results in a maximum combined value of a revenue function.

24. The method of claim 15, wherein identifying one or more winning requests further comprises identifying a set of requests that results in a combined value of a revenue function that is sufficient to satisfy the quotation policy.

25. A system for storing data from client computers at a cloud storage site, wherein the cloud storage site is coupled among multiple client computers via a network, the system comprising:

a reception module configured to receive a request to perform a storage operation at the cloud storage site from one of the client computers, wherein the request includes at least an aggregate size of data to be stored from the one client computer;

a quote module configured to identify and generate a responsive quote, wherein the responsive quote specifies at least a price for storing the aggregate size of data and a time window during which a storage operation for the aggregate size of data is to be performed by the cloud storage site; and

a transmission module configured to send the responsive quote to the one client computer.

26. The system of claim 25, further comprising an authentication module configured to authenticate the request to ensure that that the request is from a valid client computer.

27. The system of claim 25, wherein a responsive quote further specifies at least two of:

a first pricing rate for an initial upload of data;

a second pricing rate for download of data;

a third pricing rate for searching or accessing data; and

and a fourth pricing rate for continued storage of data.

28. A system for identifying storage locations for a set of data files subject to a storage policy, wherein the set of data files is generated within a storage operation cell that has multiple client computers, and wherein the storage operation cell is coupled to multiple cloud storage sites via a network, the system comprising:

means for grouping the data files into at least one logical group of data files using a storage policy, wherein the storage policy defines classes of storage locations on which the set of data files may be stored;

means for determining aggregate storage requirements of a logical group of data files based at least in part on the storage policy;

means for identifying two or more candidate cloud storage sites to store a copy of the logical group of data files,

wherein each of the two or more candidate cloud storage sites are operated by independent organizations;

means for generating a request for quotes for storing a copy of the logical group of data files on one of the candidate cloud storage sites,

wherein the request for quotes includes the aggregate storage requirements of the logical group of data files;

means for identifying a target cloud storage site from the two or more candidate cloud storage sites by evaluating, based at least in part on received quotes, storage costs of storing a copy of the logical group of data files;

wherein the storage costs include estimated monetary expenses associated with storing the logical group of data objects; and

means for transmitting for storage at least some of the logical group of data files from a client computer to the target cloud storage site.