CLOUD GATEWAY SYSTEM FOR MANAGING DATA STORAGE TO CLOUD STORAGE SITES

• US 20100333116A1
• Filed: 03/31/2010
• Published: 12/30/2010
• Est. Priority Date: 06/30/2009
• Status: Abandoned Application

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Vijayan, Manoj, Gokhale, Parag, Prahlad, Anand, Kottomtharayil, Rajiv, Kavuri, Srinivas, Muller, Marcus S.
• Application Number:
  US12/751,953
• Publication Number:
  US 20100333116A1
• Time in Patent Office: Days
• US Class Current: 719/328
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0626 Reducing size or complexity of storage systems
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0649 Lifecycle management
• CPC Class Codes: G06F 3/0667 at data level, e.g. file, record or object virtualisation
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0206 Price or cost determination based on market factors
• CPC Class Codes: G06Q 50/188 Electronic negotiation
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/06 : specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/535 : Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04L 67/56 : Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54
• CPC Class Codes: H04L 67/5682 : Policies or rules for updating, deleting or replacing the stored data
• CPC Class Codes: H04L 69/08 : Protocols for interworking; Protocol conversion
• CPC Class Codes: Y04S 40/20 : Information technology specific aspects, e.g. CAD, simulation, modelling, system security

Abstract

Systems and methods are disclosed for performing data storage operations, including content-indexing, containerized deduplication, and policy-driven storage, within a cloud environment. The systems support a variety of clients and cloud storage sites that may connect to the system in a cloud environment that requires data transfer over wide area networks, such as the Internet, which may have appreciable latency and/or packet loss, using various network protocols, including HTTP and FTP. Methods are disclosed for content indexing data stored within a cloud environment to facilitate later searching, including collaborative searching. Methods are also disclosed for performing containerized deduplication to reduce the strain on a system namespace, effectuate cost savings, etc. Methods are disclosed for identifying suitable storage locations, including suitable cloud storage sites, for data files subject to a storage policy. Further, systems and methods for providing a cloud gateway and a scalable data object store within a cloud environment are disclosed, along with other features.

Claims

1. A cloud gateway system for storing, on a target cloud storage site, a secondary copy of an original data set that comprises data blocks, wherein the cloud gateway system is coupled between one or more client computers and one or more cloud storage sites via a network, the cloud gateway system comprising:

a data reception component configured to receive the original data set from a client computer;

a local cache configured to buffer the original data set received from the client computer before the secondary copy of the original data set is stored on the target cloud storage site;

a callback layer configured to intercept calls for the original data set between a file system and the cache and to track the intercepted calls to provide information regarding when the original data set is changed, updated, and/or accessed by the file system;

a data migration component configured to transfer some or all of the original data set buffered in the cache, wherein the data migration component is further configured to receive information from the callback layer regarding when the original data set is changed, updated, and/or accessed by the file system; and

a media agent component, further comprising;

a network agent configured to establish and manage a network connection between the media agent and the target cloud storage site using at least one of HTTP and HTTP over Transport Layer Security/Secure Sockets Layer; and

a cloud storage submodule configured to open, read, and write data files stored on the target cloud storage site and direct the target cloud storage site to perform data storage operations, wherein the cloud storage submodule is further configured to;

convert received file system commands to store a copy of data blocks from the original data set into specific calls specified by an application programming interface utilized by the target cloud storage site; and

transfer at least some of the contents of the local cache over the network connection for storage at the target cloud storage site.

2. The cloud gateway system of claim 1 wherein the media agent transfers to the target cloud storage site less than all of the data blocks of the original data set when the system determines from a cryptographically unique identifier associated with each block that some of the data blocks are already stored in the target cloud storage site, and wherein the system includes a data location table mapping the less than all of the data blocks to a logical location in the target cloud storage site.

3. The cloud gateway system of claim 1, further comprising a division component that divides data objects into sub-objects, and updates one or more indexes that maintain information to associate particular data objects with their corresponding sub-objects.

4. The cloud gateway system of claim 1, further comprising a division component that divides data objects into sub-objects based on rules, wherein the rules define a minimum or maximum data size for a sub-object, a time of creation for data within a sub-object, or a type of data within a sub-object.

5. The cloud gateway system of claim 1, further comprising a division component that divides data objects into sub-objects, wherein the data object is at least one electronic mailbox, and wherein a first sub-object includes an index of the mailbox, a second sub-object includes sent electronic mail messages for the mailbox, a third sub-object includes received electronic mail messages for the mailbox, a fourth sub-object includes deleted electronic mail messages for the mailbox, and a fifth sub-object includes attachments to electronic mail messages for the mailbox.

6. A method for storing a secondary copy of an original data set on a cloud storage site using a cloud gateway, wherein the cloud gateway is coupled between one or more client computers and one or more cloud storage sites via a network, the method comprising:

identifying data blocks within a cache of the cloud gateway that satisfy certain criteria,

wherein the original data set comprises data blocks,

wherein the certain criteria are from a storage policy, and

wherein the certain criteria include time-based criteria;

performing block-level deduplication of the identified data blocks to create a deduplicated set of data, wherein the block-level deduplication includes—

determining a size for a container file to utilize when deduplicating the identified data blocks; and

deduplicating at least some of the identified data blocks to create one or more container files containing deduplicated data,

wherein at least one of the container files has the determined size; and

storing the deduplicated set of data on the cloud storage site by;

buffering data for later transmission to the cloud storage site;

repeating the following steps while the data buffer is not full;

receiving a file system request to write a group of data to the cloud storage site; and

adding the group of data to the buffer;

converting a file system request to one or more application program interface calls associated with the cloud storage site; and

transmitting contents of the buffer to the cloud storage site using the one or more application program interface calls associated with the cloud storage site.

7. The method of claim 6, further comprising identifying the cloud storage site on which to store the secondary copy of the original data set by:

identifying two or more candidate cloud storage sites;

accessing a storage policy having a set of preferences and storage criteria, wherein the set of preferences and storage criteria includes at least two of the following;

one or more preferred cloud storage sites;

one or more preferred classes or quality of cloud storage sites;

requirements regarding deduplication of the original data set,

requirements regarding encryption of the original data set,

requirements regarding compression of the original data set,

quality of a network connection available to the cloud storage site;

one or more data retention periods;

data characteristics of at least some data in the original data set;

estimated or historic usage associated with operating one or more system components;

frequency with which the original data set was accessed or modified during a particular time period;

a specified level of fault tolerance; and

,

one or more geographical locations or political states in which data storage devices for a cloud storage site exist; and

selecting at least one of the two or more of the candidate cloud storage sites based at least in part on the set of preferences and storage criteria in the storage policy.

8. The method of claim 6, further comprising identifying the cloud storage site on which to store the secondary copy of the original data set by selecting a cloud storage site based at least in part on data storage devices for the cloud storage site being in locations receiving electrical power from two different power grids.

9. The method of claim 6, further comprising:

identifying the cloud storage site on which to store the secondary copy of the original data set by selecting a first cloud storage site to store a secondary copy of at least some of the original data set for a first time period; and

,

selecting a second cloud storage site on which to store another secondary copy of at least some of the original data set for a second, longer time period.

10. The method of claim 6, further comprising:

identifying the cloud storage site on which to store the secondary copy of the original data set by selecting a first cloud storage site to store a secondary copy of at least a first portion of the original data set; and

,

selecting a second cloud storage site to store another secondary copy of at least a second portion of the original data set, wherein the first portion of the original data set includes emails or data objects from a particular organization or group.

11. The method of claim 6 wherein the contents of the buffer are transmitted to the cloud storage site using at least one of hypertext transfer protocol (HTTP) and HTTP over Transport Layer Security/Secure Sockets Layer.

12. The method of claim 6 wherein the contents of the buffer are transmitted to the cloud storage site using REST protocols.

13. The method of claim 6, wherein the original data set includes one or more data objects, and wherein the method further comprises content indexing the original data set by performing the following for each data object in the original data set:

analyzing content of the data object, including analyzing a summary of the data object as well as analyzing additional content of the data object; and

based upon the analysis, generating metadata corresponding to the data object, wherein the metadata includes at least a logical address to the cloud storage site for accessing the data object; and

storing in a content index the generated metadata of the data object, wherein the content index is not stored at the cloud storage site, but is locally accessible by the cloud gateway.

14. A system for creating a secondary copy of an original data set using a cloud storage site, wherein the original data set is received from one or more client computers, the system comprising:

means for identifying sub-objects of the original data set that satisfy certain criteria, wherein the certain criteria are related a storage policy;

means for performing deduplication of the identified data sub-objects to create a deduplicated set of data; and

,

means for storing the deduplicated set of data on the cloud storage site, wherein the means for storing includes;

means for buffering data for later transmission to the cloud storage site;

means for converting file system requests into application program interface calls associated with the cloud storage site; and

,

means for transmitting the buffered data to the cloud storage site using the one or more application program interface calls associated with the cloud storage site.

15. The system of claim 14, further comprising:

means for determining a size for a container file and for deduplicating at least some of the data sub-objects to create one or more container files containing deduplicated data, wherein at least one of the container files has the determined size.

16. The system of claim 14 wherein the means for buffering further comprises:

means for receiving a file system request to write a group of data to the cloud storage site; and

means for adding the group of data to the buffer.

17. A non-transitory computer-readable medium storing instructions that when executed by a processor perform a method for utilizing cloud storage resources to store at least a first portion of at least one data object within a network attached storage (NAS) device, wherein the NAS device includes a NAS file system and a non-volatile data store, and wherein the NAS device is communicatively coupled to access the cloud storage resources, the method comprising:

accessing calls to or from the NAS file system for reading of data from or writing of data to the non-volatile data store of the NAS device,

wherein the at least one data object consists of multiple data blocks,

wherein the non-volatile data store of the NAS device stores the multiple data blocks of the at least one data object;

wherein the NAS file system of the NAS device controls the reading of data from or the writing of data to the multiple data blocks of the at least one data object, and

wherein the accessing includes identifying individual blocks or groups of blocks within the multiple data blocks of the at least one data object that the NAS file system of the NAS device reads data from or writes data to;

based on the accessing, identifying a portion of the multiple data blocks of the at least one data object that satisfies a data storage criteria; and

automatically transferring the identified portion of the multiple data blocks for storage by the cloud storage resources.

18. A stand-alone cloud gateway device, coupled to one or more external computing devices over a network, wherein at least one cloud storage site is also connected to the cloud gateway device via the network, the cloud gateway device comprising:

at least one processor;

a communication component coupled to the at least one processor and associated with a network address for the stand-alone cloud gateway device,

wherein the communication component receives data transfer commands from the one or more external computing devices on the network,

wherein the one or more external computing devices direct the data transfer commands to the cloud gateway device via the network address for the cloud gateway device, and

wherein the data transfer commands direct operation of the cloud gateway device;

a non-volatile, internal data store, coupled to the at least one processor, wherein the internal data store stores data objects;

a data storage component that comprises program code, which when executed by the processor, performs data storage tasks with respect to the internal data store;

a file system that comprises program code, which when executed by the processor, stores and organizes data objects stored in the internal data store;

a network agent that comprises program code, which when executed by the processor, establishes and manages network connections between the media agent component and the cloud storage site; and

a cloud storage submodule that comprises program code, which when executed by the processor—

converts received file system commands to specific application programming interface calls utilized by the cloud storage site; and

transfers the data objects stored in the internal data store to the cloud storage site over the network.

19. The stand-alone cloud gateway device of claim 18, further comprising:

a call intercept layer, in communication with the file system, that comprises program code, which when executed by the processor, recognizes calls to or from the file system for reading of data from or writing of data to individual data blocks or groups of data blocks stored within the internal data store; and

a data block identification component, in communication with the call intercept layer, that comprises program code, which when executed by the processor, identifies data blocks of the data objects that satisfy a criteria, wherein the criteria is associated with recognized calls to or from the file system for reading of data from or the writing of data to the individual data blocks or groups of data blocks.