CLOUD STORAGE AND NETWORKING AGENTS, INCLUDING AGENTS FOR UTILIZING MULTIPLE, DIFFERENT CLOUD STORAGE SITES

• US 20100332818A1
• Filed: 03/31/2010
• Published: 12/30/2010
• Est. Priority Date: 06/30/2009
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Gokhale, Parag, Prahlad, Anand, Kottomtharayil, Rajiv, Kavuri, Srinivas, Vijayan, Manoj
• Granted Patent: US 8,849,955 B2
• Time in Patent Office: Days
• US Class Current: 713/150
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0626 Reducing size or complexity of storage systems
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0649 Lifecycle management
• CPC Class Codes: G06F 3/0667 at data level, e.g. file, record or object virtualisation
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0206 Price or cost determination based on market factors
• CPC Class Codes: G06Q 50/188 Electronic negotiation
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/06 : specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/535 : Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04L 67/56 : Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54
• CPC Class Codes: H04L 67/5682 : Policies or rules for updating, deleting or replacing the stored data
• CPC Class Codes: H04L 69/08 : Protocols for interworking; Protocol conversion
• CPC Class Codes: Y04S 40/20 : Information technology specific aspects, e.g. CAD, simulation, modelling, system security

Abstract

Systems and methods are disclosed for performing data storage operations, including content-indexing, containerized deduplication, and policy-driven storage, within a cloud environment. The systems support a variety of clients and cloud storage sites that may connect to the system in a cloud environment that requires data transfer over wide area networks, such as the Internet, which may have appreciable latency and/or packet loss, using various network protocols, including HTTP and FTP. Methods are disclosed for content indexing data stored within a cloud environment to facilitate later searching, including collaborative searching. Methods are also disclosed for performing containerized deduplication to reduce the strain on a system namespace, effectuate cost savings, etc. Methods are disclosed for identifying suitable storage locations, including suitable cloud storage sites, for data files subject to a storage policy. Further, systems and methods for providing a cloud gateway and a scalable data object store within a cloud environment are disclosed, along with other features.

Claims

1. A system for storing, on each of multiple target cloud storage sites, a secondary copy of an original data set, the system comprising:

a network agent comprising an hypertext transfer protocol (HTTP) subagent configured to establish and manage a network connection between the system and the multiple cloud storage sites,

wherein the network connection utilizes at least one of HTTP and HTTP over Transport Layer Security/Secure Sockets Layer,

wherein the multiple cloud storage sites are each operated by a different vendor, and

wherein each of the multiple cloud storage sites employs vendor-specific calls specified by an application programming interface for that specific cloud storage site; and

a cloud storage submodule configured to at least open, read, and write data files stored on each of the multiple cloud storage sites and to direct the multiple cloud storage sites to perform data storage operations, wherein the cloud storage submodule is configured to create a secondary copy of an original data set by at least buffering a series of received data transfer requests and a copy of a subset of the original data set;

converting a series of received generic file system commands to store the copy of the subset of the original data set into vendor-specific calls specified by the application programming interface utilized by a selected one of the multiple cloud storage sites; and

transferring the buffered copy of the subset of the original data set over the network connection established by the network agent to the selected one cloud storage site.

2. The system of claim 1 wherein the buffering includes buffering in two or more buffers coupled to, and located locally with, the network agent and the cloud storage submodule, and wherein the converting is performed after the two or more buffers are approximately filled.

3. The system of claim 1 wherein the network agent is further configured to perform at least two of the following:

accept or initiate connections to remote devices;

authenticate remote devices or specific users on remote devices;

receive data storage requests from remote devices;

log received data storage requests from remote devices;

compress or encrypt data;

serve data to remote devices;

redirect remote devices to other system components; and

implement bandwidth throttling, and wherein remote devices include client computers providing data storage requests and secondary data storage devices at cloud storage sites.

4. The system of claim 1 wherein the network agent is further configured to interpret routines, data structures, object classes, or protocols defined by the application programming interfaces for each of the multiple cloud storage sites.

5. The system of claim 1 wherein the cloud storage submodule is further configured to determine that one cloud storage site is underperforming based on previously specified performance metrics, and to direct the one cloud storage site to transfer at least some of the subset of the original data files to another cloud storage site, without first transferring the files back to a source from which the original data files originated.

6. The system of claim 1 wherein the cloud storage submodule is further configured to direct the one cloud storage site to transfer at least some of the subset of the original data files to another cloud storage site, without first transferring the files back to a source from which the original data files originated.

7. The system of claim 1 wherein the cloud storage submodule is further configured to:

manage authorization and connection information, wherein the authorization and connection information permits storage operations to the multiple cloud storage sites, and

to maintain performance metrics for a performance of each of the multiple cloud storage sites.

8. The system of claim 1 wherein the cloud storage submodule is further configured to test and record performance achieved by each of the multiple cloud storage sites, wherein performance data includes at least two of:

throughput of the cloud storage site, number of transmission failures that occurred to/from the cloud storage site, speed of data restorations from the cloud storage site, and speed of responses to queries from the cloud storage site.

9. The system of claim 1 wherein the cloud storage submodule is further configured to:

determine performance achieved by each of the multiple cloud storage sites,

dynamically or periodically adjust classifications assigned to each of the multiple cloud storage sites, and

take action for an underperforming cloud storage site, wherein the action includes requesting a reduced storage price, or transferring data from the underperforming cloud storage site to a different cloud storage site.

10. A system for managing storage of data within various storage resources, including local storage devices and remote cloud storage resources, wherein the system forms part of a storage operation cell hierarchy, wherein the storage operation cell hierarchy includes multiple storage operation cells organized in one or more hierarchical relationships, the system comprising:

one or more computing devices;

one or more local storage devices coupled to the one or more computing devices over a local or proprietary network, wherein the one or more local storage devices are configured to store data files from the one or more computing devices; and

a storage operation cell within the storage operation cell hierarchy, wherein the storage operation cell hierarchy includes multiple storage operation cells organized in one or more hierarchical relationships, wherein the storage operation cell includes—

a data agent component for accessing the data files of the one or more computing devices or the one or more local storage devices;

a secondary storage computing component for communicating with the one or more computing devices or one or more local storage devices, wherein the secondary storage computing component further comprises—

a network agent configured to establish a network connection between the secondary storage computing component and the cloud storage resources; and

a cloud storage submodule configured to request storage of the data files via the cloud storage resources, wherein the cloud storage submodule is further configured to;

convert received generic file system commands to store the data files into calls specified by an interface for the cloud storage resources; and

send at least a copy of the data files over the established network connection for storage by the cloud storage resources.

11. The system of claim 10, wherein the network agent comprises a subagent configured to establish and manage network connections between the secondary storage computing component and the cloud storage resources via at least a hypertext transfer protocol (HTTP), a file transfer protocol (FTP), or HTTP over Transport Layer Security/Secure Sockets Layer.

12. The system of claim 10, further comprising:

a storage manager component for managing the storage operation cell and the secondary storage computing component, and for managing other storage operation cells in the storage operation cell hierarchy, wherein the hierarchy includes at least two levels of cells below the storage manager component.

13. The system of claim 10, further comprising:

two or more local buffers to store received data transfer requests, wherein each buffer stores at least a portion of the data files before transfer to the cloud storage resources, and where each buffer has a storage capacity of approximately 128 kb.

14. The system of claim 10 wherein the cloud storage resources comprise multiple cloud storage sites each operated by a different vendor, wherein each of the multiple cloud storage sites employs vendor-specific application programming interfaces, and the cloud storage submodule is configured to convert received commands into vendor-specific application programming interface commands.

15. A system for storing, on each of multiple target cloud storage sites, a secondary copy of an original data set, the system comprising:

means for managing communications utilizing a network protocol, and for establishing a network connection, at least indirectly, with each of the multiple cloud storage sites,

wherein the multiple cloud storage sites are each operated by a different vendor, and

wherein each of the multiple cloud storage sites employ differing interfaces or commands for writing data file to or reading data files from the cloud storage site; and

means for transferring data files for storage within the cloud storage site, wherein the means for transferring includes means for providing storage commands to each of the multiple cloud storage sites using the differing interfaces or commands.

16. The system of claim 15 wherein the means for managing communications and the means for transferring data files are stored on a client computer, wherein the client computer is coupled to a local data store that stores a primary copy of the data files, and wherein the means for transferring data files creates a secondary copy of the data files on one of the multiple cloud storage sites by directly transferring to the one cloud storage site a copy of the primary data files.

17. A system for storing data from a client computer to a cloud storage site, the system comprising:

at least one network agent configured to—

establish and manage a first network connection between the system and the client computer, and

establish and manage a second network connection between the system and at least one of the multiple cloud storage sites;

at least one cloud storage submodule configured to at least open and read data files stored on each of the multiple cloud storage sites, and to write data files to each of the multiple cloud storage sites; and

a storage manager coupled to the at least one network agent and the at least one cloud storage submodule,

wherein the system is configured to provide software as a service (SaaS) to the client computer to permit the client computer to open and read data files stored on each of the multiple cloud storage sites, and to write data files to each of the multiple cloud storage sites.

18. The system of claim 17 wherein the first or second network connections utilize hypertext transfer protocol (HTTP) or file transfer protocol (FTP),

wherein each of the multiple cloud storage sites employs a different interface, and

wherein the cloud storage submodule is configured to translate commands received from the client computer into each of the different interfaces.

19. The system of claim 17 wherein the cloud storage submodule is further configured to:

perform additional processing of received data files, wherein the additional processing includes at least two of;

content indexing, encrypting, compressing, and data deduplicating;

convert file system commands received from the client computer into a specific command for one of the multiple cloud storage sites; and

transfer the additionally processed data files to the one cloud storage site over the second network connection.