Cloud storage and networking agents, including agents for utilizing multiple, different cloud storage sites

• US 8,849,955 B2
• Filed: 03/31/2010
• Issued: 09/30/2014
• Est. Priority Date: 06/30/2009
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Prahlad, Anand, Kottomtharayil, Rajiv, Kavuri, Srinivas, Gokhale, Parag, Vijayan, Manoj
• Primary Examiner(s): KIM, EDWARD J
• Application Number:
  US12/751,923
• Publication Number:
  US 20100332818A1
• Time in Patent Office: 1,644 Days
• Field of Search: 709/221, 709212-215, 709/246, 709/249, 709/217, 709/218, 709/230
• US Class Current: 709/219
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0626 Reducing size or complexity of storage systems
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0649 Lifecycle management
• CPC Class Codes: G06F 3/0667 at data level, e.g. file, record or object virtualisation
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0206 Price or cost determination based on market factors
• CPC Class Codes: G06Q 50/188 Electronic negotiation
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/06 : specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/535 : Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04L 67/56 : Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54
• CPC Class Codes: H04L 67/5682 : Policies or rules for updating, deleting or replacing the stored data
• CPC Class Codes: H04L 69/08 : Protocols for interworking; Protocol conversion
• CPC Class Codes: Y04S 40/20 : Information technology specific aspects, e.g. CAD, simulation, modelling, system security

Abstract

Systems and methods are disclosed for performing data storage operations, including content-indexing, containerized deduplication, and policy-driven storage, within a cloud environment. The systems support a variety of clients and cloud storage sites that may connect to the system in a cloud environment that requires data transfer over wide area networks, such as the Internet, which may have appreciable latency and/or packet loss, using various network protocols, including HTTP and FTP. Methods are disclosed for content indexing data stored within a cloud environment to facilitate later searching, including collaborative searching. Methods are also disclosed for performing containerized deduplication to reduce the strain on a system namespace, effectuate cost savings, etc. Methods are disclosed for identifying suitable storage locations, including suitable cloud storage sites, for data files subject to a storage policy. Further, systems and methods for providing a cloud gateway and a scalable data object store within a cloud environment are disclosed, along with other features.

Claims

1. A system for storing, on each of multiple target cloud storage sites, a secondary copy of an original data set, the system comprising:

a memory having instructions; and

a processor coupled to the memory to execute the instructions, the instructions including;

a network agent comprising an hypertext transfer protocol (HTTP) subagent configured to establish and manage a network connection between the system and the multiple cloud storage sites,

wherein the network connection utilizes at least one of HTTP and HTTP over Transport Layer Security/Secure Sockets Layer,

wherein the multiple cloud storage sites are each operated by a different vendor, and

wherein each of the multiple cloud storage sites employs vendor-specific calls specified by an application programming interface for that specific cloud storage site; and

a cloud storage submodule configured to at least open, read, and write data files stored on each of the multiple cloud storage sites and to direct the multiple cloud storage sites to perform data storage operations, wherein the cloud storage submodule is configured to create a secondary copy of an original data set by at least;

determining that the original data set is not already buffered in a first buffer as a result of being associated with an earlier received data transfer request;

buffering in a second buffer as a substantially parallel process a series of received data transfer requests and a copy of a subset of the original data set,

wherein the buffering includes;

 

determining a latency of the network connection,

 

increasing buffer size for at least the second buffer when the latency of the network connection increases, and

 

decreasing the buffer size for at least the second buffer when the latency of the network connection decreases, to improve data transmission efficiency;

converting a series of received generic file system commands to store the copy of the subset of the original data set into vendor-specific calls specified by the application programming interface utilized by a selected one of the multiple cloud storage sites;

indexing content associated with the secondary copy, to create an index that facilitates searching of indexed content of the secondary copy while the secondary copy resides on one or more of the multiple cloud storage sites; and

transferring the buffered copy of the subset of the original data set over the network connection established by the network agent to the selected one cloud storage site, to create the secondary copy of the original data set at the selected one cloud storage site,

wherein buffering in the second buffer the copy of the subset of the original data set occurs prior to transferring the buffered copy of the subset of the original data set to the selected one cloud storage site,

wherein indexing content of the secondary copy occurs prior to transferring the buffered copy of the subset of the original data set to the selected one cloud storage site, and,

wherein the index for the indexed content is stored locally, and not at the selected one cloud storage site, to facilitate searching of the indexed content by locally networked client computing devices.

2. The system of claim 1 wherein the buffering in the second buffer includes buffering in two or more buffers coupled to, and located locally with, the network agent and the cloud storage submodule, and wherein the converting is performed after the two or more buffers are approximately filled.

3. The system of claim 1 wherein the network agent is further configured to perform at least two of the following:

accept or initiate connections to remote devices;

authenticate remote devices or specific users on remote devices;

receive data storage requests from remote devices;

log received data storage requests from remote devices;

compress or encrypt data;

serve data to remote devices;

redirect remote devices to other system components; and

implement bandwidth throttling, and wherein remote devices include client computers providing data storage requests and secondary data storage devices at cloud storage sites.

4. The system of claim 1 wherein the network agent is further configured to interpret routines, data structures, object classes, or protocols defined by the application programming interfaces for each of the multiple cloud storage sites.

5. The system of claim 1 wherein the cloud storage submodule is further configured to determine that one cloud storage site is underperforming based on previously specified performance metrics, and to direct the one cloud storage site to transfer at least some of the subset of the original data files to another cloud storage site, without first transferring the files back to a source from which the original data files originated.

6. The system of claim 1 wherein the cloud storage submodule is further configured to direct the one cloud storage site to transfer at least some of the subset of the original data files to another cloud storage site, without first transferring the files back to a source from which the original data files originated.

7. The system of claim 1 wherein the cloud storage submodule is further configured to:

manage authorization and connection information, wherein the authorization and connection information permits storage operations to the multiple cloud storage sites, and

to maintain performance metrics for a performance of each of the multiple cloud storage sites.

8. The system of claim 1 wherein the cloud storage submodule is further configured to test and record performance achieved by each of the multiple cloud storage sites, wherein performance data includes at least two of:

throughput of the cloud storage site, number of transmission failures that occurred to/from the cloud storage site, speed of data restorations from the cloud storage site, and speed of responses to queries from the cloud storage site.

9. The system of claim 1 wherein the cloud storage submodule is further configured to:

determine performance achieved by each of the multiple cloud storage sites,

dynamically or periodically adjust classifications assigned to each of the multiple cloud storage sites, and

take action for an underperforming cloud storage site, wherein the action includes requesting a reduced storage price, or transferring data from the underperforming cloud storage site to a different cloud storage site.

10. The system of claim 1, wherein the cloud storage submodule is further configured to create a secondary copy of an original data set by deduplicating the buffered copy of the subset of the original data set prior to transferring the buffered copy to the selected one cloud storage site.

11. The system of claim 1, wherein the cloud storage submodule determines a bandwidth of the network connection, wherein the cloud storage submodule is configured to perform network-aware container sizing by at least:

determining a size for a container file to utilize when deduplicating a secondary copy of the original data set,

wherein the container file size is determined at least in part on the latency, bandwidth, or both, associated with the network connection; and

deduplicating at least some of the data content in the secondary copy in order to create one or more container files containing deduplicated data, wherein at least one of the container files has the determined size.

12. A system for managing storage of data within various storage resources, including local storage devices and remote cloud storage resources, wherein the system forms part of a storage operation cell hierarchy, wherein the storage operation cell hierarchy includes multiple storage operation cells organized in one or more hierarchical relationships, the system comprising:

one or more computing devices;

one or more local storage devices coupled to the one or more computing devices over a local or proprietary network,

wherein the one or more local storage devices are configured to store data files from the one or more computing devices; and

a storage operation cell within the storage operation cell hierarchy, wherein the storage operation cell hierarchy includes multiple storage operation cells organized in one or more hierarchical relationships, wherein the storage operation cell includes—

a data agent component for accessing the data files of the one or more computing devices or the one or more local storage devices;

a secondary storage computing component for communicating with the one or more computing devices or one or more local storage devices, wherein the secondary storage computing component further comprises—

a network agent configured to establish a network connection between the secondary storage computing component and the cloud storage resources; and

a cloud storage submodule configured to request storage of the data files via the cloud storage resources, wherein the cloud storage submodule is further configured to;

determine that the data files are not already buffered in a first buffer as a result of being associated with an earlier received storage request,

buffer in a second buffer as a substantially parallel process a copy of a subset of the data files from the one or more computing devices,

 

wherein buffering the copy of the subset of the data files includes;

 

determining a latency of the network connection, and

 

adjusting a size of the at least the second buffer based on the determined latency of the network connection;

convert received generic file system commands to store the data files into calls specified by an interface for the cloud storage resources;

create a content index of the subset of the data files to facilitate searching of content of the data files by one or more computing device,

 

wherein the content index is stored on the local or proprietary network; and

after buffering the copy of the subset of the data files and after creating the content index, send the copy of the subset of the data files over the established network connection for storage by the cloud storage resources.

13. The system of claim 12, wherein the network agent comprises a subagent configured to establish and manage network connections between the secondary storage computing component and the cloud storage resources via at least a hypertext transfer protocol (HTTP), a file transfer protocol (FTP), or HTTP over Transport Layer Security/Secure Sockets Layer.

14. The system of claim 12, further comprising:

a storage manager component for managing the storage operation cell and the secondary storage computing component, and for managing other storage operation cells in the storage operation cell hierarchy, wherein the hierarchy includes at least two levels of cells below the storage manager component.

15. The system of claim 12, further comprising:

two or more local buffers to store received data transfer requests, wherein each buffer stores at least a portion of the data files before transfer to the cloud storage resources, and where each buffer has a storage capacity of approximately 128 kb.

16. The system of claim 12 wherein the cloud storage resources comprise multiple cloud storage sites each operated by a different vendor, wherein each of the multiple cloud storage sites employs vendor-specific application programming interfaces, and the cloud storage submodule is configured to convert received commands into vendor-specific application programming interface commands.

17. A system for storing, on each of multiple target cloud storage sites, a secondary copy of an original data set, the system comprising:

network connection means for managing communications utilizing a network protocol, and for establishing a network connection, at least indirectly, with each of the multiple cloud storage sites,

wherein the multiple cloud storage sites are each operated by a different vendor, and

wherein each of the multiple cloud storage sites employ differing interfaces or commands for writing data files to or reading data files from the cloud storage site

a cloud storage means for at least opening, reading, and writing data files stored on each of the cloud storage site and to direct the cloud storage site to perform data storage operations, wherein the cloud storage means creates the secondary copy of the original data set by at least;

determining that the original data set is not already buffered in a first buffer as a result of being associated with an earlier request to store a copy of data,

buffering in a second buffer as a substantially parallel process a copy of a subset of the original data set,

wherein the buffering includes;

determining a latency of the network connection, and

adjusting buffer sizes for at least the second buffer based on the latency of the network connection;

converting a series of received generic file system commands to store the copy of the subset of the original data set into vendor-specific commands utilized by a selected one of the multiple cloud storage sites; and

indexing content of the original data set to facilitate searching the content of the original data set by one or more local computing devices while the original data set resides on one or more of the multiple cloud storage sites as the secondary copy; and

data transfer means for transferring the original data set to storage within the cloud storage site, wherein the means for transferring includes transferring the original data set after buffering the copy and after indexing content of original data set.

18. The system of claim 17 wherein the means for managing communications and the means for transferring data files are stored on a client computer, wherein the client computer is coupled to a local data store that stores a primary copy of the data files, and wherein the means for transferring data files creates a secondary copy of the data files on one of the multiple cloud storage sites by directly transferring to the one cloud storage site a copy of the primary data files.

19. A method for storing, on each of multiple target cloud storage sites, a secondary copy of an original data set, the method comprising:

managing communications with a network protocol for establishing a network connection, at least indirectly, with each of the multiple cloud storage sites,

wherein the multiple cloud storage sites are each operated by a different vendor, and

wherein each of the multiple cloud storage sites employ differing interfaces or commands for writing data files to or reading data files from the cloud storage site;

determining that the original data set is not already buffered in a first buffer as a result of being associated with an earlier received data transfer request,

buffering in a second buffer as a substantially parallel process a series of received data transfer requests and a copy of a subset of the original data set,

wherein buffering includes;

determining a latency of the network connection, and

adjusting a size of at least the second buffer used for the buffering, based on the latency determined for the network connection;

deduplicating the buffered copy of the subset to reduce a duration of transmission of the buffered copy over the network connection;

creating a local copy of a content index associated with content of the buffered copy of the subset to facilitate local searching of at least part of the original data set after the subset of the original data set is transferred to a selected one of the multiple cloud storage sites;

converting a series of received generic file system commands to store the copy of the subset of the original data set into vendor-specific calls specified by an application programming interface utilized by the selected one of the multiple cloud storage sites,

wherein the differing interfaces or commands are the vendor-specific calls specified by the application programming interface; and

transferring the buffered copy of the subset of the original data set over the network connection to the selected one of the multiple cloud storage sites.

20. The method of claim 19 wherein said managing communications and the transferring of the buffered copy is performed by a client computer,

wherein the client computer is coupled to a local data store that stores the original data set.