DATA OBJECT STORE AND SERVER FOR A CLOUD STORAGE ENVIRONMENT, INCLUDING DATA DEDUPLICATION AND DATA MANAGEMENT ACROSS MULTIPLE CLOUD STORAGE SITES

• US 20100332456A1
• Filed: 03/31/2010
• Published: 12/30/2010
• Est. Priority Date: 06/30/2009
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Gokhale, Parag, Prahlad, Anand, Kottomtharayil, Rajiv, Kavuri, Srinivas, Muller, Marcus S., Vijayan, Manoj
• Granted Patent: US 8,285,681 B2
• Time in Patent Office: Days
• US Class Current: 707/664
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0626 Reducing size or complexity of storage systems
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0649 Lifecycle management
• CPC Class Codes: G06F 3/0667 at data level, e.g. file, record or object virtualisation
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0206 Price or cost determination based on market factors
• CPC Class Codes: G06Q 50/188 Electronic negotiation
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/06 : specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/535 : Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04L 67/56 : Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54
• CPC Class Codes: H04L 67/5682 : Policies or rules for updating, deleting or replacing the stored data
• CPC Class Codes: H04L 69/08 : Protocols for interworking; Protocol conversion
• CPC Class Codes: Y04S 40/20 : Information technology specific aspects, e.g. CAD, simulation, modelling, system security

Abstract

Systems and methods are disclosed for performing data storage operations, including content-indexing, containerized deduplication, and policy-driven storage, within a cloud environment. The systems support a variety of clients and cloud storage sites that may connect to the system in a cloud environment that requires data transfer over wide area networks, such as the Internet, which may have appreciable latency and/or packet loss, using various network protocols, including HTTP and FTP. Methods are disclosed for content indexing data stored within a cloud environment to facilitate later searching, including collaborative searching. Methods are also disclosed for performing containerized deduplication to reduce the strain on a system namespace, effectuate cost savings, etc. Methods are disclosed for identifying suitable storage locations, including suitable cloud storage sites, for data files subject to a storage policy. Further, systems and methods for providing a cloud gateway and a scalable data object store within a cloud environment are disclosed, along with other features.

Claims

1. A system to provide cloud-based data management services, wherein the system is communicatively coupled to multiple client computers via at least one network, and wherein the system is communicatively coupled to multiple cloud storage sites, the system comprising:

an object server node, including a secondary storage computing device, configured to create, on multiple, different cloud storage sites, secondary copies from logical groups of data objects, wherein the object server node further comprises—

an object server agent configured to;

receive data objects from the multiple client computers; and

provide a web-based interface to the multiple client computers to permit the multiple client computers to write, read, retrieve, and manipulate the data objects received by the object server agent and stored as secondary copies of the data objects on the cloud storage sites;

a data ingestion database configured to record information about each data object received by the object server agent from the multiple client computers, wherein the recorded information for each data object includes two or more of the following;

a unique token or universal resource identifier that identifies the data object;

a client computer or a user from which the data object was received;

a sub-client identifier that identifies a logical container and associated storage policy parameters that dictate handling or management of data objects within the logical container;

a location of an instance of the data object within the multiple cloud storage sites;

a location of deduplication information pertaining to the data object; and

a cryptographically unique identifier for the data object; and

wherein the secondary storage computing device is further configured to perform at least one of the following operations before copying the logical group of data objects to at least one of the cloud storage sites—

content indexing each data object in the logical group;

performing deduplication on the data objects in the logical group; and

encrypting the data objects in the logical group.

2. The system of claim 1, further comprising:

a primary data store configured to temporarily cache data objects received by the object server agent from the multiple client computers; and

wherein the data ingestion database includes information regarding a location of the data object within the primary data store, and a time the data object was received by the object server agent.

3. The system of claim 1 wherein the object server node provides a REST interface for the multiple client computers,

wherein the object server node includes a first set of network connections to the multiple client computers, wherein the multiple client computers lack a dedicated data agent or network client agent configured to communicate with the object server node, but the multiple client computers instead communicate with the object server node using standard client-based communication components including existing local area network (LAN) protocols or existing wide area network (WAN) protocols, and

wherein the object server node includes a second, different set of network connections to the multiple cloud storage sites using existing LAN protocols or existing WAN protocols.

4. The system of claim 1 wherein the object server node includes multiple sub-clients that represent logical containers associated with separate customers or separate client computers, and

wherein the object server node includes a file system to track all data objects stored by the object server node at the multiple cloud storage sites, but the object server node, via access control lists or access control entities, exposes to each separate customer or client computer only a subset of the data objects belonging to that separate customer or client computer.

5. The system of claim 1 wherein the object server node includes multiple sub-clients that represent logical containers, and

wherein at least some sub-clients include multiple logical storage sites that each correspond to a logical data ingestion point for at least some of the multiple client computers, and correspond via a URL or web directory, to one of the multiple cloud storage sites, and wherein the one cloud storage site is a cloud storage site logically reserved for providing data storage resources to a particular customer or company.

6. The system of claim 1 wherein the object server node includes multiple sub-clients that represent logical containers, wherein each logical container includes associated storage policy parameters that describe handling or management of data objects within the logical container.

7. The system of claim 1 wherein the system includes another object server node, wherein each object server node stores data objects having certain file types or file characteristics, wherein the file types or file characteristics are selected from the group consisting of:

text-based application files, image-based application files, audio-based application files, and video-based application files.

8. The system of claim 1 wherein the system includes another object server node, wherein each object server node stores data objects having certain file types or file characteristics, wherein the file types or characteristics are selected from the group consisting of:

documents, email objects, and media objects.

9. A method for managing a storage request from a requesting client computer to store a data object by an object store system, wherein the object store system is communicatively coupled between the client computer and one or more cloud storage sites via at least one network, the method comprising:

receiving a request to store the data object, and receiving an identifier for the data object and metadata associated with the data object;

determining one or more storage policy parameters applicable to the data object;

based on the received identifier, determining if the system currently has the data object stored in a manner consistent with the storage policy parameters applicable to the data object, and—

if the system currently has the data object stored in a manner consistent with the storage policy parameters applicable to the data object, then;

updating a deduplication database to associate previously stored blocks with the received request to store the data object,

storing the received metadata, and

storing in a local data store one or more references to at least one of;

(1) a stored copy of the data object, and (2) constituent blocks that form a stored copy of the data object;

if the system does not currently have the data object stored in a manner consistent with the storage policy parameters applicable to the data object, then;

requesting a copy of the data object from the client computer,

receiving a copy of the data object, and

storing the received data object and metadata in the local data store;

aggregating the data object and the received metadata stored in the local data store into a logical group of data objects,

wherein the logical group of data objects include additional data objects from the local data store, and

wherein the additional data objects are logically related to the logical group; and

directing storage of the logical group of data objects, as a secondary copy, within the one or more cloud storage sites.

10. The method of claim 9, wherein when the system does not currently have the data object stored in a manner consistent with the storage policy parameters applicable to the data object, then:

performing content indexing of content within the data object; and

encrypting or compressing the data object.

11. The method of claim 9, further comprising:

performing deduplication of the data object using one or more of the following deduplication methods;

block-level deduplication, and sub-object-level deduplication; and

directing storage of the logical group of data objects into an archive file within a specified one of multiple cloud storage sites.

12. The method of claim 9, further comprising:

when the system currently has the data object stored in a manner consistent with the storage policy parameters applicable to the data object, then

updating a deduplication database to associate previously stored blocks with the received request to store the data object.

13. The method of claim 9, further comprising:

charging a user of the client computer, or an entity associated with the client computer, for the data storage request on a subscription basis, a volume basis, or a mixed subscription/volume basis.

14. The method of claim 9, further comprising:

charging a user of the client computer, or an entity associated with the client computer, for the data storage request based on periodic subscription fees for unlimited number or size of data uploads and downloads, so long as a total amount of data stored at any time during the period does not exceed a certain limit.

15. The method of claim 9, further comprising:

charging an entity when a user or client computer affiliated with the entity performs various actions including at least three of;

a first rate for each unit of data received or directed to be stored at the one or more cloud storage sites;

a second rate for each unit of data stored at the one or more cloud storage sites during a unit of time;

a third rate for conducting a content-based search of data stored at the one or more cloud storage sites and that retrieves information about data objects stored therein;

a fourth rate for conducting a collaborative search operation upon data objects stored at the one or more cloud storage sites; and

a fifth rate for each unit of data retrieved or restored and served back to the client computer.

16. The method of claim 9, further comprising:

when the system currently has the data object stored in a manner consistent with the storage policy parameters applicable to the data object, then apportioning a cost of storing the data object between one or more of the following;

the requesting client computer, a different client computer, an entity associated with the requesting client computer, and an entity associated with a different client computer.

17. The method of claim 9, further comprising:

determining that the system currently does have the data object stored in a manner consistent with the storage policy parameters applicable to the data object, but that the stored object is older than a threshold, and then;

requesting a copy of the data object from the client computer,

receiving a copy of the data object, and

storing the received data object and metadata in the local data store, wherein the received metadata includes one or more of the following;

object-level security information, content tags, and storage policy parameters.

18. The method of claim 9, further comprising:

deduplicating data objects received from first and second different companies;

receiving and directing storage of a particular data object from the first company within the one or more cloud storage sites;

thereafter receiving from the second company the particular data object for storage;

only storing one or more references to at least one of;

(1) a stored copy of the particular data object, and (2) constituent blocks that form a stored copy of the particular data object;

receiving from the first company a request to delete the particular data object; and

updating one or more indices to no longer associate the particular data object with the first company, but not deleting the particular data object until the second company requests deletion of the particular data object.

19. A system for managing a storage request from a requesting client computer to store a data object that comprises multiple blocks, wherein the system is communicatively coupled between the client computer and at least one cloud storage site, the system comprising:

means for receiving a request to store a data object, wherein the data object includes metadata associated therewith;

means for identifying a logical group for the received data object,

wherein the means for identifying identifies the logical group based at least in part on the metadata received with the data object and a storage policy having storage policy parameters, and

wherein the logical group aggregates for storage data objects sharing a common characteristic;

means for determining, based on a cryptographically unique identifier for the data object, whether a copy of the data object is already stored in an archive file,

wherein the archive file is stored within a cloud storage site;

means for updating at least one of a deduplication database and the archive file when a copy of the data object is already stored in the archive file;

means for performing block level deduplication of the data object when a copy of the data object is not already stored in the archive file; and

means for updating an ingestion database to reflect at least the storage request and received metadata.

20. The system of claim 19 wherein the means for performing block level deduplication further comprises:

means for determining if a copy of each block in the data object is already stored in accordance with the storage policy parameters;

means for updating a deduplication database to associate the current storage request with a received block identifier, and for updating the archive file by adding a reference within the archive file that refers to a previously stored copy of the block, when the system already has a stored copy of a block; and

means for requesting a copy of a block associated with the block identifier, for receiving the requested block, for storing a copy of the received block within the archive file, and for updating the deduplication databases when system does not already have a stored copy of a block.