Data object store and server for a cloud storage environment, including data deduplication and data management across multiple cloud storage sites

• US 8,849,761 B2
• Filed: 09/14/2012
• Issued: 09/30/2014
• Est. Priority Date: 06/30/2009
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Prahlad, Anand, Muller, Marcus S., Kottomtharayil, Rajiv, Kavuri, Srinivas, Gokhale, Parag, Vijayan, Manoj
• Primary Examiner(s): Leroux, Etienne
• Application Number:
  US13/615,999
• Publication Number:
  US 20130024424A1
• Time in Patent Office: 746 Days
• Field of Search: 707/693, 707/803, 707/640, 705/4, 705/7, 709/226, 709/232
• US Class Current: 707/640
• CPC Class Codes: G06F 11/3485 for I/O devices
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: G06F 16/1844 Management specifically adapted to replicated file systems
• CPC Class Codes: G06F 16/41 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 3/06 Digital input from, or digital output to, record carriers , e.g. RAID, emulated record carriers or networked record carriers
• CPC Class Codes: G06F 3/0605 by facilitating the interaction with a user or administrator
• CPC Class Codes: G06F 3/061 Improving I/O performance
• CPC Class Codes: G06F 3/0626 Reducing size or complexity of storage systems
• CPC Class Codes: G06F 3/0631 by allocating resources to storage systems
• CPC Class Codes: G06F 3/0641 De-duplication techniques
• CPC Class Codes: G06F 3/0649 Lifecycle management
• CPC Class Codes: G06F 3/0667 at data level, e.g. file, record or object virtualisation
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 30/0206 Price or cost determination based on market factors
• CPC Class Codes: G06Q 50/188 Electronic negotiation
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/06 : specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/535 : Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04L 67/56 : Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54
• CPC Class Codes: H04L 67/5682 : Policies or rules for updating, deleting or replacing the stored data
• CPC Class Codes: H04L 69/08 : Protocols for interworking; Protocol conversion
• CPC Class Codes: Y04S 40/20 : Information technology specific aspects, e.g. CAD, simulation, modelling, system security

Abstract

Data storage operations, including content-indexing, containerized deduplication, and policy-driven storage, are performed within a cloud environment. The systems support a variety of clients and cloud storage sites that may connect to the system in a cloud environment that requires data transfer over wide area networks, such as the Internet, which may have appreciable latency and/or packet loss, using various network protocols, including HTTP and FTP. Methods are disclosed for content indexing data stored within a cloud environment to facilitate later searching, including collaborative searching. Methods are also disclosed for performing containerized deduplication to reduce the strain on a system namespace, effectuate cost savings, etc. Methods are disclosed for identifying suitable storage locations, including suitable cloud storage sites, for data files subject to a storage policy. Further, systems and methods for providing a cloud gateway and a scalable data object store within a cloud environment are disclosed, along with other features.

Claims

1. A method for scheduling storage operations on a cloud storage site, comprising:

receiving multiple new requests for cloud storage from one or more clients,

wherein the multiple new requests each include a request for data storage, and

wherein the multiple new requests each include information associated with the data storage requested;

determining a current capacity of the cloud storage site, wherein the current capacity of the cloud storage site is determined based at least in part on;

(i) a capacity policy,

wherein the capacity policy specifies preferences and criteria associated with allocating system resources for the cloud storage site, and

(ii) at least one of;

a quotation policy,

wherein the quotation policy includes a set of preferences and criteria associated with generating a quote in response to received client requests, and

a scheduled job,

wherein the scheduled job is associated with a quote for cloud storage accepted by a client,

a quoted job,

wherein the quoted job is associated with a quote for cloud storage provided to a client, and

queued requests,

wherein queued requests include requests by clients for cloud storage for which the respective clients have not been provided a quote;

identifying one or more approved requests,

wherein the one or more approved requests are identified from pending requests based at least in part on preferences and criteria specified in the accessed quotation policy and the current capacity,

wherein pending requests comprise the received multiple new requests and queued requests;

generating a responsive quote for each approved request,

wherein the responsive quotes are generated based at least in part on preferences and criteria specified in the accessed quotation policy, and

wherein each responsive quote includes one or more pricing values;

sending a generated responsive quote to a client associated with an approved request that the responsive quote was generated for; and

receiving from the client that was sent the generated responsive quote an indication of acceptance of the generated responsive quote.

2. The method of claim 1, wherein, the capacity policy specifies system resources available during specified periods, scheduled maintenance windows and current storage capacity available on servers.

3. The method of claim 1, further comprising determining system resources required for storage operations already scheduled or quoted.

4. The method of claim 1, wherein the quotation policy specifies at least three of:

a revenue function;

a pricing function;

a pricing rate table;

information associated with marketing promotions;

a list of preferred clients;

a list of disfavored clients;

classes of storage;

retention policies;

upload time periods;

data characteristics;

compression or encryption requirements; and

estimated or historic cost of storage, including a cost of power.

5. The method of claim 1, wherein the quotation policy specifies a revenue function that describes a method for numerically evaluating a projected revenue generated by the received requests.

6. The method of claim 1, wherein the quotation policy specifies a pricing function that describes a method for generating various pricing values for a responsive quote.

7. The method of claim 1, wherein identifying one or more approved requests further comprises excluding out of the received requests those that either do not satisfy minimum requirements specified by the quotation policy or cannot be accommodated due to a lack of system resources.

8. The method of claim 1, further comprising at least one of the following:

generating a responsive quote, to be sent to a client, having at least one term that is different from a term in a request received from the client; and

queuing a received request for later evaluation.

9. The method of claim 1, wherein identifying one or more approved requests further comprises identifying a set of requests that result in a maximum combined value of a revenue function.

10. The method of claim 1, wherein identifying one or more approved requests further comprises identifying a set of requests that result in a combined value of a revenue function that is sufficient to satisfy the quotation policy.

11. A non-transitory computer-readable storage medium whose contents cause a computer system to perform a method for scheduling storage operations on a cloud storage site, the method comprising:

receiving multiple new requests for cloud storage from one or more clients,

wherein the multiple new requests each include a request for data storage, and

wherein the multiple new requests each include information associated with the data storage requested;

determining a current capacity of the cloud storage site, wherein the current capacity of the cloud storage site is determined based at least in part on;

(i) a capacity policy,

wherein the capacity policy specifies preferences and criteria associated with allocating system resources for the cloud storage site, and

(ii) at least one of;

a quotation policy,

wherein the quotation policy includes a set of preferences and criteria associated with generating a quote in response to received client requests, and

a scheduled job,

wherein the scheduled job is associated with a quote for cloud storage accepted by a client,

a quoted job,

wherein the quoted job is associated with a quote for cloud storage provided to a client, and

queued requests,

wherein queued requests include requests by clients for cloud storage for which the respective clients have not been provided a quote;

identifying one or more approved requests,

wherein the one or more approved requests are identified from pending requests based at least in part on preferences and criteria specified in the accessed quotation policy and the current capacity,

wherein pending requests comprise the received multiple new requests and queued requests;

generating a responsive quote for each approved request,

wherein the responsive quotes are generated based at least in part on preferences and criteria specified in the accessed quotation policy, and,

wherein each responsive quote includes one or more pricing values;

sending a generated responsive quote to a client associated with an approved request that the responsive quote was generated for; and

receiving from the client that was sent the generated responsive quote an indication of acceptance of the generated responsive quote.

12. The non-transitory computer-readable storage medium of claim 11, wherein the capacity policy specifies system resources available during specified periods, scheduled maintenance windows and current storage capacity available on servers.

13. The non-transitory computer-readable storage medium of claim 11, further comprising determining system resources required for storage operations already scheduled or quoted.

14. The non-transitory computer-readable storage medium of claim 11, wherein the quotation policy specifies at least three of:

a revenue function;

a pricing function;

a pricing rate table;

information associated with marketing promotions;

a list of preferred clients;

a list of disfavored clients;

classes of storage;

retention policies;

upload time periods;

data characteristics;

compression or encryption requirements; and

estimated or historic cost of storage, including a cost of power.

15. The non-transitory computer-readable storage medium of claim 11, wherein the quotation policy specifies a revenue function that describes a method for numerically evaluating a projected revenue generated by the received requests.

16. The non-transitory computer-readable storage medium of claim 11, wherein the quotation policy specifies a pricing function that describes a method for generating various pricing values for a responsive quote.

17. The non-transitory computer-readable storage medium of claim 11, further comprising at least one of the following:

generating a responsive quote, to be sent to a client, having at least one term that is different from a term in a request received from the client; and

queuing a received request for later evaluation.

18. The non-transitory computer-readable storage medium of claim 11, wherein identifying one or more approved requests further comprises identifying a set of requests that result in a maximum combined value of a revenue function.

19. The non-transitory computer-readable storage medium of claim 11, wherein identifying one or more approved requests further comprises identifying a set of requests that result in a combined value of a revenue function that is sufficient to satisfy the quotation policy.

20. A system for scheduling storage operations on a cloud storage site, the system comprising:

means for receiving multiple new requests for cloud storage from one or more clients,

wherein the multiple new requests each include a request for data storage, and

wherein the multiple new requests each include information associated with the data storage requested;

means for determining a current capacity of the cloud storage site, wherein the current capacity of the cloud storage site is determined based at least in part on;

(i) a capacity policy,

wherein the capacity policy specifies preferences and criteria associated with allocating system resources for the cloud storage site, and

(ii) at least one of;

a quotation policy,

wherein the quotation policy includes a set of preferences and criteria associated with generating a quote in response to received client requests, and

a scheduled job,

wherein the scheduled job is associated with a quote for cloud storage accepted by a client,

a quoted job,

wherein the quoted job is associated with a quote for cloud storage provided to a client, and

queued requests,

wherein queued requests include requests by clients for cloud storage for which the respective clients have not been provided a quote;

means for identifying one or more approved requests,

wherein the one or more approved requests are identified from pending requests based at least in part on preferences and criteria specified in the accessed quotation policy and the current capacity,

wherein pending requests comprise the received multiple new requests and queued requests;

means for generating a responsive quote for each approved request,

wherein the responsive quotes are generated based at least in part on preferences and criteria specified in the accessed quotation policy, and

wherein each responsive quote includes one or more pricing values;

means for sending a generated responsive quote to a client associated with an approved request that the responsive quote was generated for; and

means for receiving from the client that was sent the generated responsive quote an indication of acceptance of the generated responsive quote.