Distributed indexing system for data storage

• US 8,335,776 B2
• Filed: 07/02/2008
• Issued: 12/18/2012
• Est. Priority Date: 07/02/2008
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Gokhale, Parag
• Primary Examiner(s): Ehichioya, Fred I
• Assistant Examiner(s): BOWEN, RICHARD L
• Application Number:
  US12/166,854
• Publication Number:
  US 20100005151A1
• Time in Patent Office: 1,630 Days
• Field of Search: None
• US Class Current: 707/696
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 16/20 of structured data, e.g. relational data
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/275 Synchronous replication
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A distributed indexing system spreads out the load on an index of stored data in a data storage system. Rather than maintain a single index, the distributed indexing system maintains an index in each media agent of a federated data storage system and a master index that points to the index in each media agent. In some embodiments, the distributed indexing system includes an index server (or group of servers) that handles indexing requests and forwards the requests to the appropriate distributed systems. Thus, the distributed indexing system, among other things, increases the availability and fault tolerance of a data storage index.

Claims

1. A method of storing index information describing secondary copies of data, the method comprising:

receiving at a media agent executing in one or more computer processors, data copied during a first data storage operation, wherein the media agent is configured to convey data from a primary copy of data between a client computer and one or more first data storage devices associated with the media agent to create a secondary copy of the primary copy of data, the primary copy of data generated by one or more software applications running on the client computer and stored in a data store associated with the client computer that is separate from the one or more first data storage devices;

indexing with the media agent the secondary copy to determine content from the secondary copy, wherein indexing the secondary copy creates a first index of indexed data,

wherein the first index is associated with a primary index server and wherein the indexed data comprises information about the content of the secondary copy and information about location of the secondary copy on the first data storage devices;

selecting at least a secondary index server among multiple available index servers based on a failover policy wherein the secondary index server is configured to store a second index that is created using the first index and is a replica of the first index, wherein the multiple index servers are networked together and collectively provide a distributed index;

sending from the media agent a reference to the indexed data associated with the secondary copy to the primary index server and to the secondary index server, wherein the primary index server and the secondary index server retrieve the indexed data from the media agent using the reference such that the second index remains a replica of the first index;

updating the distributed index using the indexed data retrieved from the media agent;

receiving an index update about migrated data associated with a migration of at least a portion of the secondary copy from the first data storage devices to at least a second data storage device based on one or more storage policies;

wherein the index update comprises information about the new location of the migrated data on the second storage device and information about the content of the migrated data;

determining that the primary index server is not available; and

updating the distributed index about the migrated data using the index update retrieved from the media agent via the secondary index server.

2. The method of claim 1, further comprising:

determining a fault during updating of the distributed index;

determining a location within the distributed index when the fault occurred; and

resuming the updating of the distributed index at the determined location, without having to update the entire distributed index.

3. The method of claim 1 further comprising selecting the primary index server based on an alternating rotation among at least some of the multiple index servers.

4. The method of claim 1, further comprising:

logging new index data on a network share and associated with a network address; and

providing to the primary and secondary index servers the network address.

5. The method of claim 1, further comprising:

receiving an indication from the primary index server that the primary index server has received the indexed data and updated the distributed index; and

instructing the media agent to delete the indexed data from the one or more data storage devices associated with the media agent.

6. The method of claim 1, further comprising:

receiving a request for a document or for information about the document;

determining a department or office within a business organization associated with the document;

forwarding the received request to one of multiple index servers is associated the determined department or office.

7. The method of claim 1 further comprising selecting the primary index server at least in part by selecting from the multiple index servers an index server that is geographically closer to the media agent than other of the multiple index servers.

8. The method of claim 1, further comprising when the primary index server is unavailable, holding the indexed data at the media agent.

9. The method of claim 1 wherein the media agent acts as an index server for its own data and wherein the method further comprises selecting the primary index server at least in part by selecting the media agent.

10. The method of claim 1 further comprising selecting the primary index at least in part by accessing configuration information to determine which of the multiple index servers is currently associated with the media agent.

11. A distributed index system for maintaining an index of data storage information related to secondary copies of data, the system comprising:

a memory;

one or more computer processors;

at least one media agent executing in the one or more computer processors that is configured to receive data copied during a first data storage operation, wherein the media agent is configured to convey data from a primary copy of data between a client computer and one or more first data storage devices associated with the media agent to create a secondary copy of the primary copy of data, the primary copy of data generated by one or more software applications running on the client computer and stored in a data store associated with the client computer and that is separate from the one or more first data storage devices,

wherein the media agent is further configured to index the secondary copy to create index data that comprises information about content of the secondary copy and information about location of the secondary copy on the first storage devices;

multiple index server components configured to store index data for one or more index data sources,

wherein each index server component is configured to;

receive a reference from a media agent to the index data of the secondary copy stored at the media agent,

retrieve the index data from the media agent using the reference received from the media agent, and

update an index associated with each index server with the index data retrieved from the media agent;

wherein the multiple index server components are networked together and collectively provide a distributed index;

a primary index server component from the index server components that has a first index created from the indexed data provided by the media agent;

a failover component that selects, based on a failover policy, at least a secondary index server component from the index server components, wherein the secondary index server component comprises a second index that is a replica of the first index created using the first index;

wherein the media agent generates an index update about migrated data associated with a migration of at least a portion of the secondary copy from the first data storage devices to at least a second data storage device based on one or more storage policies and wherein the index update comprises information about the new location of the migrated data on the second storage device and information about the content of the migrated data; and

wherein when the primary index server is not available, the distributed index is updated using the index update about the migrated data via the secondary index server.

12. The system of claim 11, wherein the main index component is configured to:

receive a request for index data,

determine the index server component that has the requested index data, and

forward the request to the determined index server component.

13. The system of claim 11, further comprising an index configuration component configured to store configuration information that determines which index data sources are associated with each index server component.

14. The system of claim 11, wherein the index failover component is further configured to update the list of index server components maintained by a main index component when a failover event occurs.

15. The system of claim 11, wherein an index replication component determines which data to replicate based on the failover policy.