DISTRIBUTED INDEXING SYSTEM FOR DATA STORAGE

• US 20100005151A1
• Filed: 07/02/2008
• Published: 01/07/2010
• Est. Priority Date: 07/02/2008
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Gokhale, Parag
• Granted Patent: US 8,335,776 B2
• Time in Patent Office: Days
• US Class Current: 709/216
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 16/20 of structured data, e.g. relational data
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/275 Synchronous replication
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A distributed indexing system spreads out the load on an index of stored data in a data storage system. Rather than maintain a single index, the distributed indexing system maintains an index in each media agent of a federated data storage system and a master index that points to the index in each media agent. In some embodiments, the distributed indexing system includes an index server (or group of servers) that handles indexing requests and forwards the requests to the appropriate distributed systems. Thus, the distributed indexing system, among other things, increases the availability and fault tolerance of a data storage index.

Claims

1. A method of storing index information describing secondary copies of data, the method comprising:

receiving at a media agent data copied during a first data storage operation, wherein the media agent is configured to convey data between a computer and one or more data storage devices associated with the media agent;

indexing the copied data to determine content from the copied data, wherein indexing the copied data creates indexed data;

selecting both a primary index server and at least a secondary index server among multiple available index servers for storing the indexed data, wherein the multiple index servers are networked together and collectively provide a distributed index;

sending a reference to the indexed data to both the primary index server and to the secondary index server, wherein the first and second index servers use the reference to access the indexed data and update the distributed index maintained by both the primary and secondary index servers;

receiving at the primary index server a new update to the distributed index, wherein the new update is associated with a second data storage operation;

determining that the primary index server is not available; and

,

updating the distributed index via the secondary index server based on the new update associated with the second data storage operation.

2. The method of claim 1, further comprising:

determining a fault during updating of the distributed index;

determining a location within the distributed index when the fault occurred; and

resuming the updating of the distributed index at the determined location, without having to update the entire distributed index.

3. The method of claim 1 wherein selecting the primary and secondary index servers comprises selecting based on an alternating rotation among at least some of the multiple index servers.

4. The method of claim 1, further comprising:

logging new index data on a network share and associated with a network address; and

providing to the primary and secondary index servers the network address.

5. The method of claim 1, further comprising:

receiving an indication from the primary index server that the primary index server has received the indexed data and updated the distributed index; and

instructing the media agent to delete the indexed data from the one or more data storage devices associated with the media agent.

6. The method of claim 1, further comprising:

receiving a request for a document or for information about the document;

determining a department or office within a business organization associated with the document;

forwarding the received request to one of multiple index servers is associated the determined department or office.

7. The method of claim 1 wherein selecting the primary index server includes selecting from multiple index servers an index server that is geographically closer to the media agent than other of the multiple index servers.

8. The method of claim 1 including, when the primary index server is unavailable, holding the indexed data at the media agent and later resending the indexed data to the primary index server.

9. The method of claim 1 wherein the media agent acts as an index server for its own data and selecting the primary index server comprises selecting the media agent.

10. The method of claim 1 selecting the primary and secondary index servers comprises accessing configuration information to determine which of the multiple index servers is currently associated with the media agent.

11. A distributed index system for maintaining an index of data storage information, the system comprising:

one or more index server components configured to store index data for one or more index data sources,

wherein the index server components form part of a federated data storage system having multiple, networked data storage components,

wherein a storage manager controls at least portions of the federated data storage system,

wherein each index server component creates or updates index data based on at least one scheduled storage policy provided by the storage manager, and

wherein the storage policy is a set of preferences associated with performing a data storage operation;

a main index component communicatively coupled to the index server components within the federated data storage system, wherein the main index component is configured to maintain a list of index server components and respond to requests for index data; and

an index data replication component communicatively coupled to the index server components within the federated data storage system, wherein the index data replication component is configured to replicate data stored on each index server component to at least one other index server component.

12. The system of claim 11 including, when the main index component is configured to:

receive a request for index data,

determine the index server component that has the requested index data, and

forward the request to the determined index server component.

13. The system of claim 11, further comprising an index configuration component configured to store configuration information that determines which index data sources are associated with each index server component.

14. The system of claim 11, further comprising an index failover component configured to update the list of index server components maintained by the main index component when a failover event occurs.

15. The system of claim 11 wherein the index replication component determines which data to replicate based on a failover policy.

16. A tangible computer-readable storage medium storing instructions for controlling a computer system to search for data objects in a data storage system using a distributed index, by a method comprising:

from a client or requesting computer, sending to a main index server a search request, wherein the search request specifies one or more criteria for identifying a data object to be restored from one of multiple data storage devices within a networked data storage system;

receiving from the main index server a network address of a distributed index server that manages index data for the identified data object;

from the client or requesting computer, sending to the distributed index server the search request;

receiving from the distributed index server a response based on the index data managed by the distributed index server, wherein the response satisfies the search request, and wherein the response includes a network location of the one data storage device within the networked data storage system; and

restoring the data object from the one data storage device based on the response, wherein the restoring of the data object includes restoring the data object to a computer other than the client or requesting computer.

17. The computer-readable medium of claim 16 wherein the distributed index server is associated with a media agent that manages data storage operations for a particular storage operation cell within a business organization, and wherein the networked data storage system is a federated data storage system, and wherein at least one storage policy instructs the federated data storage system to index the data object when first stored within the one data storage device.

18. The computer-readable medium of claim 16 wherein receiving from the main index server an address of a distributed index server comprises receiving multiple addresses for multiple index servers and sending the search request comprises sending the search request to multiple index servers.

19. The computer-readable medium of claim 16 wherein receiving from the main index server an address of a distributed index server comprises:

receiving a selection of an index server by the main index server based on the data requested by the search request,

determining that the selected index server is unavailable, and

accessing a secondary index server based on the data requested by the search request.

20. The computer-readable medium of claim 16 wherein receiving from the main index server an address of a distributed index server comprises receiving a selection of an index server by the main index server based on a location of the computer system sending the search request relative to each index server.