Distributed indexing system for data storage

• US 8,805,807 B2
• Filed: 11/16/2012
• Issued: 08/12/2014
• Est. Priority Date: 07/02/2008
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Gokhale, Parag
• Primary Examiner(s): BOWEN, RICHARD L
• Application Number:
  US13/679,651
• Publication Number:
  US 20130173536A1
• Time in Patent Office: 634 Days
• Field of Search: None
• US Class Current: 707/696
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 16/20 of structured data, e.g. relational data
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/275 Synchronous replication
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A distributed indexing system spreads out the load on an index of stored data in a data storage system. Rather than maintain a single index, the distributed indexing system maintains an index in each media agent of a federated data storage system and a master index that points to the index in each media agent. In some embodiments, the distributed indexing system includes an index server (or group of servers) that handles indexing requests and forwards the requests to the appropriate distributed systems. Thus, the distributed indexing system, among other things, increases the availability and fault tolerance of a data storage index.

Claims

1. A method of storing distributed index information describing secondary copies of data, the method comprising:

copying primary data stored in one or more primary storage devices to one or more secondary storage devices, wherein copying the primary data to the secondary storage device creates a secondary copy on the one or more secondary storage devices, the primary data generated by one or more software applications running on a client computer;

indexing a first portion of the secondary copy stored on the one or more secondary storage devices, wherein indexing the first portion of the secondary copy creates a first index of the first portion of secondary data;

storing the first index in association with a first index server;

indexing a second portion of the secondary copy stored on the one or more secondary storage devices, wherein indexing the second portion of secondary copy creates a second index of the second portion of secondary data;

storing the second index in association with a second index server;

wherein the first and second index servers are networked together and collectively provide a distributed index comprising the first and second indexes;

copying the first index associated with the first index server to the second index server so that the first index is available at the first and second index servers;

identifying that a plurality of requests for secondary data are associated with the first index;

distributing the plurality of requests for the secondary data associated with the first index among the first and second index servers to lessen a load on the first index server;

performing a storage operation that copies new primary data stored in the one or more primary storage devices to the one or more secondary storage devices;

generating an index update associated with copying the new primary data stored in the one or more primary storage devices to the one or more secondary storage devices;

determining that the first index server is not available; and

updating the distributed index by copying the index update to the second index server.

2. The method of claim 1, further comprising:

determining a fault during updating of the distributed index;

determining a location within the distributed index when the fault occurred; and

resuming the updating of the distributed index at the determined location, without having to update the entire distributed index.

3. The method of claim 1 further comprising copying the second index associated with the second index server to the first index server so that the second index is available at the first and second index servers.

4. The method of claim 1, further comprising:

logging new index data on a network share and associated with a network address; and

providing to the first and second index servers the network address.

5. The method of claim 1, further comprising:

receiving an indication from the first index server that the first index server has received indexed data and updated the distributed index; and

instructing a media agent to delete the indexed data from a data store associated with the media agent.

6. The method of claim 1, further comprising:

receiving a request for a document or for information about the document;

determining a department or office within a business organization associated with the document; and

forwarding the received request to one of first and second index servers associated the determined department or office.

7. The method of claim 1, further comprising when the first index server is unavailable, holding the indexed data at a media agent.

8. The method of claim 1 further comprising copying the first and second indexes to a third index server.

9. The method of claim 8 further comprising distributing requests associated with the first index among the second and third index servers when the first index server is unavailable.

10. A distributed index system for maintaining an index of data storage information related to secondary copies of data, the comprising:

a storage manager comprising computer hardware, the storage manager configured to copy primary data stored in one or more primary storage devices to one or more secondary storage devices, wherein copying the primary data to the secondary storage device creates a secondary copy on the one or more secondary storage devices, the primary data generated by one or more software applications running on a client computer;

a first index server comprising computer hardware, the first index server configured to index a first portion of the secondary copy stored on the one or more secondary storage devices, wherein indexing the first portion of the secondary copy creates a first index of the first portion of secondary data, and wherein the first index is stored in association with a first index server;

a second index server comprising computer hardware having one or more computer processors, the second index server configured to index a second portion of the secondary copy stored on the one or more secondary storage devices, wherein indexing the second portion of secondary copy creates a second index of the second portion of secondary data, and wherein the second index is stored in association with a second index server;

wherein the first and second index servers are networked together and collectively provide a distributed index comprising the first and second indexes;

an index replication component comprising computer hardware, the index replication component configured to direct the copying the first index associated with the first index server to the second index server so that the first index is available at the first and second index servers;

a third index server comprising computer hardware that is configured to identify that a plurality of requests for the secondary data are associated with the first index, the third index server further configured to distribute the plurality of requests for the secondary data associated with the first index among the first and second index servers to lessen a load on the first index server; and

wherein the storage manager is configured to initiate a storage operation that copies new primary data stored in the one or more primary storage devices to the one or more secondary storage devices;

a secondary storage computing device that is configured to generate an index update associated with copying the new primary data stored in the one or more primary storage devices to the one or more secondary storage devices, and determine that the first index server is not available, wherein the secondary storage computing device is further configured to update the distributed index by copying the index update to the second index server.

11. The system of claim 10, wherein the index replication component is further configured to:

determine a fault during updating of the distributed index;

determine a location within the distributed index when the fault occurred; and

resume the updating of the distributed index at the determined location, without having to update the entire distributed index.

12. The system of claim 10 wherein the index replication component is further configured to copy the second index associated with the second index server to the first index server so that the second index is available at the first and second index servers.

13. The system of claim 10, wherein the secondary storage computing device is further configured to:

log new index data on a network share and associated with a network address; and

provide to the first and second index servers the network address.

14. The system of claim 10, wherein the index replication component is configured to:

receive an indication from the first index server that the first index server has received indexed data and updated the distributed index; and

instruct the secondary storage computing device to delete the indexed data from a data store associated with the media agent.

15. The system of claim 10, wherein the third index server is further configured to:

receive a request for a document or for information about the document;

determine a department or office within a business organization associated with the document; and

forward the received request to one of first and second index servers associated the determined department or office.

16. The system of claim 10, further comprising when the first index server is unavailable, holding the indexed data at the secondary storage computing device.

17. The system of claim 10 wherein the index replication component is further configured to copy the first and second indexes to the third index server.

18. The system of claim 17 wherein the third index server is configured to distribute requests associated with the first index among the second and third index servers when the first index server is unavailable.