Distributed indexing system for data storage

• US 9,646,038 B2
• Filed: 05/18/2016
• Issued: 05/09/2017
• Est. Priority Date: 07/02/2008
• Status: Active Grant

• Current Assignee: CommVault Systems Incorporated
• Original Assignee: CommVault Systems Incorporated
• Inventors: Gokhale, Parag
• Primary Examiner(s): BOWEN, RICHARD L
• Application Number:
  US15/158,349
• Publication Number:
  US 20160266829A1
• Time in Patent Office: 356 Days
• Field of Search: None
• CPC Class Codes: G06F 11/1435 using file system or storage system metadata
• CPC Class Codes: G06F 11/1451 by selection of backup contents
• CPC Class Codes: G06F 11/2069 Management of state, configuration or failover
• CPC Class Codes: G06F 16/20 of structured data, e.g. relational data
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/275 Synchronous replication
• CPC Class Codes: G06F 2201/80 Database-specific techniques
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0643 Management of files
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Abstract

A distributed indexing system spreads out the load on an index of stored data in a data storage system. Rather than maintain a single index, the distributed indexing system maintains an index in each media agent of a federated data storage system and a master index that points to the index in each media agent. In some embodiments, the distributed indexing system includes an index server (or group of servers) that handles indexing requests and forwards the requests to the appropriate distributed systems. Thus, the distributed indexing system, among other things, increases the availability and fault tolerance of a data storage index.

Claims

1. A method of creating a distributed index of a secondary copy of primary data, the method comprising:

copying primary data stored in one or more primary storage devices to one or more secondary storage devices to create a secondary copy of the primary data;

indexing a first portion of the secondary copy with a first index server, wherein indexing the first portion of the secondary copy creates a first index of the first portion;

indexing a second portion of the secondary copy with a second index server, wherein indexing the second portion of secondary copy creates a second index of the second portion;

copying the first index to the second index server so that the first index is available at both the first and second index servers;

receiving a request for the primary data;

determining that the secondary copy of the requested primary data exists in the one or more secondary storage devices; and

sending the request to at least one of the first and second index servers to retrieve a portion of the secondary copy that relates to the requested primary data;

identifying that the request is associated with the first index;

directing the request associated with the first index to the second index server when the first index server is not available.

2. The method of claim 1 further comprising copying the second index to the first index server so that the second index is available at both the first and second index servers.

3. The method of claim 1 further comprising:

sending the request to both the first and second index servers to retrieve, based on the first and second indexes, a portion of the secondary copy that relates to the requested primary data.

4. The method of claim 1 further comprising:

logging new index data on a network share and associated with a network address; and

providing to the first and second index servers the network address.

5. The method of claim 1 further comprising:

receiving a request for a document or for information about the document;

determining a department or office within a business organization associated with the document; and

forwarding the received request to one of first and second index servers associated the determined department or office.

6. The method of claim 1 further comprising copying the first and second indexes to a third index server.

7. The method of claim 1 further comprising combining the data in the first and second indexes to create a third index.

8. The method of claim 7 further comprising distributing requests associated with the first index among at least one of the second and third index servers when the first index server is unavailable.

9. The method of claim 1 further comprising an index failover component that updates a list of index servers when a failover event occurs.

10. A system that creates a distributed index of a secondary copy of primary data, the system comprising:

a storage manager comprising computer hardware, the storage manager copies primary data stored in one or more primary storage devices to one or more secondary storage devices to create a secondary copy of the primary data;

a first index server comprising computer hardware having one or more computer processors, the first index server creates a first index by indexing a first portion of the secondary copy;

a second index server comprising computer hardware having one or more computer processors, the second index server creates a second index by indexing index a second portion of the secondary copy;

an index replication component comprising computer hardware, the index replication component directs the copying the first index associated with the first index server to the second index server so that the first index is available at both the first and second index servers;

wherein the storage manager receives a request for the primary data and determines that a secondary copy of the requested primary data exists in the one or more secondary storage devices, wherein the storage manager sends the request to at least one of the first and second index servers to retrieve a portion of the secondary copy that relates to the requested primary data; and

wherein the storage manager identifies that the request is associated with the first index and directs the request associated with the first index to the second server when the first server is not available.

11. The system of claim 10 wherein the index replication component is further configured to copy the second index associated with the second index server to the first index server so that the second index is available at both the first and second index servers.

12. The system of claim 10 wherein the storage manager the request to both the first and second index servers to retrieve, based on the first and second indexes, a portion of the secondary copy that relates to the requested primary data.

13. The system of claim 10 wherein the storage manager:

logs new index data on a network share and associated with a network address; and

provides to the first and second index servers the network address.

14. The system of claim 10 wherein the storage manager:

receives a request for a document or for information about the document;

determines a department or office within a business organization associated with the document; and

forwards the received request to one of first and second index servers associated the determined department or office.

15. The system of claim 10 wherein the index replication component copies the first and second indexes to a third index server.

16. The method of claim 10 wherein the index replication component combines the data in the first and second indexes to create a third index.

17. The system of claim 16 wherein the storage manager is configured to distribute requests associated with the first index among the second and third index servers when the first index server is not available.

18. The system of claim 10 further comprising an index failover component that updates a list of index servers when a failover event occurs.