MANAGING DATA SEARCHES USING GENERATION IDENTIFIERS

• US 20150347523A1
• Filed: 08/01/2015
• Published: 12/03/2015
• Est. Priority Date: 05/15/2012
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: PATEL, VISHAL, BLANK, MITCHELL NEUMAN Jr., VASAN, SUNDAR RENEGARAJAN, SORKIN, STEPHEN PHILLIP
• Granted Patent: US 9,984,129 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/20 using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/275 Synchronous replication
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Embodiments are directed towards managing within a cluster environment having a plurality of indexers for data storage using redundancy the data being managed using a generation identifier, such that a primary indexer is designated for a given generation of data. When a master device for the cluster fails, data may continue to be stored using redundancy, and data searches performed may still be performed.

Claims

1. A method, comprising:

identifying, at a master device, a set of data, wherein data in the set of data have time stamps that fall within a particular time frame;

creating, by the master device, a first generation identifier associated with the set of data;

sending, by the master device, an indication to a first indexer among a set of indexers that the first generation identifier indicates to the first indexer that the first indexer is to serve as a primary indexer for responding to queries pertaining to the set of data when the first indexer receives a query pertaining to the set of data that includes the first generation identifier, wherein other indexers among the set of indexers have not received instructions to respond to queries pertaining to the set of data that include the first generation identifier;

indicating, by the master device, to a search head that the search head is to include the first generation identifier in a query when it sends a query to indexers pertaining to the set of data.

2. The method of claim 1, further comprising:

creating, by the master device, a second generation identifier associated with the set of data, the second generation identifier different from the first generation identifier;

sending, by the master device, an indication to a second indexer among the set of indexers that the second generation identifier indicates to the second indexer that the second indexer is to serve as a primary indexer for responding to queries pertaining to the set of data when the second indexer receives a query pertaining to the set of data that includes the second generation identifier, wherein other indexers among the set of indexers have not received instructions to respond to queries pertaining to the set of data that include the second generation identifier;

indicating, by the master device, to a second search head that the second search head is to include the second generation identifier in a query when it sends a query to indexers pertaining to the set of data.

3. The method of claim 1, further comprising:

creating, by the master device, a second generation identifier associated with the set of data, the second generation identifier different from the first generation identifier;

sending, by the master device, an indication to a second indexer among the set of indexers that the second generation identifier indicates to the second indexer that the second indexer is to serve as a primary indexer for responding to queries pertaining to the set of data when the second indexer receives a query pertaining to the set of data that includes the second generation identifier, wherein other indexers among the set of indexers have not received instructions to respond to queries pertaining to the set of data that include the second generation identifier;

wherein the first generation identifier and the second generation identifier indicate that the first indexer and the second indexer are primary indexers for the set of data in different geographical areas.

4. The method of claim 1, further comprising:

creating, by the master device, a second generation identifier associated with the set of data, the second generation identifier different from the first generation identifier;

sending, by the master device, an indication to a second indexer among the set of indexers that the second generation identifier indicates to the second indexer that the second indexer is to serve as a primary indexer for responding to queries pertaining to the set of data when the second indexer receives a query pertaining to the set of data that includes the second generation identifier, wherein other indexers among the set of indexers have not received instructions to respond to queries pertaining to the set of data that include the second generation identifier;

wherein the first generation identifier and the second generation identifier indicate that the first indexer and the second indexer are primary indexers for the set of data in different geographical areas;

indicating, by the master device, to a second search head that the second search head is to include the second generation identifier in a query when it sends a query to indexers pertaining to the set of data.

5. The method of claim 1, further comprising:

creating, by the master device, a second generation identifier associated with the set of data, the second generation identifier different from the first generation identifier;

sending, by the master device, an indication to a second indexer among the set of indexers that the second generation identifier indicates to the second indexer that the second indexer is to serve as a primary indexer for responding to queries pertaining to the set of data when the second indexer receives a query pertaining to the set of data that includes the second generation identifier, wherein other indexers among the set of indexers have not received instructions to respond to queries pertaining to the set of data that include the second generation identifier;

wherein the first generation identifier and the second generation identifier indicate that the first indexer and the second indexer are primary indexers for different types of searches on the set of data.

6. The method of claim 1, wherein a generation identifier indicates to an indexer that the indexer is the primary indexer for all sets of data that the indexer has access to.

7. The method of claim 1, wherein an indexer is the primary indexer for a plurality of sets of data that the indexer has access to and the indexer has a different generation identifier associated with each set of data in the plurality of sets of data.

8. An apparatus, comprising:

a data set identifier, at a master device, implemented at least partially in hardware, that identifies a set of data, wherein data in the set of data have time stamps that fall within a particular time frame;

a generation identifier creator, at the master device, implemented at least partially in hardware, that creates a first generation identifier associated with the set of data;

an indexer communicator, at the master device, implemented at least partially in hardware, that sends an indication to a first indexer among a set of indexers that the first generation identifier indicates to the first indexer that the first indexer is to serve as a primary indexer for responding to queries pertaining to the set of data when the first indexer receives a query pertaining to the set of data that includes the first generation identifier, wherein other indexers among the set of indexers have not received instructions to respond to queries pertaining to the set of data that include the first generation identifier;

a search head communicator, at the master device, implemented at least partially in hardware, that indicates to a search head that the search head is to include the first generation identifier in a query when it sends a query to indexers pertaining to the set of data.

9. The apparatus of claim 8,

wherein the generation identifier creator creates a second generation identifier associated with the set of data, the second generation identifier different from the first generation identifier,

wherein the indexer communicator sends an indication to a second indexer among the set of indexers that the second generation identifier indicates to the second indexer that the second indexer is to serve as a primary indexer for responding to queries pertaining to the set of data when the second indexer receives a query pertaining to the set of data that includes the second generation identifier, wherein other indexers among the set of indexers have not received instructions to respond to queries pertaining to the set of data that include the second generation identifier, and

wherein the search head communicator indicates to a second search head that the second search head is to include the second generation identifier in a query when it sends a query to indexers pertaining to the set of data.

10. The apparatus of claim 8,

wherein the generation identifier creator creates a second generation identifier associated with the set of data, the second generation identifier different from the first generation identifier,

wherein the indexer communicator sends an indication to a second indexer among the set of indexers that the second generation identifier indicates to the second indexer that the second indexer is to serve as a primary indexer for responding to queries pertaining to the set of data when the second indexer receives a query pertaining to the set of data that includes the second generation identifier, wherein other indexers among the set of indexers have not received instructions to respond to queries pertaining to the set of data that include the second generation identifier, and

wherein the first generation identifier and the second generation identifier indicate that the first indexer and the second indexer are primary indexers for the set of data in different geographical areas.

11. The apparatus of claim 8,

wherein the generation identifier creator creates a second generation identifier associated with the set of data, the second generation identifier different from the first generation identifier,

wherein the indexer communicator sends an indication to a second indexer among the set of indexers that the second generation identifier indicates to the second indexer that the second indexer is to serve as a primary indexer for responding to queries pertaining to the set of data when the second indexer receives a query pertaining to the set of data that includes the second generation identifier, wherein other indexers among the set of indexers have not received instructions to respond to queries pertaining to the set of data that include the second generation identifier,

wherein the first generation identifier and the second generation identifier indicate that the first indexer and the second indexer are primary indexers for the set of data in different geographical areas, and

wherein the search head communicator indicates to a second search head that the second search head is to include the second generation identifier in a query when it sends a query to indexers pertaining to the set of data.

12. The apparatus of claim 8,

wherein the generation identifier creator creates a second generation identifier associated with the set of data, the second generation identifier different from the first generation identifier,

wherein the indexer communicator sends an indication to a second indexer among the set of indexers that the second generation identifier indicates to the second indexer that the second indexer is to serve as a primary indexer for responding to queries pertaining to the set of data when the second indexer receives a query pertaining to the set of data that includes the second generation identifier, wherein other indexers among the set of indexers have not received instructions to respond to queries pertaining to the set of data that include the second generation identifier, and

wherein the first generation identifier and the second generation identifier indicate that the first indexer and the second indexer are primary indexers for different types of searches on the set of data.

13. The apparatus of claim 8, wherein a generation identifier indicates to an indexer that the indexer is the primary indexer for all sets of data that the indexer has access to.

14. The apparatus of claim 8, wherein an indexer is the primary indexer for a plurality of sets of data that the indexer has access to and the indexer has a different generation identifier associated with each set of data in the plurality of sets of data.

15. One or more non-transitory computer-readable storage media, storing software instructions, which when executed by one or more processors cause performance of:

identifying, at a master device, a set of data, wherein data in the set of data have time stamps that fall within a particular time frame;

creating, by the master device, a first generation identifier associated with the set of data;

sending, by the master device, an indication to a first indexer among a set of indexers that the first generation identifier indicates to the first indexer that the first indexer is to serve as a primary indexer for responding to queries pertaining to the set of data when the first indexer receives a query pertaining to the set of data that includes the first generation identifier, wherein other indexers among the set of indexers have not received instructions to respond to queries pertaining to the set of data that include the first generation identifier;

indicating, by the master device, to a search head that the search head is to include the first generation identifier in a query when it sends a query to indexers pertaining to the set of data.

16. The one or more non-transitory computer-readable storage media of claim 15, wherein the instructions, when executed by the one or more computing devices, further cause performance of:

creating, by the master device, a second generation identifier associated with the set of data, the second generation identifier different from the first generation identifier;

sending, by the master device, an indication to a second indexer among the set of indexers that the second generation identifier indicates to the second indexer that the second indexer is to serve as a primary indexer for responding to queries pertaining to the set of data when the second indexer receives a query pertaining to the set of data that includes the second generation identifier, wherein other indexers among the set of indexers have not received instructions to respond to queries pertaining to the set of data that include the second generation identifier;

indicating, by the master device, to a second search head that the second search head is to include the second generation identifier in a query when it sends a query to indexers pertaining to the set of data.

17. The one or more non-transitory computer-readable storage media of claim 15, wherein the instructions, when executed by the one or more computing devices, further cause performance of:

creating, by the master device, a second generation identifier associated with the set of data, the second generation identifier different from the first generation identifier;

sending, by the master device, an indication to a second indexer among the set of indexers that the second generation identifier indicates to the second indexer that the second indexer is to serve as a primary indexer for responding to queries pertaining to the set of data when the second indexer receives a query pertaining to the set of data that includes the second generation identifier, wherein other indexers among the set of indexers have not received instructions to respond to queries pertaining to the set of data that include the second generation identifier;

wherein the first generation identifier and the second generation identifier indicate that the first indexer and the second indexer are primary indexers for the set of data in different geographical areas.

18. The one or more non-transitory computer-readable storage media of claim 15, wherein the instructions, when executed by the one or more computing devices, further cause performance of:

creating, by the master device, a second generation identifier associated with the set of data, the second generation identifier different from the first generation identifier;

sending, by the master device, an indication to a second indexer among the set of indexers that the second generation identifier indicates to the second indexer that the second indexer is to serve as a primary indexer for responding to queries pertaining to the set of data when the second indexer receives a query pertaining to the set of data that includes the second generation identifier, wherein other indexers among the set of indexers have not received instructions to respond to queries pertaining to the set of data that include the second generation identifier;

wherein the first generation identifier and the second generation identifier indicate that the first indexer and the second indexer are primary indexers for the set of data in different geographical areas;

indicating, by the master device, to a second search head that the second search head is to include the second generation identifier in a query when it sends a query to indexers pertaining to the set of data.

19. The one or more non-transitory computer-readable storage media of claim 15, wherein the instructions, when executed by the one or more computing devices, further cause performance of:

creating, by the master device, a second generation identifier associated with the set of data, the second generation identifier different from the first generation identifier;

sending, by the master device, an indication to a second indexer among the set of indexers that the second generation identifier indicates to the second indexer that the second indexer is to serve as a primary indexer for responding to queries pertaining to the set of data when the second indexer receives a query pertaining to the set of data that includes the second generation identifier, wherein other indexers among the set of indexers have not received instructions to respond to queries pertaining to the set of data that include the second generation identifier;

wherein the first generation identifier and the second generation identifier indicate that the first indexer and the second indexer are primary indexers for different types of searches on the set of data.

20. The one or more non-transitory computer-readable storage media of claim 15, wherein a generation identifier indicates to an indexer that the indexer is the primary indexer for all sets of data that the indexer has access to.

21. The one or more non-transitory computer-readable storage media of claim 15, wherein an indexer is the primary indexer for a plurality of sets of data that the indexer has access to and the indexer has a different generation identifier associated with each set of data in the plurality of sets of data.