Site-based search affinity

• US 9,130,971 B2
• Filed: 04/30/2014
• Issued: 09/08/2015
• Est. Priority Date: 05/15/2012
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Vasan, Sundar Rengarajan, Blank, Mitchell Neuman Jr., Patel, Vishal, Xu, Da, Gopalan, Rama
• Primary Examiner(s): PHILLIPS, III, ALBERT M
• Application Number:
  US14/266,812
• Publication Number:
  US 20140236889A1
• Time in Patent Office: 496 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/20 using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
• CPC Class Codes: G06F 11/2094 Redundant storage or storage space G06F11/2056 takes precedence
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/275 Synchronous replication
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 3/0617 in relation to availability
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

According to various embodiments, techniques are described for managing data within a multi-site clustered data intake and query system. A data intake and query system as described herein generally refers to a system for collecting, retrieving, and analyzing data. In this context, a clustered data intake and query system generally refers to a system environment that is configured to provide data redundancy and other features that improve the availability of data stored by the system. For example, a clustered data intake and query system may be configured to store multiple copies of data stored by the system across multiple components such that recovery from a failure of one or more of the components is possible by using copies of the data stored elsewhere in the cluster.

Claims

1. A method comprising:

receiving, at an indexer, a set of search affinity identifiers;

wherein each search affinity identifier of the set of search affinity identifiers indicates, for a particular subset of data accessible to the indexer and for a particular site of a plurality of sites from which a query may originate, whether the indexer has primary responsibility for responding to queries originating from the particular site based on data from the particular subset of data;

wherein each site of the plurality of sites represents a user-specified grouping of one or more computing resources corresponding to a particular geographic location;

receiving, from a first search head located at a first site, (i) a first query to search a subset of data accessible to the indexer, and (ii) a first site identifier identifying the first site at which the first search head is located;

determining, based on both the first site identifier and a particular search affinity identifier of the set of search affinity identifiers, that the indexer is to respond to the first query with a result from searching the subset of data;

in response to determining that the indexer is to respond to the first query, sending, to the first search head, the result from searching the subset of data.

2. The method of claim 1, wherein determining that the indexer is to respond to the first query with a result from searching the subset of data includes determining that the indexer has primary responsibility for responding to queries for the subset of data for the first site.

3. The method of claim 1, further comprising:

receiving, at the indexer, a second query from a second search head to search the subset of data, the second query including a second site identifier identifying a second site at which the second search head is located;

determining, based on both the second site identifier and the particular search affinity identifier of the set of search affinity identifiers, that the indexer is not to respond to the second query with a result from searching the subset of data.

4. The method of claim 1, wherein the particular search affinity identifier is a bitmask, and wherein each digit of the bitmask represents a particular site of the plurality of sites from which a query may originate.

5. The method of claim 1, further comprising:

receiving, at the indexer, a second query from a second search head to search the subset of data, the second query including a second site identifier identifying a second site at which the second search head is located;

determining, based on both the second site identifier and the particular search affinity identifier of the set of search affinity identifiers, that the indexer is not to respond to the second query with a result from searching the subset of data;

wherein the first query and the second query are identical.

6. The method of claim 1, further comprising:

receiving, at the indexer, a second query from a second search head to search the subset of data, the second query including a second site identifier identifying a second site at which the second search head is located;

determining, based on both the second site identifier and the particular search affinity identifier of the set of search affinity identifiers, that the indexer is not to respond to the second query with a result from searching the subset of data;

wherein the first query and the second query are different.

7. The method of claim 1, further comprising:

receiving, at the indexer, raw data;

separating the raw data into a plurality of events included in the subset of data;

determining, for each event in the plurality of events, a time stamp; and

storing the subset of data in a data store.

8. The method of claim 1, further comprising:

receiving, at the indexer, raw data;

separating the raw data into a plurality of events included in the subset of data;

storing the subset of data in a data store;

identifying a replication factor that indicates a number of times that the subset of data is to be replicated; and

sending the subset of data to a number of other indexers, wherein the number corresponds to the replication factor.

9. The method of claim 1, further comprising:

receiving, at the indexer, raw data;

separating the raw data into a plurality of events included in the subset of data;

storing the subset of data in a data store;

identifying a site replication factor that indicates a number of sites at which the subset of data is to be replicated; and

sending the subset of data to second indexers located at the number of sites.

10. The method of claim 1, further comprising:

wherein the set of search affinity identifiers is associated with a first generation identifier;

receiving, at the indexer, a second set of search affinity identifiers associated with a second generation identifier.

11. The method of claim 1, further comprising:

wherein the indexer stores a plurality of sets of search affinity identifiers, and wherein each set of search affinity identifiers of the plurality of sets of search affinity identifiers is associated with a generation identifier;

receiving, from the first search head, a particular generation identifier identifying a particular set of search affinity identifiers of the plurality of sets of search affinity identifiers.

12. One or more non-transitory computer-readable storage media, storing software instructions, which when executed by one or more processors cause performance of steps of:

receiving, at an indexer, a set of search affinity identifiers;

wherein each search affinity identifier of the set of search affinity identifiers indicates, for a particular subset of data accessible to the indexer and for a particular site of a plurality of sites from which a query may originate, whether the indexer has primary responsibility for responding to queries originating from the particular site based on data from the particular subset of data;

wherein each site of the plurality of sites represents a user-specified grouping of one or more computing resources corresponding to a particular geographic location;

receiving, from a first search head located at a first site, (i) a first query to search a subset of data accessible to the indexer, and (ii) a first site identifier identifying the first site at which the first search head is located;

determining, based on both the first site identifier and a particular search affinity identifier of the set of search affinity identifiers, that the indexer is to respond to the first query with a result from searching the subset of data;

in response to determining that the indexer is to respond to the first query, sending, to the first search head, the result from searching the subset of data.

13. The one or more non-transitory computer-readable storage media of claim 12, wherein determining that the indexer is to respond to the first query with a result from searching the subset of data includes determining that the indexer has primary responsibility for responding to queries for the subset of data for the first site.

14. The one or more non-transitory computer-readable storage media of claim 12, wherein the instructions, when executed by the one or more computing devices, further cause performance of:

receiving, at the indexer, a second query from a second search head to search the subset of data, the second query including a second site identifier identifying a second site at which the second search head is located;

determining, based on both the second site identifier and the particular search affinity identifier of the set of search affinity identifiers, that the indexer is not to respond to the second query with a result from searching the subset of data.

15. The one or more non-transitory computer-readable storage media of claim 12, wherein the particular search affinity identifier is a bitmask, and wherein each digit of the bitmask represents a particular site of the plurality of sites from which a query may originate.

16. The one or more non-transitory computer-readable storage media of claim 12, wherein the instructions, when executed by the one or more computing devices, further cause performance of:

receiving, at the indexer, a second query from a second search head to search the subset of data, the second query including a second site identifier identifying a second site at which the second search head is located;

determining, based on both the second site identifier and the particular search affinity identifier of the set of search affinity identifiers, that the indexer is not to respond to the second query with a result from searching the subset of data;

wherein the first query and the second query are identical.

17. The one or more non-transitory computer-readable storage media of claim 12, wherein the instructions, when executed by the one or more computing devices, further cause performance of:

receiving, at the indexer, a second query from a second search head to search the subset of data, the second query including a second site identifier identifying a second site at which the second search head is located;

determining, based on both the second site identifier and the particular search affinity identifier of the set of search affinity identifiers, that the indexer is not to respond to the second query with a result from searching the subset of data;

wherein the first query and the second query are different.

18. The one or more non-transitory computer-readable storage media of claim 12, wherein the instructions, when executed by the one or more computing devices, further cause performance of:

receiving, at the indexer, raw data;

separating the raw data into a plurality of events included in the subset of data;

determining, for each event in the plurality of events, a time stamp; and

storing the subset of data in a data store.

19. The one or more non-transitory computer-readable storage media of claim 12, wherein the instructions, when executed by the one or more computing devices, further cause performance of:

receiving, at the indexer, raw data;

separating the raw data into a plurality of events included in the subset of data;

storing the subset of data in a data store;

identifying a replication factor that indicates a number of times that the subset of data is to be replicated; and

sending the subset of data to a number of other indexers, wherein the number corresponds to the replication factor.

20. The one or more non-transitory computer-readable storage media of claim 12, wherein the instructions, when executed by the one or more computing devices, further cause performance of:

receiving, at the indexer, raw data;

separating the raw data into a plurality of events included in the subset of data;

storing the subset of data in a data store;

identifying a site replication factor that indicates a number of sites at which the subset of data is to be replicated; and

sending the subset of data to second indexers located at the number of sites.

21. The one or more non-transitory computer-readable storage media of claim 12, wherein the instructions, when executed by the one or more computing devices, further cause performance of:

wherein the set of search affinity identifiers is associated with a first generation identifier;

receiving, at the indexer, a second set of search affinity identifiers associated with a second generation identifier.

22. The one or more non-transitory computer-readable storage media of claim 12, wherein the instructions, when executed by the one or more computing devices, further cause performance of:

wherein the indexer stores a plurality of sets of search affinity identifiers, and wherein each set of search affinity identifiers of the plurality of sets of search affinity identifiers is associated with a generation identifier;

receiving, from the first search head, a particular generation identifier identifying a particular set of search affinity identifiers of the plurality of sets of search affinity identifiers.