Hybrid cluster-based data intake and query

• US 9,990,423 B2
• Filed: 10/28/2014
• Issued: 06/05/2018
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Ago, Ledio, Shanaghy, Declan Gerard
• Primary Examiner(s): Mackes, Kris E
• Assistant Examiner(s): Bui, Thuy T
• Application Number:
  US14/526,493
• Publication Number:
  US 20160092558A1
• Time in Patent Office: 1,316 Days
• Field of Search: 707737
• CPC Class Codes: G06F 16/328 Management therefor
• CPC Class Codes: G06F 16/35 Clustering; Classification

Abstract

Various embodiments describe multi-site cluster-based data intake and query systems, including cloud-based data intake and query systems. Using a hybrid search system that includes cloud-based data intake and query systems working in concert with so-called “on-premises” data intake and query systems can promote the scalability of search functionality. In addition, the hybrid search system can enable data isolation in a manner in which sensitive data is maintained “on premises” and information or data that is not sensitive can be moved to the cloud-based system. Further, the cloud-based system can enable efficient leveraging of data that may already exist in the cloud.

Claims

1. A computer-implemented method comprising:

receiving, at a first cluster, a search query, the first cluster being a first data intake and query system;

transmitting, through a firewall of either the first cluster or a cloud-based cluster, a request for information identifying a plurality of indexers of the cloud-based cluster, the cloud-based cluster being a second data intake and query system;

in response to the request, obtaining, from the cloud-based cluster, the information identifying the plurality of indexers, wherein the first cluster and the cloud-based cluster each include at least one master node that is knowledgeable about active indexers within its respective cluster, and the information identifies the plurality of indexers based on the at least one master node of the cloud-based cluster identifying the active indexers;

distributing the search query to the plurality of indexers of the cloud-based cluster and one or more indexers of the first cluster, said distributing using the obtained information identifying the plurality of indexers and being through the firewall; and

receiving, at the first cluster, a response to the distributed search query from at least one of the plurality of indexers of the cloud-based cluster wherein each response from a respective indexer is produced by the respective indexer based on an evaluation, by the respective indexer, of the distributed search query.

2. The method as described in claim 1, wherein the firewall is of the cloud-based cluster.

3. The method as described in claim 1, wherein the firewall is of the first cluster.

4. The method as described in claim 1, wherein the evaluation is on events associated with timestamps, the events comprising raw portions of machine data.

5. The method as described in claim 1, wherein the search query is configured to be used in connection with a late-binding schema.

6. The method as described in claim 1, wherein the search query is configured to be used to search log data.

7. The method as described in claim 1, wherein the search query is configured to be used to wire data.

8. The method as described in claim 1, wherein said first cluster is an on-premises cluster.

9. The method as described in claim 1, wherein said first cluster and said cloud-based cluster each include a single master node that is knowledgeable about active indexers within its respective cluster, and the information identifies the plurality of indexers based on the single master node of the cloud-based cluster identifying the active indexers.

10. The method as described in claim 1, wherein the response includes information on how to communicate with the plurality of indexers that is used by the first cluster in the distributing of the search query.

11. The method as described in claim 1, wherein the firewall is of the cloud-based cluster, and the firewall is configured to allow inbound communication based on an IP address of a search head of the first cluster that requests said information.

12. The method as described in claim 1, wherein the firewall is of the cloud-based cluster, and the firewall is configured to allow an inbound communication comprising the request based on an IP address associated with the first cluster.

13. The method as described in claim 1, wherein the information includes respective IP addresses of the plurality of indexers, and the distributing is to the respective IP addresses from the information.

14. The method as described in claim 1, wherein the information includes a generation identifier to be used in the distributing of the search query, the generation identifier identifying an indexer as a primary indexer to perform the evaluation on data and return corresponding search results when multiple indexers respectively manage a corresponding copy of the data.

15. The method as described in claim 1, wherein said obtaining is performed by obtaining said information from a master node of the cloud-based cluster, the master node being configured to return a list of active indexers and a generation identifier that is to be used in distributing the search query, generation identifiers identifying primary and secondary indexers of the cloud-based cluster.

16. The method as described in claim 1, wherein the first cluster maintains information required to be maintained with a higher level of security than data in the cloud-based cluster.

17. A non-transitory computer readable storage medium, storing software instructions, which when executed by one or more processors, perform operations comprising:

receiving, at a first cluster, a search query, the first cluster being a first data intake and query system;

transmitting, through a firewall of either the first cluster or a cloud-based cluster, a request for information identifying a plurality of indexers of the cloud-based cluster, the cloud-based cluster being a second data intake and query system;

in response to the request, obtaining, from the cloud-based cluster, the information identifying the plurality of indexers, wherein the first cluster and the cloud-based cluster each include at least one master node that is knowledgeable about active indexers within its respective cluster, and the information identifies the plurality of indexers based on the at least one master node of the cloud-based cluster identifying the active indexers;

distributing the search query to the plurality of indexers of the cloud-based cluster and one or more indexers of the first cluster, said distributing using the obtained information identifying the plurality of indexers and being through the firewall; and

receiving, at the first cluster, a response to the distributed search query from at least one of the plurality of indexers of the cloud-based cluster wherein each response from a respective indexer is produced by the respective indexer based on an evaluation, by the respective indexer, of the distributed search query.

18. The non-transitory computer readable storage medium as described in claim 17, wherein said contacting occurs through the firewall is of the cloud-based cluster.

19. The non-transitory computer readable storage medium as described in claim 17, wherein said contacting occurs through the firewall is of the first cluster.

20. The non-transitory computer readable storage medium as described in claim 17, wherein the evaluation is on events associated with timestamps, the events comprising raw portions of machine data.

21. The non-transitory computer readable storage medium as described in claim 17, wherein the search query is configured to be used in connection with a late-binding schema.

22. The non-transitory computer readable storage medium as described in claim 17, wherein the search query is configured to be used to search log data.

23. The non-transitory computer readable storage medium as described in claim 17, wherein the search query is configured to be used to wire data.

24. The non-transitory computer readable storage medium as described in claim 17, wherein said first cluster is an on-premises cluster.

25. The non-transitory computer readable storage medium as described in claim 17, wherein said first cluster and said cloud-based cluster each include a single master node that is knowledgeable about active indexers within its respective cluster, and the information identifies the plurality of indexers based on the single master node of the cloud-based cluster identifying the active indexers.

26. The non-transitory computer readable storage medium as described in claim 17, wherein the information includes respective IP addresses of the plurality of indexers.

27. The non-transitory computer readable storage medium as described in claim 17, wherein the information includes a generation identifier to be used in the distributing of the search query, the generation identifier identifying an indexer as a primary indexer to perform the evaluation on data and return corresponding search results when multiple indexers respectively manage a corresponding copy of the data.

28. The non-transitory computer readable storage medium as described in claim 17, wherein said obtaining is performed by obtaining said information from a master node of the cloud-based cluster, the master node being configured to return a list of active indexers and a generation identifier that is to be used in distributing the search query, generation identifiers identifying primary and secondary indexers of the cloud-based cluster.

29. The non-transitory computer readable storage medium as described in claim 17, wherein the first cluster is configured to maintain sensitive information, and the cloud-based cluster is configured to maintain information that is not sensitive information.

30. A computer-implemented system, comprising:

one or more processors;

one or more non-transitory computer readable storage media;

computer readable instructions stored on the one or more non-transitory computer readable storage media which, when executed by the one or more processors, implement a first cluster configured to perform operations comprising;

receiving, at a first cluster, a search query, the first cluster being a first data intake and query system;

transmitting, through a firewall of either the first cluster or a cloud-based cluster, a request for information identifying a plurality of indexers of the cloud-based cluster, the cloud-based cluster being a second data intake and query system;

in response to the request, obtaining, from the cloud-based cluster, the information identifying the plurality of indexers, wherein the first cluster and the cloud-based cluster each include at least one master node that is knowledgeable about active indexers within its respective cluster, and the information identifies the plurality of indexers based on the at least one master node of the cloud-based cluster identifying the active indexers;

distributing the search query to the plurality of indexers of the cloud-based cluster and one or more indexers of the first cluster, said distributing using the obtained information identifying the plurality of indexers and being through the firewall; and

receiving, at the first cluster, a response to the distributed search query from at least one of the plurality of indexers of the cloud-based cluster wherein each response from a respective indexer is produced by the respective indexer based on an evaluation, by the respective indexer, of the distributed search query.

31. The system as described in claim 30, wherein the firewall is of the cloud-based cluster.

32. The system as described in claim 30, wherein the firewall is of the first cluster.

33. The system as described in claim 30, wherein each indexer manages a respective dataset, and said evaluation is on the respective dataset to produce results that satisfy criteria of the distributed search query, the results included in the response.

34. The system as described in claim 30, wherein the search query is configured to search events associated with timestamps, the events comprising raw portions of machine data.

35. The system as described in claim 30, wherein the search query is configured to be used in connection with a late-binding schema.

36. The system as described in claim 30, wherein the search query is configured to be used to search log data.

37. The system as described in claim 30, wherein the search query is configured to be used to wire data.

38. The system as described in claim 30, wherein said first cluster is an on-premises cluster.

39. The system as described in claim 30, wherein said first cluster and said cloud-based cluster each include a single master node that is knowledgeable about active indexers within its respective cluster, and the information identifies the plurality of indexers based on the single master node of the cloud-based cluster identifying the active indexers.

40. The system as described in claim 30, wherein the system further comprises:

one or more additional processors;

one or more additional computer readable storage media;

computer readable instructions stored on the one or more additional computer readable storage media which, when executed by the one or more processors, implement the cloud-based cluster.

41. The system as described in claim 40, wherein the cloud-based cluster is configured to perform operations comprising:

receiving, at the cloud-based cluster, the request;

preparing a response including information on how to communicate with active indexers of the cloud-based cluster; and

sending the response to the first cluster.

42. The system as described in claim 40, wherein the firewall is of the cloud-based cluster, and the firewall is configured to allow inbound communication based on an IP address of a search head that requests said information.

43. The system as described in claim 40, wherein the firewall is of the cloud-based cluster, and the firewall is configured to allow inbound communication based on an IP address associated with a cluster that performed said transmitting.

44. The system as described in claim 30, wherein the information includes respective IP addresses of the plurality of indexers.

45. The system as described in claim 30, wherein the information includes a generation identifier to be used in distributing the search query.

46. The system as described in claim 30, wherein said obtaining is performed by obtaining said information from a master node of the cloud-based cluster, the master node being configured to return a list of active indexers and a generation identifier that is to be used in distributing the search query, generation identifiers identifying primary and secondary indexers of the cloud-based cluster.

47. The system as described in claim 30, wherein the first cluster is configured to maintain sensitive information, and the cloud-based cluster is configured to maintain information that is not sensitive information.