Sharing Configuration Information for Searches in Data Intake and Query Systems

• US 20160092570A1
• Filed: 10/28/2014
• Published: 03/31/2016
• Est. Priority Date: 09/30/2014
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Ago, Ledio, Shanaghy, Declan Gerard
• Granted Patent: US 10,235,460 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24564 Applying rules; Deductive queries
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: H04L 63/029 Firewall traversal, e.g. tunnelling or, creating pinholes
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04W 4/60 Subscription-based services using application servers or record carriers, e.g. SIM application toolkits

Abstract

Various embodiments describe multi-site cluster-based data intake and query systems, including cloud-based data intake and query systems. Using a hybrid search system that includes cloud-based data intake and query systems working in concert with so-called “on-premises” data intake and query systems can promote the scalability of search functionality. In addition, the hybrid search system can enable data isolation in a manner in which sensitive data is maintained “on premises” and information or data that is not sensitive can be moved to the cloud-based system. Further, the cloud-based system can enable efficient leveraging of data that may already exist in the cloud.

Claims

1. A method, comprising:

accessing, at a first cluster, configuration information associated with searching data accessible by the first cluster; and

communicating the configuration information to a shared storage location, the shared storage location shared between at least the first cluster and a cloud-based cluster for searching different data than the data in the first cluster.

2. The method of claim 1, wherein the configuration information includes a lookup table for enriching data retrieved from a search query.

3. The method of claim 1, wherein the configuration information includes schema for searching data.

4. The method of claim 1, wherein the configuration information includes extraction rules defining fields of a late binding schema for searching data.

5. The method of claim 1, wherein the communication of the configuration information to the shared storage location is done through an outgoing one-way connection from the first cluster.

6. The method of claim 1, further comprising the cloud-based cluster retrieving the configuration information from the shared storage location.

7. The method of claim 1, further comprising the cloud-based cluster retrieving the configuration information from the shared storage location through an outgoing one-way connection.

8. The method of claim 1, further comprising the cloud-based cluster retrieving the configuration information from the shared storage location and using the configuration information in a search of the different data.

9. The method of claim 1, wherein the first cluster is an on-premises cluster.

10. The method of claim 1, wherein the first cluster is a cloud-based cluster.

11. The method of claim 1, wherein the communication is performed through a firewall associated with the first cluster.

12. The method of claim 1, further comprising the cloud-based cluster retrieving the configuration information from the shared storage location through a firewall associated with the cloud-based cluster.

13. The method of claim 1, wherein the data and the different data comprise log data.

14. The method of claim 1, wherein the data and the different data comprise wire data.

15. The method of claim 1, further comprising the first cluster searching the data using a late-binding schema.

16. The method of claim 1, further comprising the second cluster searching the data using a late-binding schema.

17. The method of claim 1, wherein the shared storage is accessible by a third cluster, and further comprising the third cluster accessing the configuration information in the shared storage.

18. A method as described in claim 1, wherein said communicating is performed by a search head at the first cluster.

19. A method as described in claim 1, wherein the configuration information is configured as “

read-only”

.

20. A method as described in claim 1, wherein the configuration information is not configured as “

read-only”

.

21. A method as described in claim 1, wherein the configuration information represents a knowledge object.

22. A method as described in claim 1, wherein the configuration information comprises one or more of:

saved searches, event types, transactions, tags, field extractions, field transforms, lookups, workflow actions, search commands, or views.

23. A non-transitory computer readable storage medium, storing software instructions, which when executed by one or more processors, perform operations comprising:

accessing, at a first cluster, configuration information associated with searching data accessible by the first cluster; and

communicating the configuration information to a shared storage location, the shared storage location shared between at least the first cluster and a cloud-based cluster for searching different data than the data in the first cluster.

24. The non-transitory computer readable storage medium of claim 23, wherein the configuration information includes a lookup table for enriching data retrieved from a search query.

25. The non-transitory computer readable storage medium of claim 23, wherein the configuration information includes schema for searching data.

26. The non-transitory computer readable storage medium of claim 23, wherein the configuration information includes extraction rules defining fields of a late binding schema for searching data.

27. The non-transitory computer readable storage medium of claim 23, wherein the communication of the configuration information to the shared storage location is done through on outgoing one-way connection from the first cluster.

28. The non-transitory computer readable storage medium of claim 23, further comprising the cloud-based cluster retrieving the configuration information from the shared storage location.

29. The non-transitory computer readable storage medium of claim 23, further comprising the cloud-based cluster retrieving the configuration information from the shared storage location through an outgoing one-way connection.

30. The non-transitory computer readable storage medium of claim 23, further comprising the cloud-based cluster retrieving the configuration information from the shared storage location and using the configuration information in a search of the different data.

31. The non-transitory computer readable storage medium of claim 23, wherein the first cluster is an on-premises cluster.

32. The non-transitory computer readable storage medium of claim 23, wherein the first cluster is a cloud-based cluster.

33. The non-transitory computer readable storage medium of claim 23, wherein the communication is performed through a firewall associated with the first cluster.

34. The non-transitory computer readable storage medium of claim 23, further comprising the cloud-based cluster retrieving the configuration information from the shared storage location through a firewall associated with the cloud-based cluster.

35. The non-transitory computer readable storage medium of claim 23, wherein the data and the different data comprise log data.

36. The non-transitory computer readable storage medium of claim 23, wherein the data and the different data comprise wire data.

37. The non-transitory computer readable storage medium of claim 23, further comprising the first cluster searching the data using a late-binding schema.

38. The non-transitory computer readable storage medium of claim 23, further comprising the second cluster searching the data using a late-binding schema.

39. The non-transitory computer readable storage medium of claim 23, wherein the shared storage is accessible by a third cluster, and further comprising the third cluster accessing the configuration information in the shared storage.

40. The non-transitory computer readable storage medium of claim 23, wherein said communicating is performed by a search head at the first cluster.

41. The non-transitory computer readable storage medium of claim 23, wherein said communicating is performed by communicating the configuration information through a firewall associated with the first cluster.

42. The non-transitory computer readable storage medium as described in claim 23, wherein the configuration information is configured as “

read-only”

.

43. The non-transitory computer readable storage medium as described in claim 23, wherein the configuration information represents a knowledge object.

44. The non-transitory computer readable storage medium as described in claim 23, wherein the configuration information can comprise one or more of:

saved searches, event types, transactions, tags, field extractions, field transforms, lookups, workflow actions, search commands, or views.

45. A system comprising:

one or more processors;

one or more computer readable storage media;

computer readable instructions stored on the one or more computer readable storage media which, when executed by the one or more processors, implement a first cluster configured to perform operations comprising;

accessing, at a first cluster, configuration information associated with searching data accessible by the first cluster; and

communicating the configuration information to a shared storage location, the shared storage location shared between at least the first cluster and a cloud-based cluster for searching different data than the data in the first cluster.

46. The system of claim 45, wherein the configuration information includes a lookup table for enriching data retrieved from a search query.

47. The system of claim 45, wherein the configuration information includes schema for searching data.

48. The system of claim 45, wherein the configuration information includes extraction rules defining fields of a late binding schema for searching data.

49. The system of claim 45, wherein the communication of the configuration information to the shared storage location is done through on outgoing one-way connection from the first cluster.

50. The system of claim 45, further comprising the cloud-based cluster retrieving the configuration information from the shared storage location.

51. The system of claim 45, further comprising the cloud-based cluster retrieving the configuration information from the shared storage location through an outgoing one-way connection.

52. The system of claim 45, further comprising the cloud-based cluster retrieving the configuration information from the shared storage location and using the configuration information in a search of the different data.

53. The system of claim 45, wherein the first cluster is an on-premises cluster.

54. The system of claim 45, wherein the first cluster is a cloud-based cluster.

55. The system of claim 45, wherein the communication is performed through a firewall associated with the first cluster.

56. The system of claim 45, further comprising the cloud-based cluster retrieving the configuration information from the shared storage location through a firewall associated with the cloud-based cluster.

57. The system of claim 45, wherein the data and the different data comprise log data.

58. The system of claim 45, wherein the data and the different data comprise wire data.

59. The system of claim 45, further comprising the first cluster searching the data using a late-binding schema.

60. The system of claim 45, further comprising the second cluster searching the data using a late-binding schema.

61. The system of claim 45, wherein the shared storage is accessible by a third cluster, and further comprising the third cluster accessing the configuration information in the shared storage.

62. The system as described in claim 45, wherein said communicating is performed by a search head at the first cluster.

63. The system as described in claim 45, wherein said communicating is performed by communicating the configuration information through a firewall associated with the first cluster.

64. The system as described in claim 45, wherein the configuration information represents a knowledge object.

65. The system as described in claim 45, wherein the configuration information can comprise one or more of:

saved searches, event types, transactions, tags, field extractions, field transforms, lookups, workflow actions, search commands, or views.