Co-located deployment of a data fabric service system

• US 10,592,561 B2
• Filed: 10/31/2016
• Issued: 03/17/2020
• Est. Priority Date: 09/26/2016
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Bhattacharjee, Arindam, Pal, Sourav
• Primary Examiner(s): Khong, Alexander
• Application Number:
  US15/339,845
• Publication Number:
  US 20190163841A1
• Time in Patent Office: 1,233 Days
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/212 with details for data modelling support
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/252 between a Database Management System and a front-end application
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

The capabilities of a data intake and query system can be improved by implementing the data fabric service (DFS) system in a co-located deployment with the data intake and query system. The DFS system can extend the capabilities of a data intake and query system by leveraging computing assets from anywhere in a big data ecosystem to collectively execute search queries on diverse data systems regardless of whether data stores are internal of the data intake and query system and/or external data stores that are communicatively coupled to the data intake and query system over a network.

Claims

1. A method, comprising:

launching, by a data intake and query system comprising a cluster of search heads, a master control to coordinate control of a search service;

launching, by the data intake and query system, a plurality of worker nodes, each worker node of the plurality of worker nodes communicatively coupled to the master control;

receiving, by the data intake and query system, a search query;

initiating, by a search head of the cluster of search heads, a local search session of the search service;

triggering, by the search head, a distributed search of internal data stores and external data stores, wherein the search head triggers the search of the internal data stores using one or more indexers communicatively coupled to the search head and triggers the search of the external data stores using at least one of the plurality of worker nodes, wherein the internal data stores store data as a plurality of time-indexed events, each event including a portion of raw machine data associated with a timestamp;

retrieving, using the one or more indexers, first partial search results from the internal data stores;

sending, using the one or more indexers, the first partial search results to the plurality of worker nodes, wherein the plurality of worker nodes receive second partial search results from the external data stores, aggregate the first partial search results and the second partial search results into aggregated partial search results and send the aggregated partial search results to the master control;

receiving, by the search head, the aggregated partial search results from the master control; and

rendering an output of the aggregated partial search results or data indicative of the aggregated partial search results for display on a user interface of a display device associated with a user.

2. The method of claim 1, wherein the plurality of worker nodes are co-located with the one or more indexers.

3. The method of claim 1, wherein the first partial search results comprise time-indexed events including portions of raw machine data.

4. The method of claim 1, wherein the search query is input by the user and expressed in a pipelined search language.

5. The method of claim 4, wherein the search query includes an explicit search parameter input by the user causing the data intake and query system to define a portion of a search scheme to apply the search query to the external data stores.

6. The method of claim 5, wherein the output is rendered by the display device as a timeline visualization.

7. The method of claim 1, wherein the external data stores store data in a structured format, and wherein the second partial search results are in a specified format.

8. The method of claim 7, wherein the structured format is different from an events format of the internal data stores.

9. The method of claim 1, wherein the master control comprises a daemon.

10. The method of claim 1, further comprising launching, by the data intake and query system, a monitor of the master control.

11. The method of claim 1, wherein the data intake and query system launches the master control using a modular input.

12. The method of claim 1, wherein the data intake and query system launches the plurality of worker nodes using a modular input.

13. The method of claim 1, further comprising launching, by the data intake and query system, a monitor to monitor the plurality of worker nodes.

14. The method of claim 1, wherein the aggregated partial search results are stored in memory at the plurality of worker nodes before being transferred between other worker nodes to execute a multi-staged parallel aggregation operation.

15. The method of claim 1, wherein the data intake and query system comprises the one or more indexers, and wherein the one or more indexers launch the plurality of worker nodes.

16. The method of claim 15, wherein each indexer of the one or more of indexers launches a corresponding worker node of the plurality of worker nodes.

17. The method of claim 16, wherein each indexer retrieves a portion of the partial search results from at least one associated internal data store of the internal data stores and sends the respective portion of the partial search results to the corresponding worker node.

18. The method of claim 1, further comprising launching, by the data intake and query system, one or more instances of the search service.

19. The method of claim 18, wherein a first search head of the cluster of search heads launches the one or more instances of the search service.

20. The method of claim 18, wherein the one or more instances of the search service comprises a plurality of search services.

21. The method of claim 1, wherein only a single master control is launched.

22. The method of claim 1, further comprising, based at least in part on a determination that the master control has failed, restarting the master control.

23. The method of claim 1, further comprising, based at least in part on a determination that the master control and a primary search head of the cluster of search heads have failed, identifying a different search head as the primary search head and relaunching the master control.

25. The method of claim 1, wherein the output is rendered by the display device as a timeline visualization.

26. The method of claim 1, wherein the first partial search results are combined as chunks of events prior to the sending the first partial search results to the plurality of worker nodes.

27. The method of claim 26, wherein the chunks of events comprise a time order of time-indexed events.

24. A system comprising:

a memory; and

a processing device coupled with the memory to;

launch a master control to coordinate control of a search service;

launch a plurality of worker nodes, each worker node of the plurality of worker nodes communicatively coupled to the master control;

receive a search query;

initiate, by a search head, a local search session of the search service;

trigger a distributed search of internal data stores and external data stores, wherein the search head triggers the search of the internal data stores using one or more indexers communicatively coupled to the search head and triggers the search of the external data stores using at least one of the plurality of worker nodes, wherein the internal data stores store data as a plurality of events, each event including a portion of raw machine data associated with a timestamp;

retrieve first partial search results from the internal data stores;

send the first partial search results to the plurality of worker nodes, wherein the plurality of worker nodes receive second partial search results from the external data stores, aggregate the first partial search results and the second partial search results into aggregated partial search results and send the aggregated partial search results to the master control;

receive the aggregated partial search results from the master control; and

render an output of the aggregated partial search results or data indicative of the aggregated partial search results for display on a user interface of a display device associated with a user.

28. The system of claim 24, wherein the search query is input by the user and expressed in a pipelined search language.

29. The system of claim 24, wherein the external data stores store data in a structured format, and wherein the second partial search results are in a specified format.

30. A non-transitory computer-readable medium encoding instructions thereon that, in response to execution by one or more processing devices, cause the one or more processing devices to perform operations comprising:

launching, by a data intake and query system comprising a cluster of search heads, a master control to coordinate control of a search service;

launching, by the data intake and query system, a plurality of worker nodes, each worker node of the plurality of worker nodes communicatively coupled to the master control;

receiving, by the data intake and query system, a search query;

initiating, by a search head of the cluster of search heads, a local search session of the search service;

triggering, by the search head, a distributed search of internal data stores and external data stores, wherein the search head triggers the search of the internal data stores using one or more indexers communicatively coupled to the search head and triggers the search of the external data stores using at least one of the plurality of worker nodes, wherein the internal data stores store data as a plurality of events, each event including a portion of raw machine data associated with a timestamp;

retrieving, using the one or more indexers, first partial search results from the internal data stores;

sending, using the one or more indexers, the first partial search results to the plurality of worker nodes, wherein the plurality of worker nodes receive second partial search results from the external data stores, aggregate the first partial search results and the second partial search results into aggregated partial search results and send the aggregated partial search results to the master control;

receiving, by the search head, the aggregated partial search results from the master control; and

rendering an output of the aggregated partial search results or data indicative of the aggregated partial search results for display on a user interface of a display device associated with a user.