Data fabric service system architecture

• US 10,353,965 B2
• Filed: 09/26/2016
• Issued: 07/16/2019
• Est. Priority Date: 09/26/2016
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Pal, Sourav, Pride, Christopher, Bhattacharjee, Arindam, Wang, Xiaowei, Hodge, James Alasdair Robert, Ahamed, Mustafa
• Primary Examiner(s): Khong, Alexander
• Application Number:
  US15/276,717
• Publication Number:
  US 20190163821A1
• Time in Patent Office: 1,023 Days
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/212 with details for data modelling support
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/252 between a Database Management System and a front-end application
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Disclosed is a technique that can be performed in a distributed computer network. The technique can include a data index and query system that receives search query, defines a search scheme for applying the search query on distributed data storage systems including an internal data storage system of the data index and query system and an external data storage system. The internal data storage system stores data as time-indexed events including respective segments of raw machine data. The data index and query system can transfer a portion of the search scheme to a search service, which can return search results obtained by application of the search scheme to the distributed data storage systems including the internal data storage system and the external data storage system. Lastly, the search results or data indicative of the search results can be output on a display device to the user.

Claims

1. A method comprising:

receiving, by a data intake and query system, a search query;

defining, by the data intake and query system, a search scheme for applying the search query on a plurality of distributed data storage systems including an internal data storage system of the data intake and query system and an external data storage system communicatively coupled to the data intake and query system over a computer network, wherein the internal data storage system stores data as a plurality of time-indexed events including respective segments of raw machine data;

transferring, by the data intake and query system, a portion of the search scheme to a search service for obtaining partial search results from the external data storage system;

producing, by the data intake and query system, partial search results by applying a portion of the search scheme to the internal data storage system;

transmitting, by the data intake and query system over the computer network, the partial search results to at least one worker node communicatively coupled to the external data storage system over the computer network, the at least one worker node being communicatively coupled to the search service;

receiving, by the data intake and query system from the search service, search results of the search query obtained by application of the search scheme to the plurality of distributed data storage systems including the internal data storage system and the external data storage system;

wherein the search results obtained by the data intake and query system from the search service are based on a combination of the partial search results of the internal data storage system and the partial search results obtained from the external data storage system combined by the at least one worker node; and

causing output of the search results or data indicative of the search results by a display device to a user.

2. The method of claim 1, wherein the search query is input by the user and expressed in a pipelined search language.

3. The method of claim 1, wherein the output is rendered by the display device as a timeline visualization.

4. The method of claim 1, wherein the external data storage system stores data in a structured format, and the search results are in a specified format.

5. The method of claim 1, wherein the search query includes an explicit search parameter input by the user causing the data intake and query system to define the portion of the search scheme transferred to the search service to apply the search query to the external data storage system.

6. The method of claim 1, wherein defining the search scheme comprises:

determining, by the data intake and query system, that the search query requires searching the plurality of distributed data storage systems; and

defining a plurality of search phases for searching the plurality of distributed data storage systems and obtaining the search results from the search service;

wherein the search scheme includes the plurality of search phases, and the portion of the search scheme includes at least one of the plurality of search phases.

7. The method of claim 1, wherein defining the search scheme comprises:

determining, by the data intake and query system, that the search query requires searching the plurality of distributed data storage systems; and

defining a plurality of ordered search phases for searching the plurality of distributed data storage systems and obtaining the search results from the search service in accordance with an order of the plurality of ordered search phases;

wherein the search scheme includes the plurality of ordered search phases, and the portion of the search scheme includes at least one of the plurality of ordered search phases for searching the external data storage system after execution of another one of the plurality of ordered search phases for searching the internal data storage system.

8. The method of claim 1, further comprising, prior to receiving the search results by the data intake and query system:

determining, by the data intake and query system, that the search query requires searching the plurality of distributed data storage systems; and

responsive to determining that the search query requires searching the plurality of distributed data storage systems, initiating a communications search protocol with the search service and causing the transferring of the portion of the search scheme to the search service by the data intake and query system.

9. The method of claim 1, further comprising, prior to receiving the search results:

executing, by the data intake and query system, a first search operation on the internal data storage system in accordance with the search scheme; and

sending, by the data intake and query system to the search service, a message causing the search service to initiate a second search operation for searching the external storage system in accordance with the search scheme.

10. The method of claim 1,

wherein the search results obtained by the data intake and query system from the search service correspond to the combination of the partial search results of the internal data storage system and the partial search results obtained from the external data storage system combined by the at least one worker node in an arrangement of time-ordered events.

11. The method of claim 1,

wherein the search results obtained by the data intake and query system from the search service are derived from the combination of the partial search results of the internal data storage system and the partial search results obtained from the external data storage system combined by the at least one worker node.

12. A method comprising:

receiving, by a search service, data representing a portion of a search scheme defined by a data intake and query system for a plurality of worker nodes to collect a plurality of partial search results over a computer network from a plurality of distributed data storage systems including a plurality of external data storage systems and an internal data storage system of the data intake and query system, wherein the search scheme comprises a plurality of search phases;

defining, by the search service, an executable search process based on the received portion of the search scheme defined by the data intake and query system, the executable search process being defined as a directed acyclic graph to include a first phase causing the plurality of worker nodes to extract and collect the plurality of partial search results from the plurality of external data storage systems, a second phase causing the plurality of worker nodes to collect the plurality of partial search results from the internal data storage system, and a third phase causing the search service to collect the plurality of partial search results of the plurality of external data storage systems and the plurality of partial search results of the internal data storage system from the plurality of worker nodes;

executing, by the search service, the executable search process to cause the plurality of worker nodes to extract and collect the plurality of partial search results from the plurality of external data storage systems and collect the plurality of partial search results from the internal data storage system over the computer network;

receiving, by the search service, a plurality of aggregate partial search results based on the plurality of partial search results of the plurality of external data storage systems and the plurality of partial search results of the internal data storage system; and

sending, by the search service, the plurality of aggregate partial search results to the data intake and query system.

13. A method comprising:

receiving, by a search service, data representing a portion of a search scheme defined by a data intake and query system for a plurality of worker nodes to collect partial search results over a computer network from a plurality of distributed data storage systems including a plurality of external data storage systems and an internal data storage system of the data intake and query system;

defining, by the search service, an executable search process based on the received portion of the search scheme defined by the data intake and query system;

executing, by the search service, the executable search process to cause the plurality of worker nodes to collect a plurality of partial search results from the plurality of external data storage systems and collect a plurality of partial search results from the internal data storage system over the computer network and produce a plurality of aggregate partial search results by combining the plurality of partial search results of the plurality of external data storage systems with the plurality of partial search results of the internal data storage system as an arrangement of time-ordered events;

receiving, by the search service, the plurality of aggregate partial search results; and

sending, by the search service, the plurality of aggregate partial search results to the data intake and query system.

14. The method of claim 13, wherein the search scheme comprises a plurality of search phases, and defining the executable search process comprises:

defining, by the search service, the executable search process as a directed acyclic graph including at least some of the plurality of search phases.

15. The method of claim 13 wherein the search scheme comprises a plurality of search phases, and defining the executable search process comprises:

defining, by the search service, the executable search process to include a first phase causing the plurality of worker nodes to extract and collect the plurality of partial search results from the plurality of external data storage systems and a second phase causing the plurality of worker nodes to collect the plurality of partial search results from the internal data storage system.

16. The method of claim 13 wherein the search scheme comprises a plurality of ordered search phases, and defining the executable search process comprises:

defining, by the search service, the executable search process to include a first phase causing the plurality of worker nodes to extract and collect the plurality of partial search results from the plurality of external data storage systems, and a second phase causing the plurality of worker nodes to collect the plurality of partial search results in parallel from peer indexers of the internal data storage system, the first phase occurring before the second phase.

17. The method of claim 13, wherein the plurality of partial search results of the internal data storage system are formatted as a plurality of time-indexed events including respective segments of raw machine data.

18. The method of claim 13, wherein the plurality of partial search results of the internal data storage system are in a first format as a plurality of time-indexed events including raw machine data, the plurality of partial search results of at least some of the external data storage systems are in a second format different from the first format, and the plurality of aggregate partial search results are in a specified format.

19. The method of claim 13, further comprising:

metering an amount of time and computing resources utilized to complete the executable search process.

20. The method of claim 13, wherein the plurality of partial search results of the plurality of external data storage systems are searched in accordance with the received portion of the search scheme, and the plurality of partial search results of the internal data storage system are searched in accordance with a different portion of the search scheme.

21. A method comprising:

receiving, by a worker node over a computer network, search instructions defined by a search service for the worker node to collect a plurality of partial search results from a plurality of distributed data storage systems including a plurality of partial search results of an external data storage system and a plurality of partial search results of an internal data storage system of a data intake and query system;

collecting, by the worker node over a computer network, the plurality of partial search results of the external data storage system searched in accordance with the search instructions;

collecting, by the worker node over the computer network, the plurality of partial search results of the internal data storage system;

producing, by the worker node, a plurality of aggregate partial search results from the plurality of partial search results of the external data storage system and the plurality of partial search results of the internal data storage system by combining the plurality of partial search results of the external data storage system with the plurality of partial search results of the internal data storage system as an arrangement of time-ordered events; and

providing, by the worker node over the computer network to the search service, the plurality of aggregate partial search results in response to the search instructions.

22. The method of claim 21, wherein the plurality of partial search results of the internal data storage system are formatted as a plurality of time-indexed events including respective segments of raw machine data.

23. The method of claim 21, wherein the plurality of partial search results of the internal data storage system are in a first format as a plurality of time-indexed events including raw machine data, the plurality of partial search results of the external data storage system are in a second format different from the first format, and the plurality of aggregate partial search results are in a specified format.

24. The method of claim 21, wherein producing the plurality of aggregate partial search results comprises:

deriving, by the worker node, the plurality of aggregate partial search results by performing an operation on a combination of the plurality of partial search results of the external data storage system and the plurality of partial search results of the internal data storage system.

25. A method comprising:

receiving, by a first worker node over a computer network, search instructions defined by a search service for the first worker node to collect a plurality of partial search results from a plurality of distributed data storage systems including a first plurality of partial search results of a first external data storage system and a second plurality of partial search results of an internal data storage system of a data intake and query system;

collecting, by the first worker node over a computer network, the first plurality of partial search results of the first external data storage system searched in accordance with the search instructions;

collecting, by the first worker node over the computer network, the second plurality of partial search results of the internal data storage system;

collecting, by the first worker node over the computer network, a third plurality of partial search results extracted from a second external data storage system by a second worker node, the second worker node being communicatively coupled over the computer network to the second external data storage system;

sending, by the first worker node over the computer network, at least a portion of the first plurality of partial search results to the second worker node;

producing, by the first worker node, a plurality of aggregate partial search results from the first plurality of partial search results of the first external data storage system, the second plurality of partial search results of the internal data storage system, and the third plurality of partial search results of the second external data storage system; and

providing, by the first worker node over the computer network to the search service, the plurality of aggregate partial search results in response to the search instructions.