Batch searches in data fabric service system

• US 10,592,563 B2
• Filed: 10/31/2016
• Issued: 03/17/2020
• Est. Priority Date: 09/26/2016
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Pal, Sourav, Bhattacharjee, Arindam, Pride, Christopher
• Primary Examiner(s): Khong, Alexander
• Application Number:
  US15/339,853
• Publication Number:
  US 20190163824A1
• Time in Patent Office: 1,233 Days
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/212 with details for data modelling support
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/252 between a Database Management System and a front-end application
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

The disclosed embodiments include a technique to obtain search results from the application of transformation operations on partial search results obtained from across internal and/or external data sources. Examples of transformation operations include arithmetic operations such as an average, mean, count, or the like. Examples of reporting transformations include join operations, statistics, sort, top head. Hence, the search results of a search query can be derived from partial search result rather than include the actual partial search results. In this case, the ordering of the search results may be nonessential. An example of a search query that requires a transformation operation is a “batch” or “reporting” search query. The related disclosed techniques involve obtaining data stored in the bid data ecosystem, and returning that data or data derived from that data.

Claims

1. A computer-implemented method for performing a batch search operation across one or more worker nodes, the method comprising:

receiving, using a first computing device, partial search results from a plurality of data sources, wherein each of the partial search results satisfies a portion of a search query received by a data intake and query system,

wherein at least one first partial search result of the partial search results is received from a subset of internal data sources of the data intake and query system, wherein the at least one first partial search result comprises one or more first events in a first format, each first event corresponding to at least one second event stored in the subset of internal data sources, wherein each second event includes raw machine data associated with a timestamp and reflects activity within an information technology infrastructure, and

wherein at least one second partial search result of the partial search results is received from one or more external data sources apart from the data intake and query system, wherein the at least one second partial search result includes data in a second format that is different from the first format;

transforming the data of the at least one second partial search result into the first format, using the first computing device, to produce a commonly formatted partial search result;

processing the commonly formatted partial search result and the at least one first partial search result to generate finalized partial search results,

wherein the commonly formatted partial search result is processed based on a determination that the commonly formatted partial search result was received from the one or more external data sources, and

wherein the at least one first partial search result is processed based on a determination that the at least one first partial search result was received from the one or more internal data stores; and

communicating the finalized partial search results to a second computing device, wherein the second computing device processes the finalized partial search results with other finalized partial search results to generate finalized search results.

2. The computer-implemented method of claim 1, wherein processing the commonly formatted partial search result and the at least one first partial search result comprises aggregating the commonly formatted partial search result and the at least one first partial search result to generate the finalized partial search results.

3. The computer-implemented method of claim 2, wherein aggregating comprises performing one or more transformation operations on the commonly formatted partial search result and the at least one first partial search result to generate the finalized partial search results.

4. The computer-implemented method of claim 2, further comprising transmitting the finalized partial search results in parallel from at least two worker nodes to a data fabric service master associated with the data intake and query system.

5. The computer-implemented method of claim 4, wherein each worker node included in the at least two worker nodes comprises a machine that is an agent of the data intake and query system and is configured to process data collected from internal data sources and external data sources.

6. The computer-implemented method of claim 2, wherein receiving the partial search results from the plurality of data sources comprises reading shards of the at least one first partial search result from the subset of internal data sources, the shards of the at least one first partial search result being generated in accordance with at least one policy that is based on at least one of one or more hash values and one or more field names.

7. The computer-implemented method of claim 6, wherein processing the commonly formatted partial search result and the at least one first partial search result further comprises aggregating the shards of the at least one first partial search result obtained from the subset of the internal data sources.

8. The computer-implemented method of claim 2, wherein receiving the at least one second partial search result from the plurality of one or more external data sources comprises transmitting, using the first computing device, one or more predicates associated with one or more portions of the search query to the one or more external data sources, wherein the one or more external data sources use the one or more predicates to identify records that satisfy the one or more portions of the search query and include the identified records in the at least one second partial search result for the first computing device.

9. The computer-implemented method of claim 8, wherein receiving the at least one second partial search result from the one or more external data sources comprises determining, using the first computing device, that the one or more external data sources are able to execute a sharded query, and reading shards of the at least one second partial search result from the one or more external data sources, the shards of the at least one second partial search result being generated in accordance with at least one policy that is based on at least one of one or more hash values and one or more field names.

10. The computer-implemented method of claim 8, wherein processing the commonly formatted partial search result and the at least one first partial search result further comprises performing one or more stream processing operations on or aggregating the commonly formatted partial search result to generate a streamed or aggregated commonly formatted partial search result.

11. A non-transitory computer-readable medium including instructions that, when executed by a processor included in a first computing device, cause the processor to perform the steps of:

receiving partial search results from a plurality of data sources, wherein each of the partial search results satisfies a portion of a search query received by a data intake and query system,

wherein at least one first partial search result of the partial search results is received from a subset of internal data sources of the data intake and query system, wherein the at least one first partial search result comprises one or more first events in a first format, each first event corresponding to at least one second event stored in the subset of internal data sources, wherein each second event includes raw machine data associated with a timestamp and reflects activity within an information technology infrastructure, and

wherein at least one second partial search result of the partial search results is received from one or more external data sources apart from the data intake and query system, wherein the at least one second partial search result includes data in a second format that is different from the first format;

transforming the data of the at least one second partial search result into the first format to produce a commonly formatted partial search result;

processing the commonly formatted partial search result and the at least one first partial search result to generate finalized partial search results,

wherein the commonly formatted partial search result is processed based on a determination that the commonly formatted partial search result was received from the one or more external data sources, and

wherein the at least one first partial search result is processed based on a determination that the at least one first partial search result was received from the one or more internal data stores; and

communicating the finalized partial search results to a second computing device, wherein the second computing device processes the finalized partial search results with other finalized partial search results to generate finalized search results.

12. The non-transitory computer-readable medium of claim 11, wherein processing the commonly formatted partial search result and the at least one first partial search result comprises aggregating the commonly formatted partial search result and the at least one first partial search result to generate the finalized partial search results.

13. The non-transitory computer-readable medium of claim 12, wherein aggregating comprises performing one or more transformation operations on the commonly formatted partial search result and the at least one first partial search result to generate the finalized partial search results.

14. The non-transitory computer-readable medium of claim 12, further comprising transmitting the finalized partial search results in parallel with at least one other computing device to a data fabric service master associated with the data intake and query system.

15. The non-transitory computer-readable medium of claim 14, wherein each of the first computing device and the at least one other computing device is an agent of the data intake and query system and is configured to process data collected from internal data sources and external data sources.

16. The non-transitory computer-readable medium of claim 12, wherein receiving the partial search results from the plurality of data sources comprises reading shards of the at least one first partial search result from the subset of internal data sources, the shards of the at least one first partial search result being generated in accordance with at least one policy that is based on at least one of one or more hash values and one or more field names.

17. The non-transitory computer-readable medium of claim 16, wherein processing the commonly formatted partial search result and the at least one first partial search result further comprises aggregating the shards of the at least one first partial search result obtained from the subset of the internal data sources.

18. The non-transitory computer-readable medium of claim 12, wherein receiving the at least one second partial search result from the one or more external data sources comprises transmitting one or more predicates associated with one or more portions of the search query to the one or more external data sources, wherein the one or more external data sources use the one or more predicates to identify records that satisfy the one or more portions of the search query and include the identified records in the at least one second partial search result for the first computing device.

19. The non-transitory computer-readable medium of claim 18, wherein receiving the at least one second partial search result from the one or more external data sources comprises determining that the one or more external data sources are able to execute a sharded query, and reading shards of the at least one second partial search result from the one or more external data sources, the shards of the at least one second partial search result being generated in accordance with at least one policy that is based on at least one of one or more hash values and one or more field names.

20. The non-transitory computer-readable medium of claim 18, wherein processing the commonly formatted partial search result and the at least one first partial search result further comprises performing one or more stream processing operations on or aggregating the commonly formatted partial search result to generate a streamed or aggregated commonly formatted partial search result.

21. A system, comprising:

one or more worker nodes, wherein each worker node includes a processor and a memory that stores respective instructions, and, when each processor executes the respective instructions, wherein at least one worker node is configured to;

receive partial search results from a plurality of data sources, wherein each of the partial search results satisfies a portion of a search query received by a data intake and query system,

wherein at least one first partial search result of the partial search results is received from a subset of internal data sources of the data intake and query system, wherein the at least one first partial search result comprises one or more first events in a first format, each first event corresponding to at least one second event stored in the subset of internal data sources, wherein each second event includes raw machine data associated with a timestamp and reflects activity within an information technology infrastructure, and

wherein at least one second partial search result of the partial search results is received from one or more external data sources apart from the data intake and query system, wherein the at least one second partial search result includes data in a second format that is different from the first format;

transform the data of the at least one second partial search result into the first format, to produce commonly formatted partial search results;

processing the commonly formatted partial search result and the at least one first partial search result to generate finalized partial search results,

wherein the commonly formatted partial search result is processed based on a determination that the commonly formatted partial search result was received from the one or more external data sources, and

wherein the at least one first partial search result is processed based on a determination that the at least one first partial search result was received from the one or more internal data stores; and

communicating the finalized partial search results to a second computing device, wherein the second computing device processes the finalized partial search results with other finalized partial search results to generate finalized search results.

22. The system of claim 21, wherein the at least one worker node is configured to process the commonly formatted partial search result and the at least one first partial search result by aggregating the commonly formatted partial search result and the at least one first partial search result to generate the finalized partial search results.

23. The system of claim 22, wherein aggregating comprises performing one or more transformation operations on the commonly formatted partial search result and the at least one first partial search result to generate the finalized partial search results.

24. The system of claim 22, wherein the one or more worker nodes comprise at least two worker nodes, and the at least two worker nodes are configured to transmit the finalized partial search results in parallel to a data fabric service master associated with the data intake and query system.

25. The system of claim 24, wherein each worker node included in the at least two worker nodes comprises a machine that is an agent of the data intake and query system and is configured to process data collected from internal data sources and external data sources.

26. The system of claim 22, wherein the at least one worker node is configured to receive the partial search results from the plurality of data sources by reading shards of the at least one first partial search result from the subset of internal data sources, the shards of the at least one first partial search result being generated in accordance with at least one policy that is based on at least one of one or more hash values and one or more field names.

27. The system of claim 26, wherein the at least one worker node is configured to process the at least one first partial search result by aggregating the shards of the at least one first partial search result obtained from the subset of the internal data sources.

28. The system of claim 22, wherein the at least one worker node is configured to receive the partial search results from the plurality of data sources by transmitting one or more predicates associated with one or more portions of the search query to the one or more external data sources, wherein the one or more external data sources use the one or more predicates to identify records that satisfy the one or more portions of the search query and include the identified records in the at least one second partial search result for the first computing device.

29. The system of claim 28, wherein the at least one worker node is further configured to receive the partial search results from the plurality of data sources by determining that the one or more external data sources are able to execute a sharded query, and reading shards of the at least one second partial search result from the one or more external data sources, the shards of the at least one second partial search result being generated in accordance with at least one policy that is based on at least one of one or more hash values and one or more field names.

30. The system of claim 28, wherein the at least one worker node is configured to process the commonly formatted partial search result by performing one or more stream processing operations on or aggregating the commonly formatted partial search result to generate commonly formatted partial search result.