Parallel exporting in a data fabric service system

• US 10,599,723 B2
• Filed: 10/31/2016
• Issued: 03/24/2020
• Est. Priority Date: 09/26/2016
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Bhattacharjee, Arindam, Pal, Sourav, Wang, Xiaowei, Pride, Christopher, Hodge, James Alasdair Robert
• Primary Examiner(s): Khong, Alexander
• Application Number:
  US15/339,835
• Publication Number:
  US 20190163823A1
• Time in Patent Office: 1,240 Days
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/212 with details for data modelling support
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/252 between a Database Management System and a front-end application
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

The disclosed embodiments include techniques for exporting partial search results in parallel from peer indexers of a data intake and query system to the worker nodes. In particular, partial search results (e.g., time-indexed events) obtained from peer indexers can be exported in parallel from the peer indexers to worker nodes. Exporting the partial search results from the peer indexers in parallel can improve the rate at which the partial search results are transferred to the worker nodes for subsequent combination with partial search results of the external data systems. As such, the rate at which the search results of a search query can be obtained from the distributed data system can be improved by implementing parallel export techniques.

Claims

1. A computer-implemented method for processing a search query, the method comprising:

receiving a subquery corresponding to a portion of a query, the query received by a data intake and query system;

obtaining a plurality of first events based on the subquery, each first event corresponding to at least one second event stored in a subset of internal data sources of the data intake and query system, wherein each second event includes raw machine data associated with a timestamp and reflects activity within an information technology infrastructure;

generating a plurality of event chunks from the plurality of first events, wherein each event chunk comprises multiple first events of the plurality of first events; and

concurrently transmitting a first event chunk of the plurality of event chunks to a first worker node and a second event chunk of the plurality of event chunks to a second worker node for additional processing.

2. The computer-implemented method of claim 1, wherein the at least one partial search result is plurality of first events based on the subquery are obtained from one or more internal data sources associated with the data intake and query system.

3. The computer-implemented method of claim 1, wherein the obtaining the plurality of first events based on the subquery, the generating the plurality of event chunks from the plurality of first events, and the concurrently transmitting the first event chunk of the plurality of event chunks to the first worker node and the second event chunk of the plurality of event chunks to the second worker node are performed by a first indexer associated with a data intake and query system.

4. The computer-implemented method of claim 1, wherein the obtaining the plurality of first events based on the subquery, the generating the plurality of event chunks from the plurality of first events, and the concurrently transmitting the first event chunk of the plurality of event chunks to the first worker node and the second event chunk of the plurality of event chunks to the second worker node are performed by each indexer included in a plurality of indexers associated with a data intake and query system.

5. The computer-implemented method of claim 1, wherein obtaining the plurality of first events based on the subquery comprises applying at least one policy to the plurality of first events, the at least one policy being based on at least one of one or more hash values and one or more field names.

6. The computer-implemented method of claim 1, further comprising determining that the first worker node and the second worker node are available to process the first event chunk and the second event chunk, respectively.

7. The computer-implemented method of claim 1, wherein instructions are transmitted to the first worker node and the second worker node via a data fabric service master associated with the data intake and query system, the instructions indicating to the first worker node and the second worker node how to process the first event chunk and the second event chunck, respectively.

8. The computer-implemented method of claim 7, wherein, based on the instructions, the first worker node and the second worker node analyze timestamps associated with the first event chunk and second event chunk, respectively, to generate time-ordered event chunks, and stream the time-ordered event chunks to the data fabric service master.

9. The computer-implemented method of claim 7, wherein, based on the instructions, the first worker node aggregates the first event chunk with other event chunks received by the first worker node to generate aggregated partial search results associated with one or more internal data sources included within the data intake and query system.

10. The computer-implemented method of claim 9, wherein, based on the instructions, the first worker node further aggregates the generated aggregated partial search results associated with the one or more internal data sources with partial search results associated with one or more external data sources to generate finalized partial search results, and transmits the finalized partial search results to the data fabric service master.

11. A non-transitory computer-readable medium including instructions that, when executed by a processor included in an indexer, cause the processor to perform the steps of:

receiving a subquery corresponding to a portion of a query, the query received by a data intake and query system;

obtaining a plurality of first events based on the subquery, each first event corresponding to at least one second event stored in a subset of internal data sources of the data intake and query system, wherein each second event includes raw machine data associated with a timestamp and reflects activity within an information technology infrastructure;

generating a plurality of event chunks from the plurality of first events, wherein each event chunk comprises multiple first events of the plurality of first events; and

concurrently transmitting a first event chunk of the plurality of event chunks to a first worker node and a second event chunk of the plurality of event chunks to a second worker node for additional processing.

12. The non-transitory computer-readable medium of claim 11, wherein the plurality of first events based on the subquery are obtained from one or more internal data sources associated with the data intake and query system.

13. The non-transitory computer-readable medium of claim 11, wherein the indexer is associated with the data intake and query system.

14. The non-transitory computer-readable medium of claim 13, wherein the steps of obtaining the plurality of first events based on the subquery, generating the plurality of event chunks from the plurality of first events, and concurrently transmitting the first event chunk of the plurality of event chunks to the first worker node and the second event chunk of the plurality of event chunks to the second worker node are also performed by at least one other indexer included in the data intake and query system.

15. The non-transitory computer-readable medium of claim 11, wherein obtaining the plurality of first events based on the subquery comprises applying at least one policy to the plurality of first events, the at least one policy being based on at least one of one or more hash values and one or more field names.

16. The non-transitory computer-readable medium of claim 11, further comprising determining that the first worker node and the second worker node are available to process the respective first or second event chunk of the plurality of event chunks.

17. The non-transitory computer-readable medium of claim 11, wherein instructions are transmitted to the first worker node and the second worker node via a data fabric service master associated with the data intake and query system, the instructions indicating to the first worker node and the second worker node how to process the respective first or second event chunk of the plurality of event chunks.

18. The non-transitory computer-readable medium of claim 17, wherein, based on the instructions, the first worker node and the second worker node analyze timestamps associated with the first or second event chunk, respectively, to generate time-ordered event chunks, and stream the time-ordered event chunks to the data fabric service master.

19. The non-transitory computer-readable medium of claim 17, wherein, based on the instructions, the first worker node aggregates the first event chunk with other event chunks received by the first worker node to generate aggregated partial search results associated with one or more internal data sources included within the data intake and query system.

20. The non-transitory computer-readable medium of claim 19, wherein, based on the instructions, the first worker node further aggregates the generated aggregated partial search results associated with the one or more internal data sources with partial search results associated with one or more external data sources to generate finalized partial search results, and transmits the finalized partial search results to the data fabric service master.

21. A data intake and query system, comprising:

one or more indexers, wherein each indexer includes a processor and a memory that stores respective instructions, and, when each processor executes the respective instructions, the one or more indexers are configured to;

receive a subquery corresponding to a portion of a query, the query received by a data intake and query system;

obtain a plurality of first events based on the subquery, each first event corresponding to at least one second event stored in a subset of internal data sources of the data intake and query system, wherein each second event includes raw machine data associated with a timestamp and reflects activity within an information technology infrastructure;

generate a plurality of event chunks from the plurality of first events, wherein each event chunk comprises multiple first events of the plurality of first events; and

concurrently transmit a first event chunk of the plurality of event chunks to a first worker node and a second event chunk of the plurality of event chunks to a second worker node for additional processing.

22. The data intake and query system of claim 21, further comprising one or more internal data sources from which the plurality of first events based on the subquery are obtained.

23. The data intake and query system of claim 21, wherein the one or more indexers comprise a first indexer, and the first indexer is configured to obtain the plurality of first events based on the subquery, generate the plurality of event chunks from the plurality of first events, and concurrently transmit the first event chunk of the plurality of event chunks to the first worker node and the second event chunk of the plurality of event chunks to the second worker node.

24. The data intake and query system of claim 21, wherein the one or more indexers comprise at least two indexers, and each indexer included in the at least two indexers is configured to obtain the plurality of first events based on the subquery, generate the plurality of event chunks from the plurality of first events, and concurrently transmit the first event chunk of the plurality of event chunks to the first worker node and the second event chunk of the plurality of event chunks to the second worker node.

25. The data intake and query system of claim 21, wherein the one or more indexers are configured to obtain the plurality of first events based on the subquery by applying at least one policy to the plurality of first events, the at least one policy being based on at least one of one or more hash values and one or more field names.

26. The data intake and query system of claim 21, wherein the one or more indexers are further configured to determine that the first worker node and the second worker node are available to process the respective first or second event chunk of the plurality of event chunks.

27. The data intake and query system of claim 21, wherein instructions are transmitted to the first worker node and the second worker node via a data fabric service master, the instructions indicating to the first worker node and the second worker node how to process the respective first or second event chunk of the plurality of event chunks.

28. The data intake and query system of claim 27, wherein, based on the instructions, the first worker node and the second worker node analyze timestamps associated with the first or second event chunk, respectively, to generate time-ordered event chunks, and stream the time-ordered event chunks to the data fabric service master.

29. The data intake and query system of claim 27, wherein, based on the instructions, the first worker node aggregates the first event chunk with other event chunks received by the first worker node to generate aggregated partial search results associated with one or more internal data sources included within the data intake and query system.

30. The data intake and query system of claim 29, wherein, based on the instructions, the first second worker node further aggregates the generated aggregated partial search results associated with the one or more internal data sources with partial search results associated with one or more external data sources to generate finalized partial search results, and transmits the finalized partial search results to the data fabric service master.