Cursored searches in a data fabric service system

• US 10,585,951 B2
• Filed: 10/31/2016
• Issued: 03/10/2020
• Est. Priority Date: 09/26/2016
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Bhattacharjee, Arindam, Pal, Sourav, Pride, Christopher
• Primary Examiner(s): Khong, Alexander
• Application Number:
  US15/339,833
• Publication Number:
  US 20190163822A1
• Time in Patent Office: 1,226 Days
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/212 with details for data modelling support
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/252 between a Database Management System and a front-end application
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

The disclosed embodiments include techniques to obtain ordered search results based on partial search results from across multiple diverse internal and/or external data sources. The ordering of the search results may be with respect to a parameter associated with the partial search results. An example of a parameter includes time. As such, the disclosed technique can provide a time-ordered search result based on partial search results obtained from across multiple internal and/or external data sources. Moreover, the disclosed technique can provide time-ordered search results regardless of whether the partial search results obtained from the diverse data sources are timestamped.

Claims

1. A computer-implemented method for performing a time-ordered search operation across one or more worker nodes, the method comprising:

receiving partial search results from a plurality of data sources, wherein each of the partial search results satisfies a portion of a search query, the search query received by a data intake and query system,

wherein at least one first partial search result of the partial search results is received from a subset of internal data sources of the data intake and query system, wherein the at least one first partial search result comprises one or more first events in a first format, each first event corresponding to at least one second event stored in the subset of internal data sources, wherein each second event includes raw machine data associated with a timestamp and reflects activity within an information technology infrastructure, and

wherein at least one second partial search result of the partial search results is received from one or more external data sources apart from the data intake and query system, wherein the at least one second partial search result includes data in a second format that is different from the first format;

transforming the data of the at least one second partial search result into the first format to produce a commonly formatted partial search result;

identifying at least a first portion of the commonly formatted partial search result that is not time-stamped;

determining a timestamp for the at least a first portion of the commonly formatted partial search result that is not time-stamped; and

processing the at least a first portion of the commonly formatted partial search result that is not time-stamped based on the determined timestamp to generate time-ordered chunks of the commonly formatted partial search result.

2. The computer-implemented method of claim 1, further comprising streaming the time-ordered chunks of the commonly formatted partial search result in parallel from at least two worker nodes of the one or more worker nodes to a data fabric service master associated with the data intake and query system.

3. The computer-implemented method of claim 2, wherein the receiving the partial search results, the transforming the data of the at least one second partial search result, the identifying the at least a first portion of the commonly formatted partial search result that is not time-stamped, and the processing the at least a first portion of the commonly formatted partial search result that is not time-stamped is performed by each worker node included in the at least two worker nodes.

4. The computer-implemented method of claim 3, wherein each worker node included in the at least two worker nodes comprises a machine that is an agent of the data intake and query system and is configured to process data collected from internal data sources and external data sources.

5. The computer-implemented method of claim 2, wherein the determining a timestamp for comprises assigning a different timestamp to each record of the at least a first portion of the commonly formatted partial search result that is not time-stamped, the assigned timestamp indicating a time associated with an ingestion time at which the respective record was ingested by a worker node.

6. The computer-implemented method of claim 5, wherein processing the at least a first portion of the commonly formatted result that is not time-stamped further comprises analyzing the different timestamps associated with the records of the at least a first portion of the commonly formatted partial search result that is not time-stamped to produce at least a portion of the time-ordered chunks of the commonly formatted partial search result.

7. The computer-implemented method of claim 2, the method further comprising identifying at least a second portion of the commonly formatted partial search result that is time-stamped;

and analyzing timestamps associated with the at least a second portion of the of the commonly formatted partial search result to produce at least a portion of the time-ordered chunks of the commonly formatted partial search result.

8. The computer-implemented method of claim 2, further comprising:

receiving the time-ordered chunks of the commonly formatted partial search result streamed in parallel from the at least two worker nodes;

performing one or more deserialization operations on the time-ordered chunks of the commonly formatted partial search result to generate deserialized time-ordered chunks of the commonly formatted partial search result; and

adding the deserialized time-ordered chunks of the commonly formatted partial search result to one or more corresponding collector queues.

9. The computer-implemented method of claim 8, further comprising collating the time-ordered chunks of the commonly formatted partial search result across the one or more collector queues in time-order.

10. The computer-implemented method of claim 1, further comprising deserializing the partial search results received from the plurality of data sources.

11. A non-transitory computer-readable medium including instructions that, when executed by a processor included in a worker node, cause the processor to perform the steps of:

receiving partial search results from a plurality of data sources, wherein each of the partial search results satisfies a portion of a search query, the search query received by a data intake and query system,

wherein at least one first partial search result of the partial search results is received from a subset of internal data sources of the data intake and query system, wherein the at least one first partial search result comprises one or more first events in a first format, each first event corresponding to at least one second event stored in the subset of internal data sources, wherein each second event includes raw machine data associated with a timestamp and reflects activity within an information technology infrastructure, and

wherein at least one second partial search result of the partial search results is received from one or more external data sources apart from the data intake and query system, wherein the at least one second partial search result includes data in a second format that is different from the first format;

transforming the data of the at least one second partial search result into the first format to produce a commonly formatted partial search result;

identifying at least a first portion of the commonly formatted partial search result that is not time-stamped;

determining a timestamp for the at least a first portion of the commonly formatted partial search result that is not time-stamped; and

processing the at least a first portion of the commonly formatted partial search result that is not time-stamped based on the determined timestamp to generate time-ordered chunks of the commonly formatted partial search result.

12. The non-transitory computer-readable medium of claim 11, wherein the instructions, when executed by the processor, further cause the processor to perform the step of:

streaming the time-ordered chunks of the commonly formatted partial search result in parallel with at least one other worker node to a data fabric service master associated with the data intake and query system.

13. The non-transitory computer-readable medium of claim 12, wherein the receiving the partial search results, the transforming the data of the at least one second partial search result, the identifying the at least a first portion of the commonly formatted partial search result that is not time-stamped, and the processing the at least a first portion of the commonly formatted partial search result that is not time-stamped is also performed by the at least one other worker node.

14. The non-transitory computer-readable medium of claim 13, wherein each of the worker node and the at least one other worker node comprises a machine that is an agent of the data intake and query system and is configured to process data collected from internal data sources and external data sources.

15. The non-transitory computer-readable medium of claim 12, wherein the determining a timestamp for comprises assigning a different timestamp to each record of the at least a first portion of the commonly formatted partial search result that is not time-stamped, the assigned timestamp indicating a time associated with an ingestion time at which the respective record was ingested by a worker node.

16. The non-transitory computer-readable medium of claim 15, wherein processing the at least a first portion of the commonly formatted result that is not time-stamped further comprises analyzing the different timestamps associated with the records of the at least a first portion of the commonly formatted partial search result that is not time-stamped to produce at least a portion of the time-ordered chunks of the commonly formatted partial search result.

17. The non-transitory computer-readable medium of claim 12, wherein the instructions, when executed by the processor, further cause the processor to perform the steps of:

identifying at least a second portion of the commonly formatted partial search result that is time-stamped; and

analyzing timestamps associated with the at least a second portion of the commonly formatted partial search result to produce at least a portion of the time-ordered chunks of the commonly formatted partial search result.

18. The non-transitory computer-readable medium of claim 12, wherein:

the time-ordered chunks of the commonly formatted partial search result are received in parallel from the worker node and the at least one other worker node;

one or more deserialization operations are performed on the time-ordered chunks of the commonly formatted partial search result to generate deserialized time-ordered chunks of the commonly formatted partial search result; and

the deserialized time-ordered chunks of the commonly formatted partial search result are added to one or more corresponding collector queues.

19. The non-transitory computer-readable medium of claim 18, wherein the time-ordered chunks of the commonly formatted partial search result are collated across the one or more collector queues in time-order.

20. The non-transitory computer-readable medium of claim 11, wherein the instructions, when executed by the processor, further cause the processor to perform the step of:

deserializing the partial search results obtained from the plurality of data sources.

21. A system, comprising:

one or more worker nodes, wherein each worker node includes a processor and a memory that stores respective instructions, and, when each processor executes the respective instructions, the one or more worker nodes are configured to;

receive partial search results from a plurality of data sources, wherein each of the partial search results satisfies a portion of a search query, the search query received by a data intake and query system,

wherein at least one first partial search result of the partial search results is received from a subset of internal data sources of the data intake and query system, wherein the at least one first partial search result comprises one or more first events in a first format, each first event corresponding to at least one second event stored in the subset of internal data sources, wherein each second event includes raw machine data associated with a timestamp and reflects activity within an information technology infrastructure, and

wherein at least one second partial search result of the partial search results is received from one or more external data sources apart from the data intake and query system, wherein the at least one second partial search result includes data in a second format that is different from the first format;

transform the data of the at least one second partial search result into the first format to produce a commonly formatted partial search result;

identify at least a first portion of the commonly formatted partial search result that is not time-stamped;

determine a timestamp for the at least a first portion of the commonly formatted partial search result that is not time-stamped; and

process the at least a first portion of the commonly formatted partial search result that is not time-stamped based on the determined timestamp to generate time-ordered chunks of the commonly formatted partial search result.

22. The system of claim 21, wherein the one or more worker nodes comprise at least two worker nodes, and the at least two worker nodes are further configured to stream the time-ordered chunks of the commonly formatted partial search result in parallel to a data fabric service master associated with the data intake and query system.

23. The system of claim 22, wherein the at least two worker nodes are configured to receive the partial search results, transform the data of the at least one second partial search result, identify the at least a first portion of the commonly formatted partial search result that is not time-stamped, and process the at least a first portion of the commonly formatted partial search result that is not time-stamped.

24. The system of claim 23, wherein each worker node included in the at least two worker nodes comprises a machine that is an agent of the data intake and query system and is configured to process data collected from internal data sources and external data sources.

25. The system of claim 22, wherein the one or more worker nodes are configured to determine a timestamp by assigning a different timestamp to each record of the at least a first portion of the commonly formatted partial search result that is not time-stamped, the assigned timestamp indicating a time associated with an ingestion time at which the respective record was ingested by a worker node.

26. The system of claim 25, wherein the one or more worker nodes are further configured to process the at least a first portion of the commonly formatted result that is not time-stamped by analyzing the different timestamps associated with the records of the at least a first portion of the commonly formatted partial search result that is not time-stamped to produce at least a portion of the time-ordered chunks of the commonly formatted partial search result.

27. The system of claim 22, wherein the one or more worker nodes are configured to process the at least a first portion of the commonly formatted partial search result that is not time-stamped by analyzing timestamps associated with the at least a first portion of the commonly formatted partial search result to produce at least a portion of the time-ordered chunks of the commonly formatted partial search result.

28. The system of claim 22, further comprising at least one search head that is configured to:

receive the time-ordered chunks of the commonly formatted partial search result streamed in parallel from the at least two worker nodes;

perform one or more deserialization operations on the time-ordered chunks of the commonly formatted partial search result to generate deserialized time-ordered chunks of the commonly formatted partial search result; and

add the deserialized time-ordered chunks of the commonly formatted partial search result to one or more corresponding collector queues.

29. The system of claim 28, wherein the at least one search head is further configured to collate the time-ordered chunks of the commonly formatted partial search result across the one or more collector queues in time-order.

30. The system of claim 21, wherein the one or more worker nodes are further configured to deserialize the partial search results received from the plurality of data sources.