Processing a system search request including external data sources

• US 9,514,189 B2
• Filed: 07/31/2014
• Issued: 12/06/2016
• Est. Priority Date: 05/03/2013
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Bitincka, Ledion, Zhang, Steve, Stojanovski, Igor, Sorkin, Stephen
• Primary Examiner(s): AL HASHEMI, SANA A
• Application Number:
  US14/449,144
• Publication Number:
  US 20140344256A1
• Time in Patent Office: 859 Days
• Field of Search: 707/622, 707/623, 707/706, 707/722, 707/758
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/953 Querying, e.g. by the use of web search engines

Abstract

A search request received at a computer of a search support system is processed by analyzing the received search request to identify request parameters and connecting to a system index of the search support system that is referenced in the request parameters. An external result provider (ERP) process is initiated that establishes communication between the search support system and a data source external to the search support system, for a virtual index referenced in the request parameters. Thus, the ERP process provides an interface between the search support system and external data sources, such as by third parties. The ERP process can operate in a streaming mode (providing real-time search results with minimal processing) and/or a reporting mode (providing results with a greater delay and processing extent) and can switch between modes. The search request results are received from the connected system indexes and the referenced virtual indexes.

Claims

1. A method, comprising:

communicating, by a first device, a search request to an external data system, the search request based at least in part on a received search request;

receiving, by the first device, streamed raw data obtained from raw data accessible by the external data system and that the external data system has streamed to the first device, the streamed raw data fulfills at least a portion of the search request, the first device processing the streamed raw data to create results for the received search request as the streamed raw data is received from the external system and providing the results to a second device;

as the first device processes the streamed raw data, receiving, by the first device, at least a partial search result from the external data system that the external data system generated in response to receiving the search request by at least partially processing the raw data accessible to the external data system in accordance with the search request, the first device providing the second device with search results for the received search request that are based at least partially on the received at least a partial search result.

2. The method of claim 1, wherein the search request indicates some but not all of processing needed to provide the search results for the received search request.

3. The method of claim 1, wherein the external data system is referenced in the received search request.

4. The method of claim 1, wherein the search request is the same as the received search request.

5. The method of claim 1, wherein processing the streamed raw data is stopped upon receiving the at least the partial search result from the external data system.

6. The method of claim 1, wherein processing the streamed raw data is stopped upon determining that providing the search results to the second device is more efficient than processing the streamed raw data.

7. The method of claim 1, wherein the streamed raw data includes unprocessed data from the external data system as the unprocessed data is retrieved by the external data system.

8. The method of claim 1, wherein the search results for the search request comprises at least one of:

statistics, aggregates, or filtered results.

9. The method of claim 1, wherein the processing of the streamed raw data includes applying a late binding schema to the streamed raw data.

10. The method as in claim 1, wherein the external data system is a HADOOP data system.

11. The method as in claim 1, wherein data is retrieved from multiple external data systems in response to the search request.

12. The method as in claim 1, further comprising:

determining whether to cache at least a portion of the search results for the search request such that the cached results are usable as a response in a subsequent received search request.

13. The method as in claim 1, further comprising:

determining whether cached results produced by a previous received search request are available to help satisfy at least a portion of the search request.

14. An apparatus, comprising:

a subsystem, at a first device, implemented at least partially in hardware, that communicates a search request to an external data system, the search request based at least in part on a received search request;

a subsystem, at the first device, implemented at least partially in hardware, that receives streamed raw data obtained from raw data accessible by the external data system and that the external data system has streamed to the first device, the streamed raw data fulfills at least a portion of the search request, the first device processes the streamed raw data to create results for the received search request as the streamed raw data is received from the external system and providing the results to a second device;

a subsystem, at the first device, implemented at least partially in hardware, that, as the first device processes the streamed raw data, receives at least a partial search result from the external data system that the external data system generated in response to receiving the search request by at least partially processing the raw data accessible to the external data system in accordance with the search request, the first device provides the second device with search results for the received search request that are based at least partially on the received at least a partial search result.

15. The apparatus of claim 14, wherein the search request indicates some but not all of processing needed to provide the search results for the received search request.

16. The apparatus of claim 14, wherein the external data system is referenced in the received search request.

17. The apparatus of claim 14, wherein the search request is the same as the received search request.

18. The apparatus of claim 14, wherein processing the streamed raw data is stopped upon receiving the at least the partial search result from the external data system.

19. The apparatus of claim 14, wherein processing the streamed raw data is stopped upon a determination that providing the search results to the second device is more efficient than processing the streamed raw data.

20. The apparatus of claim 14, wherein the streamed raw data includes unprocessed data from the external data system as the unprocessed data is retrieved by the external data system.

21. The apparatus of claim 14, wherein the search results for the search request comprises at least one of:

statistics, aggregates, or filtered results.

22. The apparatus of claim 14, wherein the processing of the streamed raw data includes applying a late binding schema to the streamed raw data.

23. The apparatus as in claim 14, wherein the external data system is a HADOOP data system.

24. The apparatus as in claim 14, wherein data is retrieved from multiple external data systems in response to the search request.

25. The apparatus as in claim 14, further comprising:

a subsystem, at the first device, implemented at least partially in hardware, that determines whether to cache at least a portion of the search results for the search request such that the cached results are usable as a response in a subsequent received search request.

26. The apparatus as in claim 14, further comprising:

a subsystem, at the first device, implemented at least partially in hardware, that determines whether cached results produced by a previous received search request are available to help satisfy at least a portion of the search request.

27. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to perform:

communicating, by a first device, a search request to an external data system, the search request based at least in part on a received search request;

receiving, by the first device, streamed raw data obtained from raw data accessible by the external data system and that the external data system has streamed to the first device, the streamed raw data fulfills at least a portion of the search request, the first device processing the streamed raw data to create results for the received search request as the streamed raw data is received from the external system and providing the results to a second device;

as the first device processes the streamed raw data, receiving, by the first device, at least a partial search result from the external data system that the external data system generated in response to receiving the search request by at least partially processing the raw data accessible to the external data system in accordance with the search request, the first device providing the second device with search results for the received search request that are based at least partially on the received at least a partial search result.

28. The non-transitory computer-readable medium of claim 27, wherein the search request indicates some but not all of processing needed to provide the search results for the search request.

29. The non-transitory computer-readable medium of claim 27, wherein the external data system is referenced in the received search request.

30. The non-transitory computer-readable medium of claim 27, wherein the search request is the same as the received search request.

31. The non-transitory computer-readable medium of claim 27, wherein processing the streamed raw data is stopped upon receiving the at least the partial search result from the external data system.

32. The non-transitory computer-readable medium of claim 27, wherein processing the streamed raw data is stopped upon determining that providing the search results to the second device is more efficient than processing the streamed raw data.

33. The non-transitory computer-readable medium of claim 27, wherein the streamed raw data includes unprocessed data from the external data system as the unprocessed data is retrieved by the external data system.

34. The non-transitory computer-readable medium of claim 27, wherein the search results for the search request comprises at least one of:

statistics, aggregates, or filtered results.

35. The non-transitory computer-readable medium of claim 27, wherein the processing of the streamed raw data includes applying a late binding schema to the streamed raw data.

36. The non-transitory computer-readable medium as in claim 27, wherein the external data system is a HADOOP data system.

37. The non-transitory computer-readable medium as in claim 27, wherein data is retrieved from multiple external data systems in response to the search request.

38. The non-transitory computer-readable medium as in claim 27, further comprising:

determining whether to cache at least a portion of the search results for the search request such that the cached results are usable as a response in a subsequent received search request.

39. The non-transitory computer-readable medium as in claim 27, further comprising:

determining whether cached results produced by a previous received search request are available to help satisfy at least a portion of the search request.