PROCESSING A SYSTEM SEARCH REQUEST ACROSS DISPARATE DATA COLLECTION SYSTEMS

• US 20140330815A1
• Filed: 05/01/2014
• Published: 11/06/2014
• Est. Priority Date: 05/03/2013
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Bitincka, Ledion, Zhang, Steve, Stojanovski, Igor, Sorkin, Stephen
• Granted Patent: US 10,049,160 B2
• Time in Patent Office: Days
• US Class Current: 707/722
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/953 Querying, e.g. by the use of web search engines

Abstract

A search request received at a computer of a search support system is processed by analyzing the received search request to identify request parameters and connecting to a system index of the search support system that is referenced in the request parameters. An external result provider (ERP) process is initiated that establishes communication between the search support system and a data source external to the search support system, for a virtual index referenced in the request parameters. Thus, the ERP process provides an interface between the search support system and external data sources, such as by third parties. The ERP process can operate in a streaming mode (providing real-time search results with minimal processing) and/or a reporting mode (providing results with a greater delay and processing extent) and can switch between modes. The search request results are received from the connected system indexes and the referenced virtual indexes.

Claims

1. A method, comprising:

automatically identifying one or more secondary data collection systems associated with a search query, the search query formatted for searching a native data collection system, the one or more secondary data collection systems different that the native data collection system;

converting at least a portion of the search query to a secondary search query compatible with at least one of the one or more secondary data collection systems;

sending the secondary search query to the at least one of the one or more secondary data collection systems;

in response to sending the secondary search query, receiving search results from the at least one of the one or more secondary data collection systems;

processing the received search results for display.

2. The method as recited in claim 1 wherein at least a portion of the search query is fulfilled by a search head.

3. The method as recited in claim 1, further comprising:

identifying at least a second portion of the search query that is associated with the native data collection system;

sending the second portion of the search query to the native data collection system;

receiving search results from the native data collection system.

4. The method as recited in claim 1, further comprising:

identifying at least a second portion of the search query that is associated with the native data collection system;

sending the second portion of the search query to the native data collection system;

receiving search results from the native data collection system;

integrating the search results received from the at least one of the one or more secondary data collection systems with the search results received from the native data collection system.

5. The method as recited in claim 1, further comprising:

caching the search results received from the at least one of the one or more secondary data collection systems;

in response to receiving a third search request that contains search parameters that can be satisfied by a particular set of cached search results, returning the particular set of cached search results.

6. The method as recited in claim 1, further comprising:

spawning a process to handle search query communications with the at least one of the one or more secondary data collection systems.

7. The method as recited in claim 1 wherein the at least one of the one or more secondary data collection systems comprises a Hadoop system.

8. The method as recited in claim 1 wherein the at least one of the one or more secondary data collection systems comprises a Cassandra system.

9. The method as recited in claim 1 wherein the at least one of the one or more secondary data collection systems comprises a MongoDB system.

10. The method as recited in claim 1 wherein the at least one of the one or more secondary data collection systems comprises a relational database management system or a structured query language system.

11. An apparatus, comprising:

a subsystem, implemented at least partially in hardware, that automatically identifies one or more secondary data collection systems associated with a search query, the search query formatted for searching a native data collection system, the one or more secondary data collection systems different that the native data collection system;

a subsystem, implemented at least partially in hardware, that converts at least a portion of the search query to a secondary search query compatible with at least one of the one or more secondary data collection systems;

a subsystem, implemented at least partially in hardware, that sends the secondary search query to the at least one of the one or more secondary data collection systems;

a subsystem, implemented at least partially in hardware, that, in response to sending the secondary search query, receives search results from the at least one of the one or more secondary data collection systems;

a subsystem, implemented at least partially in hardware, that processes the received search results for display.

12. The apparatus as recited in claim 11 wherein at least a portion of the search query is fulfilled by a search head.

13. The apparatus as recited in claim 11, further comprising:

a subsystem, implemented at least partially in hardware, that identifies at least a second portion of the search query that is associated with the native data collection system;

a subsystem, implemented at least partially in hardware, that sends the second portion of the search query to the native data collection system;

a subsystem, implemented at least partially in hardware, that receives search results from the native data collection system.

14. The apparatus as recited in claim 11, further comprising:

a subsystem, implemented at least partially in hardware, that identifies at least a second portion of the search query that is associated with the native data collection system;

a subsystem, implemented at least partially in hardware, that sends the second portion of the search query to the native data collection system;

a subsystem, implemented at least partially in hardware, that receives search results from the native data collection system;

a subsystem, implemented at least partially in hardware, that integrates the search results received from the at least one of the one or more secondary data collection systems with the search results received from the native data collection system.

15. The apparatus as recited in claim 11, further comprising:

a subsystem, implemented at least partially in hardware, that caches the search results received from the at least one of the one or more secondary data collection systems;

a subsystem, implemented at least partially in hardware, that, in response to receiving a third search request that contains search parameters that can be satisfied by a particular set of cached search results, returns the particular set of cached search results.

16. The apparatus as recited in claim 11, further comprising:

a subsystem, implemented at least partially in hardware, that spawns a process to handle search query communications with the at least one of the one or more secondary data collection systems.

17. The apparatus as recited in claim 11 wherein the at least one of the one or more secondary data collection systems comprises a Hadoop system.

18. The apparatus as recited in claim 11 wherein the at least one of the one or more secondary data collection systems comprises a Cassandra system.

19. The method as recited in claim 11 wherein the at least one of the one or more secondary data collection systems comprises a MongoDB system.

20. The apparatus as recited in claim 11 wherein the at least one of the one or more secondary data collection systems comprises a relational database management system or a structured query language system.

21. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein execution of the one or more sequences of instructions by one or more processors causes the one or more processors to perform:

automatically identifying one or more secondary data collection systems associated with a search query, the search query formatted for searching a native data collection system, the one or more secondary data collection systems different that the native data collection system;

converting at least a portion of the search query to a secondary search query compatible with at least one of the one or more secondary data collection systems;

sending the secondary search query to the at least one of the one or more secondary data collection systems;

in response to sending the secondary search query, receiving search results from the at least one of the one or more secondary data collection systems;

processing the received search results for display.

22. The non-transitory computer-readable medium as recited in claim 21 wherein at least a portion of the search query is fulfilled by a search head.

23. The non-transitory computer-readable medium as recited in claim 21, further comprising:

identifying at least a second portion of the search query that is associated with the native data collection system;

sending the second portion of the search query to the native data collection system;

receiving search results from the native data collection system.

24. The non-transitory computer-readable medium as recited in claim 21, further comprising:

identifying at least a second portion of the search query that is associated with the native data collection system;

sending the second portion of the search query to the native data collection system;

receiving search results from the native data collection system;

integrating the search results received from the at least one of the one or more secondary data collection systems with the search results received from the native data collection system.

25. The non-transitory computer-readable medium as recited in claim 21, further comprising:

caching the search results received from the at least one of the one or more secondary data collection systems;

in response to receiving a third search request that contains search parameters that can be satisfied by a particular set of cached search results, returning the particular set of cached search results.

26. The non-transitory computer-readable medium as recited in claim 21, further comprising:

spawning a process to handle search query communications with the at least one of the one or more secondary data collection systems.

27. The non-transitory computer-readable medium as recited in claim 21 wherein the at least one of the one or more secondary data collection systems comprises a Hadoop system.

28. The non-transitory computer-readable medium as recited in claim 21 wherein the at least one of the one or more secondary data collection systems comprises a Cassandra system.

29. The non-transitory computer-readable medium as recited in claim 21 wherein the at least one of the one or more secondary data collection systems comprises a MongoDB system.

30. The non-transitory computer-readable medium as recited in claim 21 wherein the at least one of the one or more secondary data collection systems comprises a relational database management system or a structured query language system.