Cloud deployment of a data fabric service system

• US 10,592,562 B2
• Filed: 10/31/2016
• Issued: 03/17/2020
• Est. Priority Date: 09/26/2016
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Pal, Sourav, Bhattacharjee, Arindam
• Primary Examiner(s): Khong, Alexander
• Application Number:
  US15/339,847
• Publication Number:
  US 20190163842A1
• Time in Patent Office: 1,233 Days
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/212 with details for data modelling support
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/252 between a Database Management System and a front-end application
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

The performance and flexibility of a data intake and query system having capabilities extended by a fabric service (DFS) system can be improved with deployment on a cloud computing platform. The DFS system can extend the capabilities of a data intake and query system by leveraging computing assets from anywhere in a big data ecosystem to collectively execute search queries on diverse data systems regardless of whether data stores are internal of the data intake and query system and/or external data stores that are communicatively coupled to the data intake and query system over a network.

Claims

1. A method comprising:

receiving, by a search service system in a shared computing resource environment, an indication of a search query;

determining, by the search service system, a type of distributed processing system to use for performing a search of a plurality of data sources based on a scope of the search query received, at least one data source of the plurality of data sources storing data as a plurality of time-indexed events associated with a time-stamp and including a portion of raw machine data;

dynamically instantiating instances of one or more components of the search service system in the shared computing resource environment using the determined type of distributed processing system based on the scope of the search query received;

receiving, by at least one instance of the instantiated instances of one or more components of the search service system, first partial search results from a first data storage system of the plurality of data sources;

aggregating, by the at least one instance, the first partial search results obtained from the first data storage system with second partial search results obtained from a second data storage system of the plurality of data sources; and

providing aggregated search results to a search head associated with the search service system.

2. The method of claim 1, wherein the search head causes output of the aggregated search results or data indicative of the aggregated search results by a display device to a user.

3. The method of claim 2, wherein the aggregate search results are displayed as a timeline visualization.

4. The method of claim 1, wherein the first partial search results include a plurality of time-indexed events including portions of raw machine data.

5. The method of claim 1, wherein the second partial search results comprise data in a structured format.

6. The method of claim 1, wherein the plurality of data sources includes a cloud storage system and an external data storage system, wherein the cloud storage system stores data as a plurality of time-indexed events including portions of raw machine data.

7. The method of claim 1, wherein the first partial search results are in a first format, the second partial search results are in a second format different than the first format, and the aggregate search results are in a specified format.

8. The method of claim 1, wherein the receiving is by the instances of the one or more components of the search service system.

9. The method of claim 1, wherein a single cluster of the distributed processing system is used for all search operations.

10. The method of claim 1, wherein a plurality of clusters of the distributed processing system of differing scale types are used for differing types of search requests.

11. The method of claim 10, wherein the plurality of clusters of the distributed processing system include a first cluster type used for search aggregation processing involving full search of the plurality of data sources and a second cluster type used for partial aggregation processing of partial search results from different data sources.

12. The method of claim 11, wherein a scale of a cluster for the first cluster type or the second cluster type is set for each search query by a user.

13. The method of claim 11, wherein a scale of a cluster for the first cluster type or the second cluster type is based, at least in part, on a role quota.

14. The method of claim 13, wherein the scale of the cluster is based, at least in part, on information regarding a user that submitted the search query.

15. The method of claim 1, wherein dynamically configuring the instances of the one or more components of the search service system comprises configuring the search head to control processing of the search query.

16. The method of claim 1, wherein dynamically configuring the instances of the one or more components of the search service system comprises configuring a cloud worker node to obtain the first partial search results.

17. The method of claim 1, wherein the aggregated search results are computed and reported at scale to the search head.

18. The method of claim 17, further comprising reducing the aggregated search results from the plurality of data sources at scale before providing the aggregated search results to the search head.

19. The method of claim 1, further comprising collecting metrics for a heuristic determination of spikes in requirements of the search service system for the search.

20. The method of claim 19, wherein the heuristic determination is accelerated through auto-scaling.

21. The method of claim 1, further comprising replicating applications of the search service system over a cluster of the distributed processing system to ensure that the applications are executed at scale.

22. The method of claim 1, establishing user-quota or role-quota levels based, at least in part, on synchronization between user management features of the search service system and access control features of the distributed processing system.

23. The method of claim 1, wherein the search query is input by a user and expressed in a pipelined search language.

24. The method of claim 1, further comprising metering an amount of time and computing resources utilized to complete the search query.

25. A search service system comprising:

a memory; and

a processing device in a shared computing resource environment and communicatively coupled with the memory, wherein the processing device is configured to;

receive an indication of a search query;

determine a type of distributed processing system to use for performing a search of a plurality of data sources based on a scope of the search query received, at least one data source of the plurality of data sources storing data as a plurality of time-indexed events associated with a time-stamp and including a portion of raw machine data;

dynamically instantiate instances of one or more components of the search service system in the shared computing resource environment using the determined type of distributed processing system based on the scope of the search query received;

receive, by at least one instance of the instantiated instances of one or more components of the search service system, first partial search results from a first data storage system of the plurality of data sources;

aggregating, by the at least one instance, the first partial search results obtained from the first data storage system with second partial search results obtained from a second data storage system of the plurality of data sources; and

provide aggregated search results to a search head.

26. The system of claim 25, wherein the aggregated search results include a plurality of time-indexed events including portions of raw machine data.

27. The system of claim 25, wherein the plurality of data sources includes a cloud storage system and an external data storage system, wherein the cloud storage system stores data as a plurality of time-indexed events including portions of raw machine data.

28. A non-transitory computer-readable medium of a search service system, the non-transitory computer-readable medium storing computer-executable instructions that, in response to execution by one or more processing devices in a shared computing resource environment, cause the one or more processing devices to perform operations comprising:

receiving an indication of a search query;

determining a type of distributed processing system to use for performing a search of a plurality of data sources based on a scope of the search query received, at least one data source of the plurality of data sources storing data as a plurality of time-indexed events associated with a time-stamp and including a portion of raw machine data;

dynamically instantiating instances of one or more components of the search service system in the shared computing resource environment using the determined type of distributed processing system based on the scope of the search query received;

receiving, by at least one instance of the instantiated instances of one or more components of the search service system, first partial search results from a first data storage system of the plurality of data sources;

aggregating, by the at least one instance, the first partial search results obtained from the first data storage system with second partial search results obtained from a second data storage system of the plurality of data sources; and

providing aggregated search results to a search head.

29. The non-transitory computer-readable medium of claim 28, wherein the aggregated search results include a plurality of time-indexed events including portions of raw machine data.

30. The non-transitory computer-readable medium of claim 28, wherein the plurality of data sources includes a cloud storage system and an external data storage system, wherein the cloud storage system stores data as a plurality of time-indexed events including portions of raw machine data.