Data fabric services

• US 10,474,723 B2
• Filed: 10/31/2016
• Issued: 11/12/2019
• Est. Priority Date: 09/26/2016
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Hodge, James Alasdair Robert, Pal, Sourav, Bhattacharjee, Arindam, Ahamed, Mustafa
• Primary Examiner(s): Khong, Alexander
• Application Number:
  US15/339,825
• Publication Number:
  US 20190163796A1
• Time in Patent Office: 1,107 Days
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/212 with details for data modelling support
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/252 between a Database Management System and a front-end application
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

The disclosed embodiments also include monitoring and metering services of the data fabric service (DFS) system. Specifically, these services can include techniques for monitoring and metering metrics of the DFS system. The metrics are standards for measuring use or misuse of the DFS system. Examples of the metrics include data or components of the DFS system. For example, a metric can include data stored or communicated by the DFS system or components of the DFS system that are used or reserved for exclusive use by customers. The metrics can be measured with respect to time or computing resources (e.g., CPU utilization, memory usage) of the DFS system. For example, a DFS service can include metering the usage of particular worker nodes by a customer over a threshold period of time.

Claims

1. A method, comprising:

monitoring, by a data intake and query system, one or more metrics of use of one or more components of a search service system;

receiving, by the data intake and query system, a search query associated with a user of the search service system, wherein the search query causes the data intake and query system to execute one or more searches of one or more internal data stores storing data as a plurality of time-indexed events including portions of raw machine data, to communicate with a search service to perform one or more searches of one or more external data stores storing data in a format that is different than the format of the one or more internal data stores, and to receive search results based at least in part on the one or more searches of the one or more internal data stores and the one or more searches of the one or more external data stores;

determining, by the data intake and query system, a status of the user of the search service system;

determining, by the data intake and query system, that the one or more metrics of use satisfy a threshold metric of use based at least in part on the search query and the status of the user; and

causing, by the data intake and query system, a display to display an indication to the user that the threshold metric of use is satisfied.

2. The method of claim 1, wherein the one or more components of the search service system comprise a worker node configured to process at least one of internal partial search results from the one or more internal data stores or external partial search results from the one or more external data stores to generate the search results.

3. The method of claim 2, wherein the worker node is configured to process the internal partial search results, and wherein the internal partial search results comprise a plurality of time-indexed events including portions of raw machine data.

4. The method of claim 2, wherein the one or more metrics of use comprise a quantity of worker nodes that run at a given time and the threshold metric of use comprises a threshold quantity of worker nodes that can run at the given time.

5. The method of claim 2, wherein the threshold metric of use comprises a predetermined size of the worker node.

6. The method of claim 5, wherein the predetermined size is based at least in part on a central processing unit and memory of the worker node.

7. The method of claim 2, wherein the one or more metrics of use comprise a use time of the worker node.

8. The method of claim 7, wherein the threshold metric of use comprises a predetermined use time of the worker node during a period of time.

9. The method of claim 1, wherein the search query is input by the user and expressed in a pipelined search language.

10. The method of claim 1, wherein the one or more metrics of use comprise data processed by the search service system and the threshold metric of use comprises a predetermined volume of data processed by the search service system during a period of time.

11. The method of claim 1, wherein the one or more metrics of use comprise data stored by the search service system and the threshold metric of use comprises a predetermined volume of data stored by the search service system during a period of time.

12. The method of claim 1, wherein the one or more metrics of use comprise at least one of computer utilization or memory usage.

13. The method of claim 1, wherein the one or more metrics of use comprise a quantity of search queries received and the threshold metric of use comprises a threshold quantity of search queries.

14. The method of claim 1, wherein the one or more metrics of use comprise a use time of the one or more components of the search service system and the threshold metric of use comprises a threshold amount of use time of the one or more components of the search service system.

15. The method of claim 1, wherein the one or more metrics of use comprise a computing resource of the search service system used and the threshold metric of use comprises a threshold quantity of computing resources of the search service system.

16. The method of claim 1, further comprising based at least in part on the determining that the one or more metrics of use satisfy a threshold metric of use, indicating to the user an additional criteria to be satisfied prior to processing the search query.

17. The method of claim 16, wherein the additional criteria comprises an additional charge.

18. The method of claim 1, further comprising preventing access to the one or more components of the search service system.

19. The method of claim 1, further comprising limiting access to the one or more components of the search service system.

20. The method of claim 1, further comprising enabling the user to complete the search query and denying access to the search service system for additional search queries.

21. The method of claim 1, further comprising causing a display device to display the one or more metrics of use of the search service system.

22. The method of claim 1, wherein the one or more metrics of use of the one or more components are based at least in part on an identity of the user.

23. The method of claim 1, wherein the threshold metric of use is based at least in part on a subscription of the user.

24. The method of claim 23, wherein the determining that the one or more metrics of use satisfy the threshold metric of use is further based at least in part on a determination that a subscription volume of the subscription has been satisfied.

25. The method of claim 23, wherein the determining that the one or more metrics of use satisfy the threshold metric of use is based at least in part on a request to access a function not included in the subscription.

26. The method of claim 23, wherein the determining that the one or more metrics of use satisfy the threshold metric of use is based at least in part on a request to perform a type of search not included in the subscription.

27. The method of claim 1, wherein the monitoring the one or more metrics of use of the one or more components is based at least in part on a subscription status of the user.

28. The method of claim 27, wherein the determining that the one or more metrics of use satisfy the threshold metric of use is based at least in part on a determination that the user does not have a subscription.

29. A system comprising:

a memory; and

a processing device coupled with the memory to;

monitor one or more metrics of use of one or more components of a search service system;

receive a search query associated with a user of the search service system, wherein the search query causes a data intake and query system to execute one or more searches of one or more internal data stores storing data as a plurality of time-indexed events including portions of raw machine data, to communicate with a search service to perform one or more searches of one or more external data stores storing data in a format that is different than the format of the one or more internal data stores, and to receive search results based at least in part on the one or more searches of the one or more internal data stores and the one or more searches of the one or more external data stores;

determine a status of the user of the search service system;

determine that the one or more metrics of use satisfy a threshold metric of use based at least in part on the search query and the status of the user; and

cause a display to display an indication to the user that the threshold metric of use is satisfied.

30. A non-transitory computer-readable medium encoding instructions thereon that, in response to execution by one or more processing devices, cause the one or more processing devices to perform operations comprising:

monitoring one or more metrics of use of one or more components of a search service system;

receiving a search query associated with a user of the search service system, wherein the search query causes a data intake and query system to execute one or more searches of one or more internal data stores storing data as a plurality of time-indexed events including portions of raw machine data, to communicate with a search service to perform one or more searches of one or more external data stores storing data in a format that is different than the format of the one or more internal data stores, and to receive search results based at least in part on the one or more searches of the one or more internal data stores and the one or more searches of the one or more external data stores;

determining a status of the user of the search service system;

determining that the one or more metrics of use satisfy a threshold metric of use based at least in part on the search query and the status of the user; and

causing a display to display an indication to the user that the threshold metric of use is satisfied.