Optimizing resource allocation for projects executing in a cloud-based environment

• US 10,362,041 B2
• Filed: 01/16/2018
• Issued: 07/23/2019
• Est. Priority Date: 08/12/2011
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Das, Robin Kumar, Ago, Ledio, Shanaghy, Declan Gerard, Gupta, Gaurav
• Primary Examiner(s): To, Baoquoc N
• Application Number:
  US15/872,837
• Publication Number:
  US 20180145989A1
• Time in Patent Office: 553 Days
• Field of Search: 707711, 707741
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/134 Distributed indices
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 16/21 Design, administration or maintenance of databases
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/901 Indexing; Data structures therefor; Storage structures for retrieval from the web G06F16/951
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 16/986 Document structures and storage, e.g. HTML extensions
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 9/4887 involving deadlines, e.g. rate based, periodic
• CPC Class Codes: G06F 9/5088 involving task migration
• CPC Class Codes: G06Q 20/145 Payments according to the detected use or quantity
• CPC Class Codes: H04L 12/1435 volume-based
• CPC Class Codes: H04L 63/10 for controlling access to devices or network resources
• CPC Class Codes: H04L 63/105 Multiple levels of security
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Embodiments are directed towards a system and method for a cloud-based front end that may abstract and enable access to the underlying cloud-hosted elements and objects that may be part of a multi-tenant application, such as a search application. Search objects may be employed to access indexed objects. An amount of indexed data accessible to a user may be based on an index storage limit selected by the user, such that data that exceeds the index storage limit may continue to be indexed. Also, one or more projects can be elastically scaled for a user to provide resources that may meet the specific needs of each project.

Claims

1. A method comprising:

identifying a resource allocation associated with a project, wherein the resource allocation identifies a plurality of computing resources assigned to the project for indexing data corresponding to the project, and the project is associated with a dynamically scalable index store for storing the data;

determining based on one or more performance rules associated with the project that the resource allocation is suboptimal;

determining, via a processor, an optimal resource allocation for the project based on one or more characteristics of the project; and

adjusting the plurality of computing resources based on the optimal resource allocation.

2. The method of claim 1, wherein the project is associated with an index store for storing the data corresponding to the project, the index store is accessible by at least one indexer server, and the at least one indexer server is associated with at least one virtual machine.

3. The method of claim 1, wherein the one or more performance rules specifies an optimal speed for indexing data corresponding to the project.

4. The method of claim 1, wherein the one or more performance rules includes a rule-based trigger that is triggered when a speed for indexing the data corresponding to the project falls below a threshold.

5. The method of claim 1, wherein the one or more performance rules indicates a scheduled trigger for changing the plurality of computing resources assigned to the project.

6. The method of claim 1, wherein determining the optimal resource allocation comprises determining a change in a first subset of the plurality of computing resources associated with a first processing dimension based on a performance characteristic of the project.

7. The method of claim 1, wherein determining the optimal resource allocation comprises:

determining a first change in a first subset of the plurality of computing resources associated with a first processing dimension based on a performance characteristic of the project; and

determining, based on the first change, a second change in a second subset of the plurality of computing resources associated with a second processing dimension.

8. The method of claim 1, wherein adjusting the plurality of computing resources comprises placing a request to adjust the plurality of computing resources in a request queue until the request can be fulfilled.

9. The method of claim 1, further comprising:

determining an impact of the optimal resource allocation on an overall performance of an indexing system that indexes data corresponding to a plurality of projects including the project; and

delaying a request to adjust the plurality of computing resources until the request based on the impact.

10. The method of claim 1, further comprising causing a user interface to display at least one user interface control to adjust at least one parameter related to the project, and determining the optimal resource allocation in response to input from the at least one user interface control.

31. The method of claim 1, wherein the one or more performance rules are based on at least one of a scheduled time, a pattern prediction that indicates a potential change in utilization requirements, and a determination that a performance metric threshold has been exceeded.

32. The method of claim 1, wherein adjusting the plurality of computing resources comprises adjusting an allocation associated with the dynamically scalable index store.

11. A computer readable storing medium comprising instructions that, when executed by a processor, cause the processor to:

identify a resource allocation associated with a project, wherein the resource allocation identifies a plurality of computing resources assigned to the project for indexing data corresponding to the project, and the project is associated with a dynamically scalable index store for storing the data;

determine based on one or more performance rules associated with the project that the resource allocation is suboptimal;

determine an optimal resource allocation for the project based on one or more characteristics of the project; and

adjust the plurality of computing resources based on the optimal resource allocation.

12. The computer readable storing medium of claim 11, wherein the project is associated with an index store for storing the data corresponding to the project, the index store is accessible by at least one indexer server, and the at least one indexer server is associated with at least one virtual machine.

13. The computer readable storing medium of claim 11, wherein the one or more performance rules specifies an optimal speed for indexing data corresponding to the project.

14. The computer readable storing medium of claim 11, wherein the one or more performance rules includes a rule-based trigger that is triggered when a speed for indexing the data corresponding to the project falls below a threshold.

15. The computer readable storing medium of claim 11, wherein the one or more performance rules indicates a scheduled trigger for changing the plurality of computing resources assigned to the project.

16. The computer readable storing medium of claim 11, wherein determining the optimal resource allocation comprises determining a change in a first subset of the plurality of computing resources associated with a first processing dimension based on a performance characteristic of the project.

17. The computer readable storing medium of claim 11, wherein determining the optimal resource allocation comprises:

determining a first change in a first subset of the plurality of computing resources associated with a first processing dimension based on a performance characteristic of the project; and

determining, based on the first change, a second change in a second subset of the plurality of computing resources associated with a second processing dimension.

18. The computer readable storing medium of claim 11, wherein adjusting the plurality of computing resources comprises placing a request to adjust the plurality of computing resources in a request queue until the request can be fulfilled.

19. The computer readable storing medium of claim 11, wherein the instructions, when executed by the processor, further cause the processor to:

determine an impact of the optimal resource allocation on an overall performance of an indexing system that indexes data corresponding to a plurality of projects including the project; and

delay a request to adjust the plurality of computing resources until the request based on the impact.

20. The computer readable storing medium of claim 11, wherein the instructions, when executed by the processor, further cause the processor to cause a user interface to display at least one user interface control to adjust at least one parameter related to the project, and determining the optimal resource allocation in response to input from the at least one user interface control.

21. A computer system comprising:

a memory storing instructions; and

a processor for executing the instructions to perform the steps of;

identifying a resource allocation associated with a project, wherein the resource allocation identifies a plurality of computing resources assigned to the project for indexing data corresponding to the project, and the project is associated with a dynamically scalable index store for storing the data;

determining based on one or more performance rules associated with the project that the resource allocation is suboptimal;

determining an optimal resource allocation for the project based on one or more characteristics of the project; and

adjusting the plurality of computing resources based on the optimal resource allocation.

22. The computer system of claim 21, wherein the project is associated with an index store for storing the data corresponding to the project, the index store is accessible by at least one indexer server, and the at least one indexer server is associated with at least one virtual machine.

23. The computer system of claim 21, wherein the one or more performance rules specifies an optimal speed for indexing data corresponding to the project.

24. The computer system of claim 21, wherein the one or more performance rules includes a rule-based trigger that is triggered when a speed for indexing the data corresponding to the project falls below a threshold.

25. The computer system of claim 21, wherein the one or more performance rules indicates a scheduled trigger for changing the plurality of computing resources assigned to the project.

26. The computer system of claim 21, wherein determining the optimal resource allocation comprises determining a change in a first subset of the plurality of computing resources associated with a first processing dimension based on a performance characteristic of the project.

27. The computer system of claim 21, wherein determining the optimal resource allocation comprises:

determining a first change in a first subset of the plurality of computing resources associated with a first processing dimension based on a performance characteristic of the project; and

determining, based on the first change, a second change in a second subset of the plurality of computing resources associated with a second processing dimension.

28. The computer system of claim 21, wherein adjusting the plurality of computing resources comprises placing a request to adjust the plurality of computing resources in a request queue until the request can be fulfilled.

29. The computer system of claim 21, wherein the processor further performs the steps of:

determining an impact of the optimal resource allocation on an overall performance of an indexing system that indexes data corresponding to a plurality of projects including the project; and

delaying a request to adjust the plurality of computing resources until the request based on the impact.

30. The computer system of claim 21, wherein the processor further performs the step of causing a user interface to display at least one user interface control to adjust at least one parameter related to the project, and determining the optimal resource allocation in response to input from the at least one user interface control.