Data volume scaling for storing indexed data

• US 9,356,934 B2
• Filed: 04/17/2015
• Issued: 05/31/2016
• Est. Priority Date: 08/12/2011
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Das, Robin Kumar, Ago, Ledio, Shanaghy, Declan Gerard, Gupta, Gaurav
• Primary Examiner(s): To, Baoquoc N
• Application Number:
  US14/690,249
• Publication Number:
  US 20150220585A1
• Time in Patent Office: 410 Days
• Field of Search: 707/661, 707/E17.01, 709/226
• US Class Current: 1/1
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/134 Distributed indices
• CPC Class Codes: G06F 16/162 Delete operations erasing in storage systems G06F3/0652
• CPC Class Codes: G06F 16/21 Design, administration or maintenance of databases
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/901 Indexing; Data structures therefor; Storage structures for retrieval from the web G06F16/951
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 16/986 Document structures and storage, e.g. HTML extensions
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 9/4887 involving deadlines, e.g. rate based, periodic
• CPC Class Codes: G06F 9/5088 involving task migration
• CPC Class Codes: G06Q 20/145 Payments according to the detected use or quantity
• CPC Class Codes: H04L 12/1435 volume-based
• CPC Class Codes: H04L 63/10 for controlling access to devices or network resources
• CPC Class Codes: H04L 63/105 Multiple levels of security
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

Embodiments are directed towards a system and method for a cloud-based front end that may abstract and enable access to the underlying cloud-hosted elements and objects that may be part of a multi-tenant application, such as a search application. Search objects may be employed to access indexed objects. An amount of indexed data accessible to a user may be based on an index storage limit selected by the user, such that data that exceeds the index storage limit may continue to be indexed. Also, one or more projects can be elastically scaled for a user to provide resources that may meet the specific needs of each project.

Claims

1. A method, comprising:

indexing received data;

while indexing the received data;

storing the indexed data in at least one data storage device;

determining whether an amount of the indexed data stored for a particular project in the at least one data storage device has reached a data volume limit associated with the particular project, the particular project among a plurality of projects, each project in the plurality of projects having an associated data volume limit; and

in response to a determination that the amount of stored indexed data for the particular project has reached the associated data volume limit, continuing storing indexed data for the project in at least one data storage area that is not accessible by users of the particular project;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, wherein the associated data volume limit is purchased by a user.

3. The method of claim 1, further comprising:

in response to the determination that the amount of indexed data for the particular project has reached the associated data volume limit, automatically stopping indexing of received data for the project.

4. The method of claim 1, further comprising enabling a user to access the new indexed data for the project that is stored in the at least one data storage area that is not accessible by users of the particular project when additional data storage to increase the associated data volume limit is purchased.

5. The method of claim 1, further comprising:

determining an amount of new indexed data for the project to copy from the at least one data storage area that is not accessible by users of the particular project to the at least one data storage device based on an increase to the associated data volume limit;

copying the amount of new indexed data from the at least one data storage area that is not accessible by users of the particular project to the at least one data storage device.

6. The method of claim 1, wherein the received data is raw data.

7. The method of claim 1, wherein the received data is unstructured data.

8. The method of claim 1, wherein the received data is machine data.

9. The method of claim 1, wherein the received data is log data.

10. The method of claim 1, wherein indexing the received data further comprises:

including portions of the received data in time-stamped, searchable events.

11. The method of claim 1, wherein the at least one data storage device is used in providing a cloud service.

12. The method of claim 1, wherein the at least one data storage device is used in providing a multi-tenant cloud service.

13. The method of claim 1, wherein the at least one data storage device is used in providing a hosted cloud service.

14. The method of claim 1, wherein the project is associated with a multi-user account.

15. The method of claim 1, further comprising providing role-based access controls for the particular project.

16. The method of claim 1, further comprising maintaining a record of whether a particular user has a right to search indexed data stored for the particular project.

17. The method of claim 1, further comprising maintaining a record of whether a particular user has a right to provide data to be indexed and stored for the particular project.

18. The method of claim 1, further comprising providing secure access to the particular project.

19. The method of claim 1, further comprising notifying a user via email that the associated data volume limit has been reached.

20. The method of claim 1, further comprising enabling the stored indexed data to be searched.

21. The method of claim 1, further comprising making the stored indexed data available for distributed search.

22. The method of claim 1, wherein the data volume limit is based on a pre-indexing amount of data.

23. The method of claim 1, wherein the data volume limit is based on a post-indexing amount of data.

24. The method of claim 1, wherein the data volume limit is based on a pre-indexing amount of data and a post-indexing amount of data.

25. One or more non-transitory computer-readable storage media, storing one or more sequences of instructions, which when executed by one or more processors cause performance of:

indexing received data;

while indexing the received data;

storing the indexed data in at least one data storage device;

determining whether an amount of indexed data stored for a particular project in the at least one data storage device has reached a data volume limit associated with the particular project, the particular project among a plurality of projects, each project in the plurality of projects having an associated data volume limit; and

in response to a determination that the amount of stored indexed data for the particular project has reached the associated data volume limit, continuing storing indexed data for the project in at least one data storage area that is not accessible by users of the particular project.

26. An apparatus, comprising:

an indexer, implemented at least partially in hardware, that indexes received data;

a storage subsystem, implemented at least partially in hardware, that, while the indexer indexes the received data;

stores the indexed data in at least one data storage device;

determines whether an amount of indexed data stored for a particular project in the at least one data storage device has reached a data volume limit associated with the particular project, the particular project among a plurality of projects, each project in the plurality of projects having an associated data volume limit; and

in response to a determination that the amount of stored indexed data for the particular project has reached the associated data volume limit, continues to store indexed data for the project in at least one data storage area that is not accessible by users of the particular project.