SCALING AVAILABLE STORAGE BASED ON COUNTING GENERATED EVENTS

• US 20160306871A1
• Filed: 04/30/2015
• Published: 10/20/2016
• Est. Priority Date: 04/20/2015
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Shahbaz, Banipal, Chauhan, Vijay, Hazekamp, David
• Granted Patent: US 10,817,544 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/2477 Temporal data queries
• CPC Class Codes: G06F 16/287 Visualization; Browsing

Abstract

A data intake and query system measures an amount of raw data ingested by the system during defined periods of time. As used herein, ingesting raw data generally refers to receiving the raw data from one or more computing devices and processing the data for storage and searchability. Processing the data may include, for example, parsing the raw data into “events,” where each event includes a portion of the received data and is associated with a timestamp. Based on a calculated number of events generated by the system during one or more defined time periods, the system may calculate various metrics including, but not limited to, a number of events generated during a particular day, a number of events generated per day over a period of time, a maximum number of events generated in a day over a period of time, an average number of events generated per day, etc.

Claims

1. A method comprising:

receiving raw data from one or more devices;

generating a plurality events from the raw data by;

parsing the raw data into a plurality of events, each event of the plurality of events including a portion of the raw data;

determining a respective timestamp for each event of the plurality of events;

determining a number of events of the plurality of events that were generated during a defined time period;

comparing the number of events that were generated during the defined time period to an allocated event count;

in response to a determination that the number of events that were generated during the defined time period has reached an allocated event count, performing one or more actions to raw data received subsequent to the allocated event count being reached.

2. The method of claim 1, wherein the allocated event count specifies a maximum number of events generated during the defined time period.

3. The method of claim 1, wherein the allocated event count specifies a maximum average number of events generated during the defined time period.

4. The method of claim 1, wherein performing one or more actions to raw data received subsequent to the allocated event count being reached includes storing events generated from the raw data in a non-searchable index.

5. The method of claim 1, wherein performing one or more actions to raw data received subsequent to the allocated event count being reached includes storing events generated from the raw data in a non-searchable index;

enabling access to the events that are stored in the non-searchable index when an increase to the allocated event count is purchased.

6. The method of claim 1, wherein performing one or more actions to raw data received subsequent to the allocated event count being reached includes storing events generated from the raw data in a non-accessible index.

7. The method of claim 1, wherein performing one or more actions to raw data received subsequent to the allocated event count being reached includes automatically deleting events that exceed the allocated event count.

8. The method of claim 1, further comprising, in response to the determination that the number of events that were generated during the defined time period has reached the allocated event count, automatically increasing the allocated event count.

9. The method of claim 1, further comprising storing at least one event of the plurality of events in an index.

10. The method of claim 1, further comprising:

storing at least one event of the plurality of events in an index;

in response to the determination that the number of events that were generated during the defined time period has reached the allocated event count, ceasing to generate new events based on the raw data received from the one or more devices.

11. The method of claim 1, further comprising, in response to the determination that the number of events that were generated during the defined time period has reached the allocated event count, ceasing to accept raw data from the one or more devices.

12. The method of claim 1, further comprising, in response to the determination that the number of events that were generated during the defined time period has reached the allocated event count, generating an alert indicating that the allocated event count has been reached.

13. The method of claim 1, further comprising, in response to the determination that the number of events that were generated during the defined time period has reached the allocated event count, sending an alert message to a particular user.

14. The method of claim 1, wherein determining the number of events of the plurality of events that were generated during the defined time period includes determining that the number of events are associated with a particular user account of a plurality of user accounts.

15. The method of claim 1, wherein determining the number of events of the plurality of events that were generated during the defined time period includes determining that the number of events are associated with a particular project of a plurality of projects.

16. The method of claim 1, wherein parsing the raw data into a plurality of events further comprises determining event boundaries for the plurality of events.

17. The method of claim 1, wherein the plurality of events are searchable using a late-binding schema comprising one or more extraction rules for extracting values from the events.

18. The method of claim 1, wherein the defined time period corresponds to one or more days.

19. The method of claim 1, wherein the defined time period corresponds to one or more seconds.

20. The method of claim 1, further comprising calculating an average number of events that were generated over a plurality of time periods.

21. The method of claim 1, further comprising calculating a fee amount based on the number of events of the plurality of events that were generated during the defined time period.

22. The method of claim 1, wherein the one or more devices are managed by a managed security service provider (MSSP).

23. The method of claim 1, further comprising calculating a number of events that are stored in one or more particular indexes of a plurality of indexes.

24. The method of claim 1, wherein the raw data is associated with a particular project of a plurality of projects, each project of the plurality of projects having an associated licensed amount of data ingestion.

25. The method of claim 1, further comprising:

in response to the determination that the number of events that were generated during the defined time period has reached the allocated event count, further determining whether a user account associated with the one or more devices is permitted to exceed the allocated event count;

in response to determining that the user account is permitted to exceed the allocated event count, generating up to a threshold number of additional events.

26. The method of claim 1, further comprising:

in response to the determination that the number of events that were generated during the defined time period has reached the allocated event count, further determining whether a user account associated with the one or more devices is permitted to exceed the allocated event count;

in response to determining that the user account is permitted to exceed the allocated event count;

generating up to a threshold number of additional events; and

charging a fee to the user account for exceeding the allocated event count.

27. The method of claim 1, further comprising:

wherein the one or more devices includes both a first set of devices associated with a first company and a second set of devices associated with a second company, each of the first set of devices and the second set of devices managed by a managed security service provider (MSSP);

wherein the raw data includes first raw data received from the first set of devices and second raw data received from the second set of devices;

wherein determining the number of events of the plurality of events that were generated during a defined time period includes determining a first number of events generated based on the first raw data and a second number of events generated based on the second raw data;

causing display of a user interface including separately displaying first metrics based on the first number of events associated with the first company, and second metrics based on the second number of events associated with the second company.

28. The method of claim 1, further comprising, in response to determining that the number of events of the plurality of events that were generated during the defined time period has reached a threshold number that is different than the allocated event count, generating an alert.

29. One or more non-transitory computer-readable storage media, storing instructions, which when executed by one or more processors cause performance of:

receiving raw data from one or more devices;

generating a plurality events from the raw data by;

parsing the raw data into a plurality of events, each event of the plurality of events including a portion of the raw data;

determining a respective timestamp for each event of the plurality of events;

determining a number of events of the plurality of events that were generated during a defined time period;

comparing the number of events that were generated during the defined time period to an allocated event count;

in response to a determination that the number of events that were generated during the defined time period has reached an allocated event count, performing one or more actions to raw data received subsequent to the allocated event count being reached.

30. An apparatus, comprising:

a receiving subsystem, implemented at least partially in hardware, that receives raw data from one or more devices;

an event generation subsystem, implemented at least partially in hardware, that generates a plurality of events from the raw data by;

parsing the raw data into a plurality of events, each event of the plurality of events including a portion of the raw data;

determining a respective timestamp for each event of the plurality of events;

an event counting subsystem, implemented at least partially in hardware, that determines a number of events of the plurality of events that were generated during a defined time period;

a comparison subsystem, implemented at least partially in hardware, that compares the number of events that were generated during the defined time period to an allocated event count;

a data processing subsystem, implemented at least partially in hardware, that in response to a determination that the number of events that were generated during the defined time period has reached an allocated event count, performing one or more actions to raw data received subsequent to the allocated event count being reached.