Display of data ingestion information based on counting generated events

• US 10,282,455 B2
• Filed: 04/20/2015
• Issued: 05/07/2019
• Est. Priority Date: 04/20/2015
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Chauhan, Vijay, Shahbaz, Banipal, Hazekamp, David
• Primary Examiner(s): Shiferaw, Eleni A
• Assistant Examiner(s): Huang, Cheng-Feng
• Application Number:
  US14/691,475
• Publication Number:
  US 20160307173A1
• Time in Patent Office: 1,478 Days
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/26 Visual data mining; Browsing structured data
• CPC Class Codes: G06F 16/901 Indexing; Data structures therefor; Storage structures for retrieval from the web G06F16/951
• CPC Class Codes: G06Q 2220/18 Licensing

Abstract

A data intake and query system measures an amount of raw data ingested by the system during defined periods of time. As used herein, ingesting raw data generally refers to receiving the raw data from one or more computing devices and processing the data for storage and searchability. Processing the data may include, for example, parsing the raw data into “events,” where each event includes a portion of the received data and is associated with a timestamp. Based on a calculated number of events generated by the system during one or more defined time periods, the system may calculate various metrics including, but not limited to, a number of events generated during a particular day, a number of events generated per day over a period of time, a maximum number of events generated in a day over a period of time, an average number of events generated per day, etc.

Claims

1. A method comprising:

receiving raw data from one or more data sources, the raw data including first raw data received via one or more first devices for a first user account and second raw data received via one or more second devices for a second user account;

generating a plurality of events from the raw data by parsing the raw data into the plurality of events, each event of the plurality of events including a portion of the raw data, the plurality of events enabling execution of time-based searches across the portions of the raw data included in the plurality of events;

determining a number of events of the plurality of events that were generated from the raw data during a defined time period by determining a first number of events generated from the first raw data during the defined time period and a second number of events generated from the second raw data during the defined time period;

calculating one or more first metrics related to a first amount of data ingestion based on the determined first number of events generated from the first raw data during the defined time period and one or more second metrics related to a second amount of data ingestion based on the determined second number of events generated from the second raw data during the defined time period;

generating a user interface that includes a display of the one or more first metrics related to the first amount of data ingestion associated with the first user account, a display of the one or more second metrics related to the second amount of data ingestion associated with the second user account, and a display of a threshold input option to receive a user-specified threshold value related to data ingestion associated with at least the first user account that, when exceeded, results in an alert; and

causing display of the user interface.

2. The method of claim 1, further comprising storing the plurality of events in an indexed store.

3. The method of claim 1, wherein the plurality of events includes a first set of events associated with a first project and a second set of events associated with a second project, and wherein the user interface displays a third set of metrics associated with the first project and a fourth set of metrics associated with the second project.

4. The method of claim 1, wherein parsing the raw data into the plurality of events further comprises determining event boundaries for the plurality of events.

5. The method of claim 1, wherein the plurality of events are searchable using a late-binding schema comprising one or more extraction rules for extracting values from the events.

6. The method of claim 1, wherein the defined time period corresponds to one or more days.

7. The method of claim 1, wherein the defined time period corresponds to one or more seconds.

8. The method of claim 1, further comprising calculating an average number of events that were generated over a plurality of time periods.

9. The method of claim 1, further comprising calculating a fee amount based on the first number of events of the plurality of events that were generated during the defined time period.

10. The method of claim 1, further comprising:

comparing the first number of events to a number of events that have been licensed for the first user account;

in response to determining that the first number of events exceeds the number of events that have been licensed, storing excess events in a non-searchable index.

11. The method of claim 1, further comprising:

comparing the first number of events to a licensed amount;

in response to determining that the first number of events exceeds the licensed amount, storing excess events in a non-searchable index;

enabling access the indexed events that are stored in the non-searchable index when additional capacity to increase the licensed amount is purchased.

12. The method of claim 1, further comprising:

comparing the first number of events to a licensed amount;

in response to determining that the first number of events exceeds the licensed amount, automatically increasing the licensed amount.

13. The method of claim 1, further comprising:

comparing the first number of events to a licensed amount;

in response to determining that the first number of events exceeds the licensed amount, generating an alert.

14. The method of claim 1, further comprising:

comparing the first number of events to a licensed amount;

in response to determining that the first number of events exceeds the licensed amount, sending an alert to a particular user.

15. The method of claim 1, further comprising calculating a fee amount based on a peak number of events generated during the defined time period.

16. The method of claim 1, further comprising calculating a fee amount based on a number of devices from which the raw data is received.

17. The method of claim 1, further comprising calculating a fee amount based on both of a first fee rate for the first number of events generated and a second fee rate for the second number of events generated.

18. The method of claim 1, wherein the first user account and the second user account are managed by a managed security service provider (MSSP).

19. The method of claim 1, wherein the one or more first metrics include the first number of events generated during the defined period of time.

20. The method of claim 1, wherein the one or more first metrics include a number of events generated during each of one or more previous periods of time.

21. The method of claim 1, wherein the one or more first metrics include a comparison of a number of events generated during at least two different time periods.

22. The method of claim 1, further comprising calculating a number of events that are stored in one or more particular indexes of one or more indexes.

23. The method of claim 1, wherein the first raw data is associated with a particular project of a plurality of projects, each project of the plurality of projects having an associated licensed amount of data ingestion.

24. The method of claim 1, further comprising:

wherein a third number of events is associated with a first project, and a fourth number of events is associated with a second project;

determining a third number of events associated with the first project generated during the defined time period;

determining a fourth number of events associated with the second project generated during the defined time period.

25. The method of claim 1,

wherein the first user account is associated with a first company and the second user account is associated with a second company, each of the first user account and the second user account managed by a managed security service provider (MSSP).

26. The method of claim 1, wherein the number of events are determined by counting events as the events are stored in an index wherein counting events as the events are stored in the index comprises incrementing an event counter each time an event generated from the raw data is stored in the index.

27. The method of claim 1, wherein the number of events are determined by counting events as the events are stored in the index, wherein counting events as the events are stored in the index comprises incrementing an event counter each time an event generated from the raw data is stored in the index, the event counter corresponding with the particular user account, and the event counter tracking a total number of events stored in the index during a period of time and resetting upon termination of the period of time.

28. One or more non-transitory computer-readable storage media, storing instructions, which when executed by one or more processors cause performance of:

receiving raw data from one or more data sources, the raw data including first raw data received via one or more first devices for a first user account and second raw data received via one or more second devices for a second user account;

generating a plurality of events from the raw data by parsing the raw data into the plurality of events, each event of the plurality of events including a portion of the raw data, the plurality of events enabling execution of time-based searches across the portions of the raw data included in the plurality of events;

determining a number of events of the plurality of events that were generated from the raw data during a defined time period by determining a first number of events generated from the first raw data during the defined time period and a second number of events generated from the second raw data during the defined time period;

calculating one or more first metrics related to a first amount of data ingestion based on the determined first number of events generated from the first raw data during the defined time period and one or more second metrics related to a second amount of data ingestion based on the determined second number of events generated from the second raw data during the defined time period;

generating a user interface that includes a display of the one or more first metrics related to the first amount of data ingestion associated with the first user account, a display of the one or more second metrics related to the second amount of data ingestion associated with the second user account, and a display of a threshold input option to receive a user-specified threshold value related to data ingestion associated with at least the first user account that, when exceeded, results in an alert; and

causing display of the user interface.

29. An apparatus, comprising:

one or more memories storing instructions; and

one or more hardware processors to execute the instructions, the instructions comprising;

receiving raw data from one or more data sources, the raw data including first raw data received via one or more first devices for a first user account and second raw data received via one or more second devices for a second user account;

generating a plurality of events from the raw data by parsing the raw data into the plurality of events, each event of the plurality of events including a portion of the raw data, the plurality of events enabling execution of time-based searches across the portions of the raw data included in the plurality of events;

determining a number of events of the plurality of events that were generated from the raw data during a defined time period by determining a first number of events generated from the first raw data during the defined time period and a second number of events generated from the second raw data during the defined time period;

calculating one or more first metrics related to a first amount of data ingestion based on the determined first number of events generated from the first raw data during the defined time period and one or more second metrics related to a second amount of data ingestion based on the determined second number of events generated from the second raw data during the defined time period;

generating a user interface that includes a display of the one or more first metrics related to the first amount of data ingestion associated with the first user account, a display of the one or more second metrics related to the second amount of data ingestion associated with the second user account, and a display of a threshold input option to receive a user-specified threshold value related to data ingestion associated with at least the first user account that, when exceeded, results in an alert; and

causes display of the user interface.