Predictive estimation for ingestion, performance and utilization in a data indexing and query system

• US 10,216,862 B1
• Filed: 09/26/2016
• Issued: 02/26/2019
• Est. Priority Date: 09/26/2016
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Shrigondekar, Anish A., Bond, Eric, Koshe, Dhananjay, Kerai, Jagannath, Lin, Michael C.
• Primary Examiner(s): Uddin, Mohammed R
• Application Number:
  US15/276,652
• Time in Patent Office: 883 Days
• Field of Search: 707746, 707673, 707754, 707696, 707999003
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/901 Indexing; Data structures therefor; Storage structures for retrieval from the web G06F16/951
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957

Abstract

Disclosed herein is a data estimation technique for a data intake and query system. The system receives user inputs indicative that a first data source is to be the subject of a storage related estimate. The system receives a first plurality of events generated by the first data source. The system indexes only a sample of the received first plurality of events, based on a sampling criterion, where the sample is fewer than all of the first plurality of events. The system generates the storage related estimate based on at least some of the first plurality of events, and causes an indication of the estimate to be output to a user.

Claims

1. A method comprising:

receiving, by a data intake and query system, user input specifying that a first data source is to be a subject of a storage related estimate;

receiving, by the data intake and query system, raw data generated by the first data source;

parsing, by the data intake and query system, the raw data generated by the first data source into a first plurality of events;

generating the storage related estimate, by the data intake and query system, based on at least some of the first plurality of events, the storage related estimate being an estimate of an amount of storage space that would be needed for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for a time period;

causing an indication of the storage related estimate to be output to a user of the data intake and query system; and

completing, by the data intake and query system, an indexing of only a sample of the received first plurality of events based on a sampling criterion, the sample being fewer than all of the first plurality of events, and not completing indexing of a remainder of the first plurality of events in the absence of a user input indicative that the first data source should be indexed, wherein completing indexing includes committing data being indexed or to be indexed to persistent storage.

2. A method as recited in claim 1, wherein the storage related estimate comprises a storage indication of an amount of storage space that would be needed for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for a time period;

the method further comprising;

causing to be output to the user, in association with the storage indication, an indication of an amount of storage space that would be needed if the data intake and query system were not to index or store any data from the first data source for the time period.

3. A method as recited in claim 1, wherein the storage related estimate comprises a license indication of a license requirement for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for a time period.

4. A method as recited in claim 1, wherein the storage related estimate comprises a license indication of a license requirement for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for a time period;

the method further comprising;

causing to be output to the user, in association with the license indication, an indication of a current license associated with the user.

5. A method as recited in claim 1, wherein the storage related estimate comprises:

a storage indication indicative of an amount of storage space that would be needed for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for a time period; and

a license indication indicative of a license requirement for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for the time period;

the method further comprising;

causing to be output to the user, in association with the storage indication,

an indication of an amount of storage space that would be needed if the data intake and query system were not to index or store any data from the first data source for the time period, and

an indication of a current license associated with the user.

6. A method as recited in claim 1, further comprising:

receiving, by the data intake and query system, second data from a second data source that has been designated to be indexed, the second data including a second plurality of events;

completing indexing, by the data intake and query system, of all of the second plurality of events.

7. A method as recited in claim 1, wherein the storage related estimate comprises an indication of a license upgrade required for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for a time period.

8. A method as recited in claim 1, wherein causing an indication of the storage related estimate to be output to the user comprises generating and causing to be displayed a graphical indication that includes:

a first graphical element indicative of a total amount of data to be indexed by the data intake and query system for a time period if the data intake and query system were not to index any data from the second data source; and

a second graphical element indicative of a total amount of data to be indexed by the data intake and query system if the data intake and query system were to index all data received from the second data source, for the time period.

9. A method as recited in claim 1, wherein causing an indication of the storage related estimate to be output to the user comprises generating and causing to be displayed a graphical indication that includes:

a first graphical element indicative of a total amount of data to be indexed by the data intake and query system for a time period if the data intake and query system were not to index any data from the second data source; and

a second graphical element indicative of a total amount of data to be indexed by the data intake and query system if the data intake and query system were to index all data received from the second data source, for the time period; and

wherein one of the first graphical element or the second graphical element is overlaid on the other of the first graphical element or the second graphical element.

10. A method as recited in claim 1, wherein causing an indication of the storage related estimate to be output to the user comprises generating and causing to be displayed a graphical indication that includes:

a first graphical element indicative of a total amount of data to be indexed by the data intake and query system for a time period if the data intake and query system were not to index any data from the second data source; and

a second graphical element indicative of a total amount of data to be indexed by the data intake and query system if the data intake and query system were to index all data received from the second data source, for the time period;

the method further comprising;

causing to be output to the user an indication of a current license associated with the user, in proximity to the first and second plurality of graphical elements.

11. A method as recited in claim 1, wherein causing an indication of the storage related estimate to be output to the user comprises generating and causing to be displayed a graphical indication that includes:

a first graphical element indicative of a total amount of data to be indexed by the data intake and query system for a time period if the data intake and query system were not to index any data from the second data source; and

a second graphical element indicative of a total amount of data to be indexed by the data intake and query system if the data intake and query system were to index all data received from the second data source, for the time period;

the method further comprising;

causing to be output to the user an indication of a license requirement for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for the time period, in proximity to the first and second plurality of bars.

12. A method as recited in claim 1, wherein causing an indication of the storage related estimate to be output to a user comprises causing to be displayed to the user a bar graph that includes:

a first plurality of bars, lengths of which are representative of total amounts of data indexed or to be indexed by the data intake and query system for a respective plurality of time periods; and

a second plurality of bars, lengths of which are representative of total amounts of data to be indexed by the data intake and query system if the data intake and query system were to index all data received from the first data source, for the respective plurality of time periods.

13. A method as recited in claim 1, wherein causing an indication of the storage related estimate to be output to a user comprises causing to be displayed to the user a bar graph that includes:

a first plurality of bars, lengths of which are representative of total amounts of data indexed or to be indexed by the data intake and query system for a respective plurality of time periods; and

a second plurality of bars, lengths of which are representative of total amounts of data to be indexed by the data intake and query system if the data intake and query system were to index all data received from the first data source, for the respective plurality of time periods; and

wherein at least some of the first plurality of bars are overlaid on respective ones of the second plurality of bars, or vice versa.

14. A method as recited in claim 1, wherein causing an indication of the storage related estimate to be output to a user comprises causing to be displayed to the user a bar graph that includes:

a first plurality of bars, lengths of which are representative of total amounts of data indexed or to be indexed by the data intake and query system for a respective plurality of time periods; and

a second plurality of bars, lengths of which are representative of total amounts of data to be indexed by the data intake and query system if the data intake and query system were to index all data received from the first data source, for the respective plurality of time periods; and

wherein causing an indication of the storage related estimate to be output to a user comprises causing an indication of a current license associated with the user to be displayed in proximity to the first and second plurality of bars.

15. A method as recited in claim 1, wherein causing an indication of the storage related estimate to be output to a user comprises causing to be displayed to the user a bar graph that includes:

a first plurality of bars, lengths of which are representative of total amounts of data indexed or to be indexed by the data intake and query system for a respective plurality of time periods; and

a second plurality of bars, lengths of which are representative of total amounts of data to be indexed by the data intake and query system if the data intake and query system were to index all data received from the first data source, for the respective plurality of time periods; and

wherein causing an indication of the storage related estimate to be output to a user comprises causing an indication of a license requirement to be displayed in proximity to the first and second plurality of bars, the license requirement being a requirement for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for a time period.

16. A method as recited in claim 1, wherein each of the first plurality of events includes machine data.

17. A method as recited in claim 1, wherein each of the first plurality of events includes a timestamped machine data.

18. A method as recited in claim 1, further comprising:

enabling the user to view event data in events that are included in said sample of the first plurality of events, while received data from the first data source has not been fully indexed.

19. A method as recited in claim 1, further comprising, while received data from the first data source has not been fully indexed:

executing a user-specified search query on events included in said sample of the first plurality of events; and

enabling the user to view a result of the user-specified search query.

20. A method as recited in claim 1, wherein each of the first plurality of events includes a timestamped machine data, the method further comprising:

enabling the user to view raw machine data in events that are included in said sample of the first plurality of events while received data from the first data source has not been fully indexed.

21. A method as recited in claim 1, further comprising:

causing a user input feature to be presented to the user to enable the user to specify that a status of the first data source should be changed from data source to be estimated to data source to be fully indexed; and

in response to a user input directed to the user input feature, causing all data received from the first data source to be indexed by the data intake and query system.

22. A method as recited in claim 1, further comprising:

generating a plurality of metrics for the first plurality of events;

wherein the storage related estimate is generated based on at least one of the plurality of metrics.

23. A method as recited in claim 1, further comprising:

generating a plurality of metrics for the first plurality of events;

wherein the storage related estimate is generated based on at least one of the plurality of metrics, and wherein the plurality of metrics comprise at least one of;

number of events in the first plurality of events, size of events in the first plurality of events, impact on processor utilization, impact on input/output utilization, impact on memory utilization, search performance, network performance or indexing performance.

24. A method as recited in claim 1, further comprising:

storing the sample of the first plurality of events in a dedicated persistent storage container separate from fully indexed data, the dedicated persistent storage container being allocated only for data of estimated data sources.

25. A method as recited in claim 1, further comprising:

causing to be displayed to the user, for each of a plurality of future time periods, an indication of an amount of storage space that would be required if the data intake and query system were to index all data received by the data intake and query system from the first data source and other data sources from which the data intake and query system receives data.

26. A method as recited in claim 1, wherein the sampling criterion comprises a sampling rate.

27. A method as recited in claim 1, wherein the sampling criterion comprises a prioritization based on an indexing workload associated with indexing data sources that are designated as to be indexed.

28. A system comprising:

a communication device through which to communicate on a computer network; and

at least one processor operatively coupled to the communication device and configured to perform operations including

receiving user inputs specifying that a first data source is to be a subject of a storage related estimate;

receiving data generated by the first data source;

parsing the data generated by the first data source into a first plurality of events;

generating the storage related estimate, based on at least some of the first plurality of events, the storage related estimate being an estimate of an amount of storage space that would be needed for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for a time period;

causing an indication of the storage related estimate to be output to a user; and

completing an indexing of only a sample of the received first plurality of events based on a sampling criterion, and not completing indexing of a remainder of the first plurality of events in the absence of a user input indicative that the first data source should be indexed, the sample being fewer than all of the first plurality of events, wherein completing indexing includes committing data being indexed or to be indexed to persistent storage.

29. A non-transitory machine-readable storage medium for use in a processing system of a data intake and query system, the non-transitory machine-readable storage medium storing instructions, an execution of which in the processing system causes the processing system to perform operations comprising:

receiving user inputs specifying that a first data source is to be a subject of a storage related estimate;

receiving raw data generated by the first data source;

parsing the raw data generated by the first data source into a first plurality of events;

generating the storage related estimate, based on at least some of the first plurality of events, the storage related estimate being an estimate of an amount of storage space that would be needed for the data intake and query system to index, including to persistently store, all data received by the data intake and query system from the first data source for a time period;

causing an indication of the storage related estimate to be output to a user; and

completing indexing of only a sample of the received first plurality of events based on a sampling criterion, and not completing indexing of a remainder of the first plurality of events in the absence of a user input indicative that the first data source should be indexed, the sample being fewer than all of the first plurality of events, wherein completing indexing includes committing data being indexed or to be indexed to persistent storage.