Adjusting network data storage based on event stream statistics

• US 10,700,950 B2
• Filed: 04/29/2015
• Issued: 06/30/2020
• Est. Priority Date: 04/15/2014
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Hsiao, Fang I., Jiang, Wei, Shcherbakov, Vladimir A., Chandrasekharan, Ramkumar, Ching, Clayton S.
• Primary Examiner(s): Bates, Kevin T
• Assistant Examiner(s): McCray, Clarence D
• Application Number:
  US14/699,787
• Publication Number:
  US 20150295796A1
• Time in Patent Office: 1,889 Days
• Field of Search: 709248
• CPC Class Codes: H04L 41/0813 characterised by the conditions triggering a change of settings
• CPC Class Codes: H04L 41/142 using statistical or mathematical methods
• CPC Class Codes: H04L 43/045 for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/0894 Packet rate

Abstract

The disclosed embodiments provide a system that facilitates the processing of network data. During operation, the system causes for display a graphical user interface (GUI) for configuring the generation of time-series event data from network packets captured by one or more remote capture agents. Next, the system causes for display, in the GUI, a first set of user-interface elements for managing one or more event streams containing the time-series event data, wherein managing the one or more event streams includes enabling the generation of a set of statistics from an event stream without subsequently storing and processing at least a first portion of the event stream by one or more components on a network. The GUI then updates the configuration information based on input received through the first set of user-interface elements.

Claims

1. A method performed by a configuration server coupled to a network, the method comprising:

generating a graphical user interface (GUI) including at least one interface element used to define settings related to an event stream comprising timestamped event data, the event stream to be generated by at least one remote capture agent coupled to the network, the settings including at least one setting related to generation of at least one statistic;

generating configuration information based on input received via the at least one interface element;

receiving a first portion of the event stream from a remote capture agent;

generating, based on the configuration information, the at least one statistic based on the first portion of the event stream without subsequently storing and processing the first portion of the event stream;

determining, based on the at least one statistic and a storage limit, a percentage of a second portion of the event stream to store;

receiving the second portion of the event stream from the remote capture agent; and

causing the percentage of the second portion of the event stream to be stored.

2. The method of claim 1, the method further comprising:

sending the configuration information over the network to the at least one remote capture agent, wherein the configuration information is used by the at least one remote capture agent to generate the event stream.

3. The method of claim 1, further comprising generating the at least one statistic from a plurality of event streams received by the configuration server.

4. The method of claim 1, wherein the GUI includes at least one interface element displaying the at least one statistic generated based on the first portion of the event stream.

5. The method of claim 1, wherein the input is first input, and further comprising:

receiving, via the at least one interface element, second input defining settings related to storing the second portion of the event stream based on the at least one statistic; and

updating the configuration information based on the second input.

6. The method of claim 1, wherein the input is first input, and further comprising:

receiving, via the at least one interface element, second input defining settings related to storing the second portion of the event stream based on the at least one statistic;

updating the configuration information based on the second input to obtain updated configuration information; and

sending, based on the updated configuration, the percentage of the second portion of the event stream to another component on the network for subsequent processing and storage.

7. The method of claim 1, further comprising:

determining, based on the at least one statistic, an amount of storage associated with the event stream;

determining that the amount of storage associated with the event stream exceeds the storage limit; and

in response to determining that the amount of storage associated with the event stream exceeds a specified storage limit, sending the percentage of the second portion of the event stream to another component on the network for subsequent processing and storage.

8. The method of claim 1, wherein the input includes at least one setting related to an amount of storage of the event stream based on the at least one statistic and the storage limit;

and

wherein the GUI includes at least one interface element that displays a suggestion for setting the amount of storage of the event stream based on the at least one statistic and the storage limit.

9. The method of claim 1, wherein the GUI includes at least one interface element used to adjust an amount of storage of the event stream based on the at least one statistic and the storage limit;

and

wherein the GUI includes at least one interface element that displays a price associated with storing the event stream above the storage limit.

10. The method of claim 1, wherein the GUI further includes at least one interface element that displays:

event stream information for a plurality of event streams; and

a graph of a metric associated with the timestamped event data in the event stream.

11. The method of claim 1, wherein the GUI further includes at least one interface element that displays:

event stream information for a plurality of event streams including the event stream;

a first graph of a metric associated with the timestamped event data in the event stream; and

a second graph of an aggregated metric calculated based on the plurality of event streams.

12. The method of claim 1, wherein the GUI further includes at least one interface element used to define one or more event attributes to include in the timestamped event data of the event stream.

13. The method of claim 1, wherein the GUI further includes display of at least a portion of the timestamped event data.

14. The method of claim 1, wherein the settings further include settings related to at least one of:

enabling the event stream, disabling the event stream, and deleting the event stream.

15. The method of claim 1, wherein the at least one statistic includes at least one of:

a total number of events in the event stream, a total amount of incoming traffic, a total amount of outgoing traffic, a total amount of traffic, and an index volume associated with the event stream.

16. The method of claim 1, wherein the GUI further includes at least one interface element that displays the at least one statistic generated from the event stream, the method further comprising:

updating the at least one statistic in real-time based on additional timestamped event data received from the at least one remote capture agent to obtain an updated statistic; and

updating the GUI to reflect the updated statistic.

17. The method of claim 1, further comprising:

determining that a size of the event stream does not exceed the storage limit; and

in response to determining that the size of the event stream does not exceed the storage limit associated with the event stream, sending the event stream to another component on the network for subsequent storage and processing of the event stream.

18. The method of claim 1, further comprising:

determining an index volume of the event stream; and

in response to determining the index volume of the event stream, sending the percentage of the second portion of the event stream to another component on the network that subsequently processes and stores the percentage of the second portion of the event stream.

19. The method of claim 1, further comprising:

detecting, based on the at least one statistic, a high traffic volume associated with the event stream; and

in response to detecting the high traffic volume associated with the event stream, sending the percentage of the second portion of the event stream to another component on the network that subsequently processes and stores the percentage of the second portion of the event stream.

20. The method of claim 1, wherein the GUI includes display of the at least one statistic generated based on the event stream and further includes display of at least one additional statistic generated based on at least one additional event stream received by the configuration server.

21. The method of claim 1, wherein the GUI includes display of the at least one statistic generated based on the event stream and further includes display of at least one additional statistic generated based on at least one additional event stream received by the configuration server, and wherein the at least one additional event stream is not subsequently processed and stored by another component on the network.

22. The method of claim 1, wherein the GUI includes display of the at least one statistic generated based on the event stream and further includes display of at least one additional statistic generated based on at least one additional event stream received by the configuration server, and wherein the at least one additional event stream is sent to another component on the network for subsequent processing and storage.

23. The method of claim 1, further comprising:

determining, based on a remaining storage limit associated with the event stream, to send the second portion of the event stream to another component on the network for subsequent processing and storage; and

sending the second portion of the event stream to the another component on the network.

24. The method of claim 1, further comprising:

detecting a historical trend associated with the at least one statistic; and

in response to detecting the historical trend associated with the at least one statistic, sending the second portion of the event stream to another component on the network that subsequently processes and stores at least a portion of the event stream.

25. The method of claim 1, further comprising:

detecting a high traffic volume associated with the event stream; and

in response to detecting the high traffic volume associated with the event stream, sending the second portion of the event stream to another component on the network that subsequently processes and stores at the second portion of the event stream.

26. The method of claim 1, further comprising:

detecting a light traffic volume associated with the event stream; and

in response to detecting the light traffic volume associated with the event stream, sending the second portion of the event stream to another component on the network that subsequently processes and stores the second portion of the event stream.

27. The method of claim 1, further comprising:

generating a plurality of statistics based on a plurality of respective event streams; and

aggregating the plurality of statistics to obtain an aggregated statistic.

28. The method of claim 1, further comprising:

generating a plurality of statistics based on a plurality of respective event streams;

aggregating the plurality of statistics to obtain an aggregated statistic; and

generating a GUI including display of the aggregated statistic.

29. An apparatus, comprising:

one or more processors; and

memory storing instructions that, when executed by the one or more processors, cause the apparatus to;

generate a graphical user interface (GUI) including at least one interface element used to define settings related to an event stream comprising timestamped event data, the event stream to be generated by at least one remote capture agent coupled to a network, the settings including at least one setting related to generation of at least one statistic; and

generate configuration information based on input received via the at least one interface element;

receive a first portion of the event stream from a remote capture agent;

generate, based on the configuration information, the at least one statistic based on the first portion of the event stream without subsequently storing and processing the first portion of the event stream;

determine, based on the at least one statistic and a storage limit, a percentage of a second portion of the event stream to store;

receive the second portion of the event stream from the remote capture agent; and

cause the percentage of the second portion of the event stream to be stored, the percentage determined based on the at least one statistic.

30. A non-transitory computer-readable storage medium storing instructions which, when executed by a computer, cause a configuration server coupled to a network to perform operations comprising:

generating a graphical user interface (GUI) including at least one interface element used to define settings related to an event stream comprising timestamped event data, the event stream to be generated by at least one remote capture agent coupled to the network, the settings including at least one setting related to generation of at least one statistic based on the event stream without subsequently storing and processing at least a portion of the event stream;

generating configuration information based on input received via the at least one interface element;

receiving a first portion of the event stream from a remote capture agent;

generating, based on the configuration information, the at least one statistic based on the first portion of the event stream without subsequently storing and processing the first portion of the event stream;

determining, based on the at least one statistic and a storage limit, a percentage of a second portion of the event stream to store;

receiving the second portion of the event stream from the remote capture agent; and

causing the percentage of the second portion of the event stream to be stored, the percentage determined based on the at least one statistic.