Metadata tracking for a pipelined search language (data modeling for fields)

• US 8,583,631 B1
• Filed: 01/31/2013
• Issued: 11/12/2013
• Est. Priority Date: 01/31/2013
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Ganapathi, Archana Sulochana, Zhang, Steve Yu, Neels, Alice Emily, Robichaud, Marc Vincent, Sorkin, Stephen Phillip
• Primary Examiner(s): Vy, Hung T
• Application Number:
  US13/756,105
• Time in Patent Office: 285 Days
• Field of Search: 707/722, 707/705, 715174-178
• US Class Current: 707/722
• CPC Class Codes: G06F 16/2425 Iterative querying; Query formulation based on the results of a preceding query
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/2455 Query execution
• CPC Class Codes: G06F 16/248 Presentation of query results

Abstract

Embodiments are directed towards determining and tracking metadata for the generation of visualizations of requested data. A user may request data by providing a query that may be employed to search for the requested data. The query may include a plurality of commands, which may be employed in a pipeline to perform the search and to generate a table of the requested data. In some embodiments, each command may be executed to perform an action on a set of data. The execution of a command may generate one or more columns to append and/or insert into the table of requested data. Metadata for each generated column may be determined based on the actions performed by executing the commands. The table of requested data and the column metadata may be employed to generate and display a visualization of at least a portion of the requested data to a user.

Claims

1. A computer-implemented method, comprising:

receiving a query, wherein the query includes a first command to retrieve a data set and a second command to process the retrieved data set, wherein processing includes characterizing a portion of the retrieved data set;

retrieving the data set in accordance with the first command, wherein the data set includes a plurality of events, wherein each event in the plurality of events is time-stamped, and wherein at least one of the events in the plurality of events is derived from unstructured data;

dividing the retrieved data set into a plurality of portions in accordance with the second command, wherein the retrieved data set is divided based on a first attribute in the data;

generating a plurality of results, wherein each result in the plurality of results is generated from one of the plurality of portions in accordance with the second command, and wherein a result of the plurality of results characterizes a portion of the plurality of portions;

generating metadata for each result in the plurality of results, wherein the metadata indicates that a result has a value for a second attribute;

identifying a set of results of the plurality of results, wherein the set of results share a same value for the second attribute; and

generating a visualization of the data set, wherein the visualization includes the identified set of results, and wherein the visualization groups the identified set of results.

2. The method of claim 1, the method further comprising:

generating a plurality of second results, wherein each second result in the plurality of second results is generated from one of the plurality of portions, and wherein a second result of the plurality of second results characterizes a portion of the plurality of portions; and

generating second metadata for each second result in the plurality of second results, wherein the second metadata indicates that a second result has a second value for the second attribute,

wherein the visualization includes the plurality of the second results, and wherein the visualization groups the second results separately from the set of results.

3. The method of claim 1, the method further comprising:

generating a plurality of second results, wherein each second result in the plurality of second results is generated from one of the plurality of portions, and wherein a second result of the plurality of second results characterizes a portion of the plurality of portions; and

generating second metadata for each second result in the plurality of second results, wherein the second metadata indicates that a second result has a second value for the second attribute,

wherein the metadata for each result in the plurality of results further includes a value for the first attribute,

wherein the second metadata for each second result in the plurality of second results further includes a value for the first attribute,

wherein each result in the plurality of results has a same value for the first attribute as a corresponding second result in the second plurality of results, and

wherein the visualization includes the plurality of the second results and displays them grouped together separately from the set of results.

4. The method of claim 1, wherein the second command includes a pipeline series of commands, and wherein each result in the plurality of results is generated in accordance with a sequential execution of the pipeline series of commands.

5. The method of claim 1, further comprising:

storing each result of the plurality of results in a table.

6. The method of claim 1, wherein each result in the set of results is represented in the visualization using a data series.

7. The method of claim 1, wherein each result in the set of results is represented in the visualization by a bar.

8. The method of claim 1, wherein the results in the identified set of results are grouped along an axis in a graph in the visualization.

9. The method of claim 1, wherein the second command to process the retrieved data set includes a statistical calculation.

10. The method of claim 1, further comprising:

receiving input corresponding to an identification of a label corresponding to the value for the second attribute; and

labeling the identified set of results in the visualization with the label.

11. The method of claim 1, wherein the visualization includes at least one of a bar graph, line chart, and pie chart.

12. A network device for managing data, comprising:

a memory for storing data and instructions; and

a processor that executes the instructions to enable actions, including;

receiving a query, wherein the query includes a first command to retrieve a data set and a second command to process the retrieved data set, wherein processing includes characterizing a portion of the retrieved data set;

retrieving the data set in accordance with the first command, wherein the data set includes a plurality of events, wherein each event in the plurality of events is time-stamped, and wherein at least one of the events in the plurality of events is derived from unstructured data;

dividing the retrieved data set into a plurality of portions in accordance with the second command, wherein the retrieved data set is divided based on a first attribute in the data;

generating a plurality of results, wherein each result in the plurality of results is generated from one of the plurality of portions in accordance with the second command, and wherein a result of the plurality of results characterizes a portion of the plurality of portions;

generating metadata for each result in the plurality of results, wherein the metadata indicates that a result has a value for a second attribute;

identifying a set of results of the plurality of results, wherein the set of results share a same value for the second attribute; and

generating a visualization of the data set, wherein the visualization includes the identified set of results, and wherein the visualization groups the identified set of results.

13. The network device of claim 12, the actions further comprising:

generating a plurality of second results, wherein each second result in the plurality of second results is generated from one of the plurality of portions, and wherein a second result of the plurality of second results characterizes a portion of the plurality of portions; and

generating second metadata for each second result in the plurality of second results, wherein the second metadata indicates that a second result has a second value for the second attribute,

wherein the visualization includes the plurality of the second results, and wherein the visualization groups the second results separately from the set of results.

14. The network device of claim 12, wherein the second command includes a pipeline series of commands, and wherein each result in the plurality of results is generated in accordance with a sequential execution of the pipeline series of commands.

15. The network device of claim 12, the actions further comprising:

storing each result of the plurality of results in a table.

16. The network device of claim 12, wherein each result in the set of results is represented in the visualization using a data series.

17. The network device of claim 12, wherein each result in the set of results is represented in the visualization by a bar.

18. The network device of claim 12, wherein the results in the identified set of results are grouped along an axis in a graph in the visualization.

19. The network device of claim 12, wherein the second command to process the retrieved data set includes a statistical calculation.

20. The network device of claim 12, the actions further comprising:

receiving input corresponding to an identification of a label corresponding to the value for the second attribute; and

labeling the identified and grouped set of results in the visualization with the label.

21. The network device of claim 12, wherein the visualization includes at least one of a bar graph, line chart, and pie chart.

22. A processor-readable non-transitory storage media that includes instructions for displaying data on at least one computing device, wherein the execution of the instructions by a processor enables actions, comprising:

receiving a query, wherein the query includes a first command to retrieve a data set and a second command to process the retrieved data set, wherein processing includes characterizing a portion of the retrieved data set;

retrieving the data set in accordance with the first command, wherein the data set includes a plurality of events, wherein each event in the plurality of events is time-stamped, and wherein at least one of the events in the plurality of events is derived from unstructured data;

dividing the retrieved data set into a plurality of portions in accordance with the second command, wherein the retrieved data set is divided based on a first attribute in the data;

generating a plurality of results, wherein each result in the plurality of results is generated from one of the plurality of portions in accordance with the second command, and wherein a result of the plurality of results characterizes a portion of the plurality of portions;

generating metadata for each result in the plurality of results, wherein the metadata indicates that a result has a value for a second attribute;

identifying a set of results of the plurality of results, wherein the set of results share a same value for the second attribute; and

generating a visualization of the data set, wherein the visualization includes the identified set of results, and wherein the visualization groups the identified set of results.

23. The storage media of claim 22, the actions further comprising:

generating a plurality of second results, wherein each second result in the plurality of second results is generated from one of the plurality of portions, and wherein a second result of the plurality of second results characterizes a portion of the plurality of portions; and

generating second metadata for each second result in the plurality of second results, wherein the second metadata indicates that a second result has a second value for the second attribute,

wherein the visualization includes the plurality of the second results, and wherein the visualization groups the second results separately from the set of results.

24. The storage media of claim 22, wherein the second command includes a pipeline series of commands, and wherein each result in the plurality of results is generated in accordance with a sequential execution of the pipeline series of commands.

25. The storage media of claim 22, wherein the actions further comprise:

storing each result of the plurality of results in a table.

26. The storage media of claim 22, wherein each result in the set of results is represented in the visualization using a data series.

27. The storage media of claim 22, wherein each result in the set of results is represented in the visualization by a bar.

28. The storage media of claim 22, wherein the results in the identified set of results are grouped along an axis in a graph in the visualization.

29. The storage media of claim 22, wherein the second command to process the retrieved data set includes a statistical calculation.

30. The storage media of claim 22, wherein the visualization includes at least one of a bar graph, line chart, and pie chart.