Generation of a data model applied to queries

• US 9,128,980 B2
• Filed: 01/31/2015
• Issued: 09/08/2015
• Est. Priority Date: 09/07/2012
• Status: Active Grant

• Current Assignee: Splunk Inc.
• Original Assignee: Splunk Inc.
• Inventors: Neels, Alice Emily, Ganapathi, Archana Sulochana, Robichaud, Marc, Sorkin, Stephen Phillip, Zhang, Steve Yu
• Primary Examiner(s): NGUYEN, PHONG H
• Application Number:
  US14/611,232
• Publication Number:
  US 20150142847A1
• Time in Patent Office: 220 Days
• Field of Search: 707/759, 707/760, 707/763, 707/765, 707/779
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2425 Iterative querying; Query formulation based on the results of a preceding query
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 40/186 Templates

Abstract

Embodiments include generating data models that may give semantic meaning for unstructured or structured data that may include data generated and/or received by search engines, including a time series engine. A method includes generating a data model for data stored in a repository. Generating the data model includes generating an initial query string, executing the initial query string on the data, generating an initial result set based on the initial query string being executed on the data, determining one or more candidate fields from one or results of the initial result set, generating a candidate data model based on the one or more candidate fields, iteratively modifying the candidate data model until the candidate data model models the data, and using the candidate data model as the data model.

Claims

1. A method of accessing data, including:

accessing a data model structure, the data model structure comprising;

a root object query that, when executed, returns a set of time stamped events in a data store on a computing device, each event including a portion of unstructured data;

a model schema that references fields that can be extracted, by an extraction rule or regular expression, from the unstructured data in the time stamped events without modifying the unstructured data; and

one or more submodels;

each of the submodels comprising;

a child object that provides for narrower search criteria than the root object query such that, when the child object query is executed against the time stamped events, the child object query returns a subset of the set of time stamped events that is smaller than the set;

a submodel schema that inherits one or more fields referenced in the model schema; and

the submodel schema further references additional fields that can be extracted, by an extraction rule or regular expression, from the unstructured data in the time stamped events without modifying the unstructured data;

receiving electronically a data request comprising reference to a submodel selected from the data model structure and a query to be performed against the subset referenced by the selected submodel; and

identifying responsive events, including extracting values from at least some of the events in the subset at query time using the extraction rule or regular expression in the submodel schema without modifying the unstructured event and matching the extracted values to the query;

returning at least some values from or derived from the fields in the responsive events referenced by the submodel schema.

2. The method of claim 1, wherein the values derived from the fields in the responsive events are results of aggregate calculations from the values in the fields of responsive events.

3. The method of claim 1, further including causing display of the values from or derived from the fields in the responsive events as values in rows in a pivot report template.

4. A method, comprising:

receiving a selection of an object included in a data model that has both a root object and at least one child object;

the root object includes;

a root object query that, when executed, returns a set of time stamped events from a data store on a computing device, each event including a portion of raw machine data; and

a root object schema identifying a set of one or more fields, each field defined by an extraction rule or regular expression that can be used to extract a value for the field from each event in a subset of the set of time stamped events, each extraction operating on the raw machine data in an event without modifying the event'"'"'s raw machine data;

the at least one child object includes;

a child object query that provides for narrower search criteria than the root object query such that, when the child object query is executed against the time stamped events in the data store, the child object query returns a second set of time stamped events that is a subset of the set of time stamped events generated by executing the root object query against the time stamped events in the data store; and

a child object schema identifying a second set of one or more fields, each field defined by an extraction rule or regular expression that can be used to extract a value for the field from each event in a subset of the second set of time stamped events, each extraction operating on the raw machine data in an event without modifying the event'"'"'s raw machine data;

receiving from a user a search query that uses only fields that are included in the object schema of the selected object; and

executing the search query received from the user only against the set of time stamped events generated by executing the object query of the selected object against the set of time stamped events in the data store.

5. The method of claim 4,

wherein the second set of one or more fields comprises a subset of the set of one or more fields identified by the root object schema.

6. The method of claim 4,

wherein the second set of one or more fields comprises both (i) a subset of the set of one or more fields identified by the root object schema, and (ii) additional fields not identified by the root object schema.

7. The method of claim 4,

wherein the second set of one or more fields has no fields in common with the set of one or more fields identified by the root object schema.

8. The method of claim 4, wherein the selection of the data model is indicated by receiving a selection of an object included in the data model.

9. The method of claim 4, wherein the selection of the data model is indicated by receiving a selection of a root object or child object included in the data model.

10. The method of claim 4, further comprising causing display of references to a plurality of data models in a graphical user interface enabling selection of a data model, and wherein the selection of the data model is received through the graphical user interface.

11. The method of claim 4, further comprising causing display of references to a plurality of data models in a graphical user interface that enables hierarchical display of child objects included in a data model for which a reference has been displayed.

12. The method of claim 4, further comprising:

populating a pivot graphical user interface with controls for manipulating data that correspond to the set of one or more fields in the root object schema for the selected data model, and wherein the pivot graphical user interface enables a user to further filter the set of time stamped events that are returned when the root object query is executed against the set of time stamped events in the data store.

13. The method of claim 4, further comprising:

populating a pivot graphical user interface with controls for manipulating data that correspond to the set of one or more fields in the root object schema for the selected data model, and wherein the pivot graphical user interface enables a user to perform aggregate calculations on a field in the set of time stamped events that are returned when the root object query is executed against the set of time stamped events in the data store.

14. The method of claim 4, further comprising:

populating a pivot graphical user interface with controls for manipulating data that correspond to the set of one or more fields in the root object schema for the selected data model, and wherein the pivot graphical user interface enables a user to generate a data visualization whose features depend on fields or criteria for fields in the set of time stamped events that are returned when the root object query is executed against the set of time stamped events in the data store.

15. The method of claim 4, further comprising:

populating a pivot graphical user interface with controls for manipulating data that correspond to a set of one or more fields in an object schema included in an object in the selected data model, and wherein the pivot graphical user interface enables a user to thither filter the set of time stamped events that are returned when a child object query included in the object is executed against the set of time stamped events in the data store.

16. The method of claim 4, further comprising:

populating a pivot graphical user interface with controls for manipulating data that correspond to a set of one or more fields in an Object schema included in an object in the selected data model, and wherein the pivot graphical user interface enables a user to perform aggregate calculations on a field in a set of time stamped events that are returned when an object query in the object is executed against the set of time stamped events in the data store.

17. The method of claim 4, further comprising:

populating a pivot graphical user interface with controls for manipulating data that correspond to a set of one or more fields in an object schema included in an object in the selected data model, and wherein the pivot graphical user interface enables a user to generate a data visualization whose features depend on fields or criteria for fields in the set of time stamped events that are returned when an object query in the object is executed against the set of time stamped events in the data store.