GENERATION OF A DATA MODEL APPLIED TO QUERIES

• US 20150142847A1
• Filed: 01/31/2015
• Published: 05/21/2015
• Est. Priority Date: 09/07/2012
• Status: Active Grant

• Current Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Original Assignee: Splunk Inc. (Cisco Systems, Inc.)
• Inventors: Robichaud, Marc, SORKIN, Stephen Phillip, ZHANG, Steve Yu, NEELS, Alice Emily, GANAPATHI, Archana Sulochana
• Granted Patent: US 9,128,980 B2
• Time in Patent Office: Days
• US Class Current: 707/769
• CPC Class Codes: G06F 16/2425 Iterative querying; Query formulation based on the results of a preceding query
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 40/186 Templates

Abstract

Embodiments include generating data models that may give semantic meaning for unstructured or structured data that may include data generated and/or received by search engines, including a time series engine. A method includes generating a data model for data stored in a repository. Generating the data model includes generating an initial query string, executing the initial query string on the data, generating an initial result set based on the initial query string being executed on the data, determining one or more candidate fields from one or results of the initial result set, generating a candidate data model based on the one or more candidate fields, iteratively modifying the candidate data model until the candidate data model models the data, and using the candidate data model as the data model.

Claims

1. A method of accessing data, including:

accessing a data model structure, the data model structure comprising;

a set of time stamped events in a data store on a computing device, each event including a portion of unstructured data;

a model schema that references fields that can be extracted, by an extraction rule or regular expression, from the unstructured data in the time stamped events without modifying the unstructured data; and

one or more submodels;

each of the submodels comprising;

a subset of the set of time stamped events that is smaller than the set;

a submodel schema that inherits one or more fields referenced in the model schema; and

the submodel schema further references additional fields that can be extracted, by an extraction rule or regular expression, from the unstructured data in the time stamped events without modifying the unstructured data;

receiving electronically a data request comprising reference to a submodel selected from the data model structure and a query to be performed against the subset referenced by the selected submodel; and

identifying responsive events, including extracting values from at least some of the events in the subset at query time using the extraction rule or regular expression in the submodel schema without modifying the unstructured event and matching the extracted values to the query;

returning at least some values from or derived from the fields in the responsive events referenced by the submodel schema.

2. The method of claim 1, wherein the values derived from the fields in the responsive events are results of aggregate calculations from the values in the fields of responsive events.

3. The method of claim 1, wherein the values derived from the fields in the responsive events include counts of responsive events.

4. The method of claim 1, wherein the values derived from the fields in the responsive events average the values in the fields of responsive events.

5. The method of claim 1, wherein the values derived from the fields in the responsive events are presented as data views.

6. The method of claim 1, further including causing display of the values from or derived from the fields in the responsive events as values in rows in a pivot report template.

7. The method of claim 1, further including causing display of unique values among the values from the fields in the responsive events as rows in a pivot report template.

8. The method of claim 1, further including causing display of the values from the fields in the responsive events as values in rows of a pivot report interface and further causing display of statistics derived from the rows in the pivot report.

9. The method of claim 1, further including causing display of the values from the fields in the responsive events as values in rows of a pivot report interface and further causing display of aggregates derived from values in the fields of the responsive events.

10. The method of claim 1, further including:

nesting of the data model structure and submodel is further nested in a tree fashion to sub-sub models and sub-sub-sub models along some branches of the data model structure.

11. The method of claim 1, further including:

nesting of the data model structure and submodel is further nested in a tree fashion to sub-sub models and sub-sub-sub models along some branches of the data model structure with inheritance of data constraints and available field extractions from levels down to further sub levels.

12. A system, comprising:

one or more processors; and

one or more non-transitory computer-readable storage media storing instructions configured to cause the one or more processors to perform operations including;

accessing a data model structure, the data model structure comprising;

a set of time stamped events in a data store on a computing device, each event including a portion of unstructured data;

a model schema that references fields that can be extracted, by an extraction rule or regular expression, from the unstructured data in the time stamped events without modifying the unstructured data; and

one or more submodels;

each of the submodels comprising;

a subset of the set of time stamped events that is smaller than the set;

a submodel schema that inherits one or more fields referenced in the model schema; and

the submodel schema further references additional fields that can be extracted, by an extraction rule or regular expression, from the unstructured data in the time stamped events without modifying the unstructured data;

receiving electronically a data request comprising reference to a submodel selected from the data model structure and a query to be performed against the subset referenced by the selected submodel; and

identifying responsive events, including extracting values from at least some of the events in the subset at query time using the extraction rule or regular expression in the submodel schema without modifying the unstructured event and matching the extracted values to the query;

returning at least some values from or derived from the fields in the responsive events referenced by the submodel schema.

13. The system of claim 12, wherein the values derived from the fields in the responsive events are results of aggregate calculations from the values in the fields of responsive events.

14. The system of claim 12, wherein the values derived from the fields in the responsive events are presented as data views.

15. The system of claim 12, further including causing display of the values from or derived from the fields in the responsive events as values in rows in a pivot report template.

16. The system of claim 12, further including causing display of unique values among the values from the fields in the responsive events as rows in a pivot report template.

17. The system of claim 12, further including causing display of the values from the fields in the responsive events as values in rows of a pivot report interface and further causing display of statistics derived from the rows in the pivot report.

18. The system of claim 12, further including:

nesting of the data model structure and submodel is further nested in a tree fashion to sub-sub models and sub-sub-sub models along some branches of the data model structure with inheritance of data constraints and available field extractions from levels down to further sub levels.

19. A computer-program product, tangibly embodied in one or more non-transitory machine-readable media, including instructions configured to cause one or more data processing apparatuses to carry out actions including:

accessing a data model structure, the data model structure comprising;

a set of time stamped events in a data store on a computing device, each event including a portion of unstructured data;

a model schema that references fields that can be extracted, by an extraction rule or regular expression, from the unstructured data in the time stamped events without modifying the unstructured data; and

one or more submodels;

each of the submodels comprising;

a subset of the set of time stamped events that is smaller than the set;

a submodel schema that inherits one or more fields referenced in the model schema; and

the submodel schema further references additional fields that can be extracted, by an extraction rule or regular expression, from the unstructured data in the time stamped events without modifying the unstructured data;

receiving electronically a data request comprising reference to a submodel selected from the data model structure and a query to be performed against the subset referenced by the selected submodel; and

identifying responsive events, including extracting values from at least some of the events in the subset at query time using the extraction rule or regular expression in the submodel schema without modifying the unstructured event and matching the extracted values to the query;

returning at least some values from or derived from the fields in the responsive events referenced by the submodel schema.

20. The computer program product of claim 19, wherein the values derived from the fields in the responsive events are results of aggregate calculations from the values in the fields of responsive events.

21. The computer program product of claim 19, wherein the values derived from the fields in the responsive events are presented as data views.

22. The computer program product of claim 19, further including causing display of the values from or derived from the fields in the responsive events as values in rows in a pivot report template.

23. The computer program product of claim 19, further including causing display of unique values among the values from the fields in the responsive events as rows in a pivot report template.

24. The computer program product of claim 19, further including causing display of the values from the fields in the responsive events as values in rows of a pivot report interface and further causing display of statistics derived from the rows in the pivot report.

25. The computer program product of claim 19, further including causing display of the values from the fields in the responsive events as values in rows of a pivot report interface and further causing display of aggregates derived from values in the fields of the responsive events.

26. The computer program product of claim 19, further including:

nesting of the data model structure and submodel is further nested in a tree fashion to sub-sub models and sub-sub-sub models along some branches of the data model structure with inheritance of data constraints and available field extractions from levels down to further sub levels.