Computer-implemented systems and methods for efficient structuring of time series data

• US 9,244,887 B2
• Filed: 07/13/2012
• Issued: 01/26/2016
• Est. Priority Date: 07/13/2012
• Status: Active Grant

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: Leonard, Michael James, Crowe, Keith Eugene, Christian, Stacey M., Beeman, Jennifer Leigh Sloan, Elsheimer, David Bruce, Blair, Edward Tilden
• Primary Examiner(s): Obisesan, Augustine K
• Assistant Examiner(s): Brooks, David T
• Application Number:
  US13/548,282
• Publication Number:
  US 20140019448A1
• Time in Patent Office: 1,292 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/3452 Performance evaluation by statistical analysis
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06Q 10/04 Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" market predictions or forecasting for commercial activities G06Q30/0202

Abstract

Systems and methods are provided for analyzing through one-pass of unstructured time stamped data of a physical process. A distribution of time-stamped unstructured data is analyzed to identify a plurality of potential hierarchical structures for the unstructured data. A hierarchical analysis of the potential hierarchical structures is performed to determine an optimal frequency and a data sufficiency metric for the potential hierarchical structures. One of the potential hierarchical structures is selected as a selected hierarchical structure based on the data sufficiency metrics. The unstructured data is structured according to the selected hierarchical structure and the optimal frequency associated with the selected hierarchical structure, where said structuring of the unstructured data is performed via a single pass though the unstructured data. The identified statistical analysis of the physical process is performed using the structured data.

Claims

1. A computer-implemented method comprising:

analyzing, using a time series engine, a distribution of unstructured time-stamped data to identify a plurality of potential time series data hierarchies for structuring the unstructured time-stamped data,

wherein a potential time series data hierarchy is a framework for structuring the data using of multiple time series, and

wherein the time series engine is at a server layer of a time series computing system;

performing, using the time series engine, an analysis of the plurality of potential time series data hierarchies, wherein performing the analysis of the plurality of potential time series data hierarchies includes determining an optimal time series frequency and a data sufficiency metric for each of the plurality of potential time series data hierarchies;

comparing data sufficiency metrics for the plurality of potential time series data hierarchies;

selecting a hierarchy of the plurality of potential time series data hierarchies based on the comparison of the data sufficiency metrics;

structuring the unstructured time-stamped data into structured time-stamped data according to the hierarchy and the optimal time series frequency,

wherein structuring the transformed time-stamped data into the structured time-stamped data is performed using a single pass of the unstructured time-stamped data through the time series engine;

computing a plurality of transformations of the structured time-stamped data using the single pass of the structured time-stamped data through the time series engine;

transforming the structured time-stamped data into transformed time-stamped data according to the plurality of transformations; and

providing, using an application programming interface, the transformed time-stamped data for visual presentation.

2. The method of claim 1, further comprising:

deriving a time series using the transformed time-stamped data; and

analyzing the derived time series using one or more time series analysis functions.

3. The method of claim 2, further comprising:

comparing the derived time series to a time series of interest to identify a similar portion of the derived time series; and

extracting the similar portion of the derived time series, wherein a selected time series analysis function operates on the extracted similar portion.

4. The method of claim 2, further comprising:

identifying a characteristic of the derived time series; and

selecting a data model for a time series analysis function based on the identified characteristic, wherein the time series analysis function is performed using the selected data model, and wherein the identified characteristic is a seasonal pattern, a trend pattern, a growth pattern, or a delay pattern.

5. The method of claim 2, wherein a time series analysis function is applied to the derived time series, and wherein the time series analysis function comprises a forecasting function.

6. The method of claim 5, wherein the derived time series are provided to a forecasting application for forecasting using the time series analysis function.

7. The method of claim 1, wherein the unstructured time-stamped data is analyzed by applying a seasonality test, an intermittency test, or a trending data test.

8. The method of claim 1, wherein the unstructured time-stamped data is analyzed by applying a user defined test.

9. The method of claim 1, further comprising:

providing a first portion of the derived time series to a first processor for performing a statistical analysis; and

providing a second portion of the derived time series to a second processor for performing the statistical analysis, wherein the first portion and the second portion are based upon a portion of a hierarchy in which the first portion and the second portion reside.

10. The method of claim 1, further comprising:

selecting additional hierarchies of the plurality of potential time series data hierarchies based on the comparison of the data sufficiency metrics; and

structuring the unstructured time-stamped data into structured time-stamped data according to the additional hierarchies and the optimal time series frequency, wherein structuring the unstructured time-stamped data using a hierarchy of the additional hierarchies is executed with a corresponding process thread.

11. The method of claim 1, further comprising:

outputting, using the time series engine, information corresponding to the structured data, wherein outputting the information is performed using the single pass of the time stamped unstructured data.

12. The method of claim 1, further comprising:

generating, using the time series engine, an electronic representation of the structured data using the single pass of the time stamped unstructured data.

13. The method of claim 1, wherein analyzing the distribution of the unstructured time-stamped data includes performing a time frequency analysis on the unstructured data using the time series engine.

14. The method of claim 1, wherein analyzing the distribution of the unstructured time-stamped data includes performing a data aggregation frequency analysis on the unstructured data using the time series engine.

15. The method of claim 1, wherein analyzing the distribution of the unstructured time-stamped data includes performing a cluster analysis of the unstructured data using the time series engine.

16. The method of claim 1, wherein determining an optimal time series frequency includes aggregating the unstructured time-stamped data into aggregated time-stamped data using the time series engine, and wherein the aggregating is performed using a potential hierarchical structure and a plurality of candidate frequencies for the potential hierarchical structure.

17. The method of claim 16, wherein determining a data sufficiency metric uses the aggregated time-stamped data.

18. The method of claim 1, further comprising:

providing a recommendation according to the analysis of the plurality of potential time series data hierarchies, wherein the recommendation is provided using a graphical user interface; and

receiving user input associated with the recommendation, wherein the unstructured time-stamped data is structured according to the user input.

45. The machine-readable non-transitory storage medium of claim 16, further comprising instructions configured to cause a data processing system to perform operations including:

providing a first portion of the derived time series to a first processor for performing a statistical analysis; and

providing a second portion of the derived time series to a second processor for performing the statistical analysis, wherein the first portion and the second portion are based upon a portion of a hierarchy in which the first portion and the second portion reside.

19. A system comprising:

one or more processors;

one or more computer-readable storage mediums containing instructions configured to cause the one or more processors to perform operations including;

analyzing, using a time series engine, a distribution of unstructured time-stamped data to identify a plurality of potential time series data hierarchies for structuring the unstructured time-stamped data,

wherein a potential time series data hierarchy is a framework for structuring the data through use of multiple time series, and

wherein the time series engine is at a server layer of a time series computing system;

performing, using the time series engine, an analysis of the plurality of potential time series data hierarchies, wherein performing the analysis of the plurality of potential time series data hierarchies includes determining an optimal time series frequency and a data sufficiency metric for each of the plurality of potential time series data hierarchies;

comparing data sufficiency metrics for the plurality of potential time series data hierarchies;

selecting a hierarchy of the plurality of potential time series data hierarchies based on the comparison of the data sufficiency metrics;

structuring the unstructured time-stamped data into structured time-stamped data according to the hierarchy and the optimal time series frequency,

wherein structuring the transformed time-stamped data into the structured time-stamped data is performed using a single pass of the unstructured time-stamped data through the time series engine;

computing a plurality of transformations of the structured time-stamped data using the single pass of the structured time-stamped data through the time series engine;

transforming the structured time-stamped data into transformed time-stamped data according to the plurality of transformations; and

providing, using an application programming interface, the transformed time-stamped data for visual presentation.

20. The system of claim 19, further comprising instructions, which when executed by the one or more processors, cause the computing device to perform operations including:

deriving a time series using the transformed time-stamped data; and

analyzing the derived time series using one or more time series analysis functions.

21. The system of claim 20, further comprising instructions, which when executed by the one or more processors, cause the computing device to perform operations including:

comparing the derived time series to a time series of interest to identify a similar portion of the derived time series; and

extracting the similar portion of the derived time series, wherein a selected time series analysis function operates on the extracted similar portion.

22. The system of claim 20, further comprising instructions, which when executed by the one or more processors, cause the computing device to perform operations including:

identifying a characteristic of the derived time series; and

selecting a data model for a time series analysis function based on the identified characteristic, wherein the time series analysis function is performed using the selected data model, and wherein the identified characteristic is a seasonal pattern, a trend pattern, a growth pattern, or a delay pattern.

23. The system of claim 20, wherein a time series analysis function is applied to the derived time series, and wherein the time series analysis function comprises a forecasting function.

24. The system of claim 23, wherein the derived time series are provided to a forecasting application for forecasting using the time series analysis function.

25. The system of claim 19, wherein the unstructured time-stamped data is analyzed by applying a seasonality test, an intermittency test, or a trending data test.

26. The system of claim 19, wherein the unstructured time-stamped data is analyzed by applying a user defined test.

27. The system of claim 19, further comprising instructions, which when executed by the one or more processors, cause the computing device to perform operations including:

providing a first portion of the derived time series to a first processor for performing a statistical analysis; and

providing a second portion of the derived time series to a second processor for performing the statistical analysis, wherein the first portion and the second portion are based upon a portion of a hierarchy in which the first portion and the second portion reside.

28. The system of claim 19, further comprising instructions, which when executed by the one or more processors, cause the computing device to perform operations including:

selecting additional hierarchies of the plurality of potential time series data hierarchies based on the comparison of the data sufficiency metrics; and

structuring the unstructured time-stamped data into structured time-stamped data according to the additional hierarchies and the optimal time series frequency, wherein structuring the unstructured time-stamped data using a hierarchy of the additional hierarchies is executed with a corresponding process thread.

29. The system of claim 19, further comprising instructions, which when executed by the one or more processors, cause the computing device to perform operations including:

outputting, using the time series engine, information corresponding to the structured data, wherein outputting the information is performed using the single pass of the time stamped unstructured data.

30. The system of claim 19, further comprising instructions, which when executed by the one or more processors, cause the computing device to perform operations including:

generating, using the time series engine, an electronic representation of the structured data using the single pass of the time stamped unstructured data.

31. The system of claim 19, wherein analyzing the distribution of the unstructured time-stamped data includes performing a time frequency analysis on the unstructured data using the time series engine.

32. The system of claim 19, wherein analyzing the distribution of the unstructured time-stamped data includes performing a data aggregation frequency analysis on the unstructured data using the time series engine.

33. The system of claim 19, wherein analyzing the distribution of the unstructured time-stamped data includes performing a cluster analysis of the unstructured data using the time series engine.

34. The system of claim 19, wherein determining an optimal time series frequency includes aggregating the unstructured time-stamped data into aggregated time-stamped data using the time series engine, and wherein the aggregating is performed using a potential hierarchical structure and a plurality of candidate frequencies for the potential hierarchical structure.

35. The system of claim 34, wherein determining a data sufficiency metric uses the aggregated time-stamped data.

36. The system of claim 19, further comprising instructions, which when executed by the one or more processors, cause the computing device to perform operations including:

providing a recommendation according to the analysis of the plurality of potential time series data hierarchies, wherein the recommendation is provided using a graphical user interface; and

receiving user input associated with the recommendation, wherein the unstructured time-stamped data is structured according to the user input.

37. A machine-readable non-transitory storage medium, including instructions configured to cause a data processing system to perform operations including:

analyzing, using a time series engine, a distribution of unstructured time-stamped data to identify a plurality of potential time series data hierarchies for structuring the unstructured time-stamped data,

wherein a potential time series data hierarchy is a framework for structuring the data through use of multiple time series, and

wherein the time series engine is at a server layer of a time series computing system;

performing, using the time series engine, an analysis of the plurality of potential time series data hierarchies, wherein performing the analysis of the plurality of potential time series data hierarchies includes determining an optimal time series frequency and a data sufficiency metric for each of the plurality of potential time series data hierarchies;

comparing data sufficiency metrics for the plurality of potential time series data hierarchies;

selecting a hierarchy of the plurality of potential time series data hierarchies based on the comparison of the data sufficiency metrics;

structuring the unstructured time-stamped data into structured time-stamped data according to the hierarchy and the optimal time series frequency,

wherein structuring the transformed time-stamped data into the structured time-stamped data is performed using a single pass of the unstructured time-stamped data through the time series engine;

computing a plurality of transformations of the structured time-stamped data using the single pass of the structured time-stamped data through the time series engine;

transforming the structured time-stamped data into transformed time-stamped data according to the plurality of transformations; and

providing, using an application programming interface, the transformed time-stamped data for visual presentation.

38. The machine-readable non-transitory storage medium of claim 37, further comprising instructions configured to cause a data processing system to perform operations including:

deriving a time series using the transformed time-stamped data; and

analyzing the derived time series using one or more time series analysis functions.

39. The machine-readable non-transitory storage medium of claim 38, further comprising instructions configured to cause a data processing system to perform operations including:

comparing the derived time series to a time series of interest to identify a similar portion of the derived time series; and

extracting the similar portion of the derived time series, wherein a selected time series analysis function operates on the extracted similar portion.

40. The machine-readable non-transitory storage medium of claim 38, further comprising instructions configured to cause a data processing system to perform operations including:

identifying a characteristic of the derived time series; and

selecting a data model for a time series analysis function based on the identified characteristic, wherein the time series analysis function is performed using the selected data model, and wherein the identified characteristic is a seasonal pattern, a trend pattern, a growth pattern, or a delay pattern.

41. The machine-readable non-transitory storage medium of claim 38, wherein a time series analysis function is applied to the derived time series, and wherein the time series analysis function comprises a forecasting function.

42. The machine-readable non-transitory storage medium of claim 41, wherein the derived time series are provided to a forecasting application for forecasting using the time series analysis function.

43. The machine-readable non-transitory storage medium of claim 37, wherein the unstructured time-stamped data is analyzed by applying a seasonality test, an intermittency test, or a trending data test.

44. The machine-readable non-transitory storage medium of claim 37, wherein the unstructured time-stamped data is analyzed by applying a user defined test.

46. The machine-readable non-transitory storage medium of claim 37, further comprising instructions configured to cause a data processing system to perform operations including:

selecting additional hierarchies of the plurality of potential time series data hierarchies based on the comparison of the data sufficiency metrics; and

structuring the unstructured time-stamped data into structured time-stamped data according to the additional hierarchies and the optimal time series frequency, wherein structuring the unstructured time-stamped data using a hierarchy of the additional hierarchies is executed with a corresponding process thread.

47. The machine-readable non-transitory storage medium of claim 37, further comprising instructions configured to cause a data processing system to perform operations including:

outputting, using the time series engine, information corresponding to the structured data, wherein outputting the information is performed using the single pass of the time stamped unstructured data.

48. The machine-readable non-transitory storage medium of claim 37, further comprising instructions configured to cause a data processing system to perform operations including:

generating, using the time series engine, an electronic representation of the structured data using the single pass of the time stamped unstructured data.

49. The machine-readable non-transitory storage medium of claim 37, wherein analyzing the distribution of the unstructured time-stamped data includes performing a time frequency analysis on the unstructured data using the time series engine.

50. The machine-readable non-transitory storage medium of claim 37, wherein analyzing the distribution of the unstructured time-stamped data includes performing a data aggregation frequency analysis on the unstructured data using the time series engine.

51. The machine-readable non-transitory storage medium of claim 37, wherein analyzing the distribution of the unstructured time-stamped data includes performing a cluster analysis of the unstructured data using the time series engine.

52. The machine-readable non-transitory storage medium of claim 37, wherein determining an optimal time series frequency includes aggregating the unstructured time-stamped data into aggregated time-stamped data using the time series engine, and wherein the aggregating is performed using a potential hierarchical structure and a plurality of candidate frequencies for the potential hierarchical structure.

53. The machine-readable non-transitory storage medium of claim 52, wherein determining a data sufficiency metric uses the aggregated time-stamped data.

54. The machine-readable non-transitory storage medium of claim 37, further comprising instructions configured to cause a data processing system to perform operations including:

providing a recommendation according to the analysis of the plurality of potential time series data hierarchies, wherein the recommendation is provided using a graphical user interface; and

receiving user input associated with the recommendation, wherein the unstructured time-stamped data is structured according to the user input.