Method and system to identify discrete trends in time series

• US 20030009399A1
• Filed: 03/22/2001
• Published: 01/09/2003
• Est. Priority Date: 03/22/2001
• Status: Active Grant

• Current Assignee: Sean T. Boerner
• Original Assignee: Sean T. Boerner
• Inventors: Boerner, Sean T.
• Granted Patent: US 7,818,224 B2
• Time in Patent Office: Days
• US Class Current: 705/35
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06Q 40/00 Finance; Insurance; Tax strategies; Processing of corporate or income taxes
• CPC Class Codes: G06Q 40/04 Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange

Abstract

A signal processing system and method for breaking a time series into piece-wise discrete trends and determining whether new data represents the continuation of a trend. The method identifies and utilizes at least one set of trend determination parameters, which have favorable trend fit characteristics relative to other possible sets of parameters. In a semi-automated embodiment, the error and trend length characteristics are cross-plotted for multiple sets of possible parameters, and one or more of the parameter sets is selected from the graph. In an automated embodiment, an objective function is formulated from the characteristics, and an optimization technique is applied to identify one or more good parameter sets.

Claims

1. A system, for use in an information processing apparatus, for breaking a time series into a plurality of discontinuous trends, the system comprising:

input means for inputting time series data, the time series comprising a plurality of data elements;

processing means for

generating a plurality of sets of trend determination parameters for the time series;

selecting a useful group of sets from the plurality of sets of trend determination parameters for the time series, such that the useful group of sets includes at least one member;

processing the time series with each member of the useful group of sets of trend determination parameters to generate a set of trends and trend attributes for each member; and

output means for outputting the trends for each member.

2. The system of claim 1 wherein the processing means further comprises deciding, based on the composite results of a plurality of the members of the useful group of sets of trend determination parameters , whether the newest data element of the time series represents a continuation of a dynamic trend.

3. The system of claim 1 wherein the input means for inputting time series data further comprises

inputting a plurality of time series data sets to a computer;

and

selecting a particular time series from a plurality of time series data sets.

4. The system of claim 1 wherein the input means for inputting time series data further comprises

inputting a plurality of of vector datasets to a computer;

and

selecting a particular vector data set from a plurality of time series data sets.

5. The system of claim 3 wherein selecting a particular time series from a plurality of time series data sets is comprised of

for each of the plurality of time series data sets:

selecting at least a portion of the elements in the data set to create a selected data subset;

normalizing the selected data subset to generate a normalized subset for the time series;

storing the normalized subset on the computer;

calculating, on a processor, the slope of a best-fit polynomial regression through the normalized subset; and

selecting a particular time series that has a large absolute slope and a large correlation coefficient between the trend and the data elements.

6. The system of claim 1 wherein generating a plurality of sets of trend determination parameters is comprised of

specifying a range of values for each of a plurality of trend determination parameters;

and

generating the sets of trend determination parameters by selecting unique combinations of trend determination parameter values, such that the values are within the range of values for each of the plurality of trend determination parameters.

7. The system of claim 6 wherein specifying a range of values for each of a plurality of trend determination parameters is comprised of

specifying a minimum value for a first trend determination parameter of initial data window size;

specifying a maximum value for a first trend determination parameter of initial data window size;

specifying a minimum value for a second trend determination parameter of deviation limit; and

specifying a maximum value for a second trend determination parameter of deviation limit.

8. The system of claim 1 wherein generating a plurality of sets of trend determination parameters for the time series and selecting a useful group of sets from the plurality of sets of trend determination parameters for the time series, such that the useful group of sets includes at least one member is comprised of:

specifying a range of potential values for each of a plurality of trend determination parameters;

creating an objective function from at least one indications of trend results, such that the objective function generates a resultant value for a set of trend determination parameters; and

selecting a useful group of sets of trend determination parameters by applying an optimization procedure to the objective function and the range of potential values for each of the plurality of trend determination parameters.

9. The system of claim 1 wherein selecting a useful group of sets from the plurality of sets of trend determination parameters for the time series is comprised of:

for each trend determination parameter set;

applying the trend determination parameters to at least a portion of the time series data elements, thereby generating at least one trend and at least one trend attribute parameter;

identifying the dynamic trend and at least one dynamic trend attribute parameter. calculating at least one indication of trend results between the time series and the trend set; and

storing at least one indication of trend results on the computer;

selecting at least one set of trend determination parameters based on at least one indication of trend results for each of the sets.

10. The system of claim 9 wherein applying the trend determination parameters to at least a portion of the time series data elements thereby generating at least one trend and at least one trend attribute parameter is comprised of

assigning each of the data elements to at least one trend by

generating a first trend with at least a portion of the data elements;

identifying the first trend as the current trend;

evaluating each subsequent data element to determine whether the data element is a continuation of the current trend, and

assigning the data element to the current trend if it is a continuation of the current trend, and

assigning the data element to a new trend if it is not continuation of the current trend, and identifying the new trend as the current trend; and

determining at least one trend attribute for each trend.

11. The system of claim 9 wherein calculating at least one indication of trend results between the time series and trend set is comprised of calculating at least one measure from the group consisting of:

the number of trends in the subset of the time series,

the RMS Error between the input data values and trend values,

the average trend length;

the average trend length divided by the minimum number of data points needed to define

a trend (window size parameter),

the average percent return of the trends,

the summed cumulative percent return of the trends;

the fraction of correct predictions,

the fraction of incorrect predictions,

the quotient of the root mean square error and the average length of the trends divided by the minimum number of data points needed to define a trend,

and the RMS error*L_s

the efficiency of the trends, where efficiency is defined as the average return of the trends divided by the average length of the trends, and

compounded return of the trends.

12. The system of claim 9 wherein selecting at least one set of trend determination parameters based on at least one indication of trend results for each of the sets is comprised of

plotting on a graph a representative data point for each of the plurality of sets of trend determination parameters and its associated trend results, where a first axis represents a first measure of trend results and a second axis represents a second measure of trend results;

and

selecting from the graph at least one data point that represents a trend determination parameters set that has desirable trend results.

13. The system of claim 12 wherein plotting on a graph a representative data point for each of the plurality of sets of trend determination parameters and their associated trend results and selecting from the graph at least one data point that represents a parameters set that has desirable trend results is further comprised of:

plotting on a graph a representative data point for each of the plurality of sets of trend determination parameters, where the y-axis represents the product of the root mean square error and the deviation limit for the set of trend determination parameters, and the x-axis represents the average trend length for the set of trend determination parameters;

selecting at least one set of trend determination parameters associated with a point from the graph that has a minimum value for the product of the root mean square error and the deviation limit for a given average trend length;

14. The system of claim 12 wherein plotting on a graph a representative data point for each of the plurality of sets of trend determination parameters and their associated trend results and selecting from the graph at least one data point that represents a parameters set that has desirable trend results is further comprised of:

plotting on a graph a representative data point for each of the plurality of sets of trend determination parameters, where the y-axis represents the root mean square error for the set of trend determination parameters, and x-axis represents the average trend length for the set of trend determination parameters;

selecting at least one set of trend determination parameters associated with a point from the graph that has a minimum value for the root mean square error for a given average trend length;

15. The system of claim 12 wherein plotting on a graph a representative data point for each of the plurality of sets of trend determination parameters and their associated trend results and selecting from the graph at least one data point that represents a parameters set that has desirable trend results is further comprised of:

plotting on a graph a representative data point for each of the plurality of sets of trend determination parameters, where the y-axis represents the summed cumulative percent return of the trends for the set of trend determination parameters, and x-axis represents the average percent return for the set of trend determination parameters;

selecting at least one set of trend determination parameters associated with a point from the graph that has a maximum value for both the summed cumulative percent return and the average percent return;

16. The system of claim 12 wherein plotting on a graph a representative data point for each of the plurality of sets of trend determination parameters and their associated trend results and selecting from the graph at least one data point that represents a parameters set that has desirable trend results is further comprised of:

plotting on a graph a representative data point for each of the plurality of sets of trend determination parameters, where the y-axis represents the summed cumulative percent return of the trends for the set of trend determination parameters, and x-axis represents the fraction of correct predictions for the set of trend determination parameters;

selecting at least one set of trend determination parameters associated with a point from the graph that has a maximum value for both the summed cumulative percent return and the fraction of correct predictions;

17. The system of claim 12 wherein plotting on a graph a representative data point for each of the plurality of sets of trend determination parameters and their associated trend results and selecting from the graph at least one data point that represents a parameters set that has desirable trend results is further comprised of:

plotting on a graph a representative data point for each of the plurality of sets of trend determination parameters, where the y-axis represents the average percent return of the trends for the set of trend determination parameters, and x-axis represents the fraction of correct predictions for the set of trend determination parameters;

selecting at least one set of trend determination parameters associated with a point from the graph that has a maximum value for both the average percent return and the fraction of correct predictions;

18. The system of claim 12 wherein plotting on a graph a representative data point for each of the plurality of sets of trend determination parameters and their associated trend results and selecting from the graph at least one data point that represents a parameters set that has desirable trend results is further comprised of:

plotting on a graph a representative data point for each of the plurality of sets of trend determination parameters, where the y-axis represents the average percent return for the trends for the set of trend determination parameters, and x-axis represents the average trend length for the set of trend determination parameters;

selecting at least one set of trend determination parameters associated with a point from the graph that has a maximum value for the average percent return for a given average trend length;

19. The system of claim 9 wherein selecting at least one set of trend determination parameters based on at least one indication of trend results for each of the sets is comprised of

specifying an objective function that incorporates at least one measure of trend results, such that minimization of the objective function produces desirable trend results;

applying an optimization technique to the objective function such that the optimization technique minimizes the objective function; and

selecting at least one set of trend determination parameters as a result of minimizing the objective function.

20. The system of claim 1 wherein processing the time series with each member of the useful group of sets of trend determination parameters to generate a set of trends and trend attributes for each member is comprised of

for each member

applying the trend determination parameters to at least a portion of the time series data elements, thereby generating at least one trend and at least one trend attribute parameter;

and

identifying the dynamic trend and at least one dynamic trend attribute parameter; and

storing at least one dynamic trend attribute parameter on the computer.

21. The system of claim 20 wherein applying the trend determination parameters to at least a portion of the time series data elements thereby generating at least one trend and at least one trend attribute parameter is comprised of

assigning each of the data elements to at least one trend by

generating a first trend with at least a portion of the data elements;

identifying the first trend as the current trend;

evaluating each subsequent data element to determine whether the data element is a continuation of the current trend, and

assigning the data element to the current trend if it is a continuation of the current trend, and

assigning the data element to a new trend if it is not continuation of the current trend, and identifying the new trend as the current trend; and

determining at least one trend attribute for each trend.

22. The system of claim 21 wherein generating a first trend with at least a portion of the data elements is comprised of

recalling the first trend determination parameter, the data window size, from the set of trend determination parameters;

forming a proposed trend data set by selecting a number of data elements from the time series, such that the number of data elements selected is at least as large as the value of the data window size trend determination parameter;

calculating a first best-fit curve through the proposed trend data set; and

identifying the best fit curve as the first trend.

23. The system of claim 21 wherein evaluating each subsequent data element to determine whether the data element is a continuation of the current trend is comprised of

forming a proposed trend data set from the selected data elements;

calculating a new best-fit curve through the proposed trend data set;

calculating at least one measure of predictive error for the new best-fit curve with respect to the values of the data elements in the proposed trend data set;

projecting the best-fit curve to the location of the subsequent data element;

evaluating the deviation of the subsequent data element from the projected best-fit curve value at the new location; and

applying at least one acceptance criteria to the measure of predictive error, and

if the acceptance criteria is met

setting the subsequent data element to the proposed trend data set,

identifying the new best-fit curve as the current trend, and

if the acceptance criteria is not met

setting the subsequent data element to a new trend set,

determining a new trend, and

identifying the new trend as the current trend.

24. The system of claim 23 wherein calculating at least one measure of predictive error for the new best-fit curve with respect to the values of the data elements in the proposed data set is comprised of

evaluating the derivative of the best-fit curve;

obtaining an estimated value of the best-fit curve at the new element;

calculating the residuals between the proposed trend data set and the new best-fit curve;

normalizing the residuals;

quantifying the spread of the distribution of the normalized residuals;

calculating the deviation of predicted trend value at the new element from the actual value of the new element; and

normalizing the deviation of predicted trend value at the new element from the actual value of the new element using the spread of the distribution of normalized residuals.

25. The system of claim 23 wherein applying at least one acceptance criteria to the measure of predictive error is comprised of

designating a first test criterion as true if the sign of the derivative of the trend curve generated through the proposed trend dataset changes compared to the sign of the derivative of the trend curve generated previously;

designating a second test criterion as true if the absolute value of normalized deviation exceeds the deviation limit parameter;

designating a third test criterion as true if the absolute value of normalized deviation exceeds the deviation limit parameter and the normalized deviation is in the opposite direction as the direction of the trend as designated by the derivative of the trend curve through the proposed trend dataset;

designating a fourth test criterion as true if the correlation coefficient between the proposed trend curve and the proposed trend dataset decreases below the correlation limit parameter;

designating a fifth test criterion as true if the number of times that

the absolute value of normalized deviation exceeds the deviation limit parameter; and

the normalized deviation is in the opposite direction as the direction of the trend as designated by the derivative of the trend curve through the proposed trend dataset;

exceeds the number of values parameter;

designating a sixth test criterion as true if the absolute value of the normalized deviation from a flat trend exceeds the flat trend deviation limit; and

determining whether at least one of the first, second, third, fourth, fifth, and sixth test criteria is true.

26. A system for use in an information processing apparatus for breaking a time series into a plurality of discontinuous trends and determining if a new data element represents a continuation of the latest trend the system comprising:

input means for inputting time series data, the time series comprising a plurality of data elements;

processing means for

generating a plurality of sets of trend determination parameters for the time series by

specifying a range of values for each of a plurality of trend determination parameters, and

generating the sets of trend determination parameters by selecting unique combinations of trend determination parameter values, such that the values are within the range of values for each of the plurality of trend determination parameters;

selecting a useful group of sets from the plurality of sets of trend determination parameters for the time series, such that the useful group of sets includes at least one member by,

for each trend determination parameter set;

applying the trend determination parameters to at least a portion of the time series data elements, thereby generating at least one trend and at least one trend attribute parameter,

identifying the dynamic trend and at least one dynamic trend attribute parameter, calculating at least one indication of trend results between the time series and the trend set, and

storing at least one indication of trend results on the computer,

selecting at least one set of trend determination parameters based on at least one indication of trend results for each of the sets;

processing the time series with each member of the useful group of sets of trend determination parameters to generate a set of trends and trend attributes for each member by

applying the trend determination parameters to at least a portion of the time series data elements, thereby generating at least one trend and at least one trend attribute parameter,

assigning each of the data elements to at least one trend by

generating a first trend with at least a portion of the data elements;

identifying the first trend as the current trend;

evaluating each subsequent data element to determine whether the data element is a continuation of the current trend, and

assigning the data element to the current trend if it is a continuation of the current trend, and

assigning the data element to a new trend if it is not a continuation of the current trend, and identifying the new trend as the current trend; and

determining at least one trend attribute for each trend, and

identifying the dynamic trend and at least one dynamic trend attribute parameter;

evaluating at least one new data element for the time series by determining for each member whether the new data element represents a continuation of a dynamic trend;

deciding, based on the composite results of the members, whether the data element represents a continuation of a dynamic trend; and

output means for outputting the trends for each member.