System and method for filling gaps of missing data using source specified data

• US 6,862,540 B1
• Filed: 03/25/2003
• Issued: 03/01/2005
• Est. Priority Date: 03/25/2003
• Status: Expired due to Term

• Current Assignee: Johnson Controls Tyco IP Holdings LLP (Johnson Controls International plc)
• Original Assignee: Johnson Controls Technology Company (Johnson Controls International plc)
• Inventors: Seem, John E., Welch, Henry L.
• Primary Examiner(s): Nghiem, Michael
• Assistant Examiner(s): Walling, Meagan S
• Application Number:
  US10/396,953
• Time in Patent Office: 707 Days
• Field of Search: 702/44, 702/60, 702/61, 702/182, 700/276, 340/870.02, 300/870.02
• US Class Current: 702/44
• CPC Class Codes: G01F 15/063 using electrical means
• CPC Class Codes: H04Q 2209/60 for transmitting utility meters data, i.e. transmission of data from the reader of the utility meter
• CPC Class Codes: H04Q 2209/84 Measuring functions
• CPC Class Codes: H04Q 9/00 Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom

Abstract

A method and apparatus are provided for filling a gap of missing data using source specified data. The method includes analyzing a first set of data representative of utility consumption for a plurality of days to determine clusters of days of the week having similar utility consumption. The method further includes generating an individual profile of utility consumption for each cluster, and copying data from at least one individual profile into the gap to form a second set of data without any gaps. A system for filling a gap of missing data using source specified data includes a microprocessor programmed to copy data from at least one individual profile into the gap, and to adjust the copied data up or down to account for local trend information adjacent to the gap.

Claims

1. A method for filling a gap of missing data using source specified data, comprising:

analyzing a first set of data representative of utility consumption for a plurality of days to determine clusters of days of the week having similar utility consumption;

generating an individual profile of utility consumption for each cluster;

copying data from at least one individual profile into the gap to form a second set of data without any gaps; and

identifying and removing abnormal utility consumption data from the first series of data,

wherein the identifying and removing step includes performing an outlier analysis on the data.

2. The method of claim 1, further including a step of gathering the first set of data using at least one sensor.

3. The method of claim 2, further including positioning the at least one sensor to monitor at least one of an electric meter, a gas meter, a water meter and a supply connection to a specific piece of equipment.

4. The method of claim 1, wherein the utility being consumed is at least one of electricity, gas and water.

5. The method of claim 1, further including reading the first set of data from a database contained within a memory of a computerized system.

6. The method of claim 1, further including receiving the data via a communication link.

7. The method of claim 6, wherein the communication link is at least one of a direct cable connection, a local area network, a wide area network, a telephone line, and a wireless connection.

8. The method of claim 1, wherein the first series of data is a time series of building utility consumption data.

9. The method of claim 8, wherein the building utility consumption data is at least one of electricity use, natural gas consumption, district heating consumption, cooling requirements, and heating requirements.

10. A method for filling a gap of missing data using source specified data, comprising:

analyzing a first set of data representative of utility consumption for a plurality of days to determine clusters of days of the week having similar utility consumption;

generating an individual profile of utility consumption for each cluster; and

copying data from at least one individual profile into the gap to form a second set of data without any gaps,

wherein the analyzing step includes performing a clustering algorithm.

11. The method of claim 10, wherein the clustering algorithm comprises a form of an agglomerative hierarchical clustering method.

12. A method for filling a gap of missing data using source specified data, comprising:

analyzing a first set of data representative of utility consumption for a plurality of days to determine clusters of days of the week having similar utility consumption;

generating an individual profile of utility consumption for each cluster; and

copying data from at least one individual profile into the gap to form a second set of data without any gaps,

wherein the copying step includes adjusting the individual profile using statistics computed from data adjacent to the gap.

13. The method of claim 12, wherein the step of adjusting the individual profile is accomplished by computing at least one statistical Z-score for data adjacent to the gap.

14. The method of claim 13, wherein computing the at least one statistical Z-score comprises determining a starting Z-score for pre-gap data and determining an ending Z-score for post-gap data.

15. The method of claim 14, further comprising computing a plurality of interpolated Z-scores between the starting and ending Z-scores using linear interpolation.

16. The method of claim 12, wherein the data adjacent to the gap comprises at least one of pre-gap data immediately before the gap and post-gap data immediately after the gap.

17. The method of claim 16, wherein the at least one of pre-gap data and post-gap data comprises one day'"'"'s worth of data before or after the gap, or one day each before and after the gap.

18. A method for filling a gap of missing data using source specified data, comprising:

analyzing a first set of data representative of utility consumption for a plurality of days to determine clusters of days of the week having similar utility consumption;

generating an individual profile of utility consumption for each cluster; and

copying data from at least one individual profile into the gap to form a second set of data without any gaps,

wherein the first set of data is a time series of sufficient duration that each individual profile includes about 15 days worth of data prior to the gap and about 15 days worth of data following the gap.

19. The method of claim 18, further including the step of increasing the size of the window prior to the gap when insufficient data is available following the gap.

20. A method for filling a gap of missing data using source specified data, comprising:

analyzing a first set of data representative of utility consumption for a plurality of days to determine clusters of days of the week having similar utility consumption;

generating an individual profile of utility consumption for each cluster;

copying data from at least one individual profile into the gap to form a second set of data without any gaps; and

displaying the second set of data on a display using indicia to distinguish the copied data.

21. The method of claim 20, wherein the distinguishing indicia includes at least one of color, line thickness, labels, and emphasis.

22. A method for filling a gap of missing data using source specified data, comprising:

analyzing a first set of data representative of utility consumption for a plurality of days to determine clusters of days of the week having similar utility consumption;

generating an individual profile of utility consumption for each cluster;

copying data from at least one individual profile into the gap to form a second set of data without any gaps; and

performing summary calculations on the second set of data including the copied data.

23. The method of claim 22, wherein the summary calculations include at least one of a demand charge, a time of day charge, and a consumption charge.

24. The method of claim 22, wherein the summary calculations include at least one of aggregation and averaging.

25. An apparatus for filling a gap of missing data using source specified data, comprising:

means for analyzing a first set of data representative of utility consumption for a plurality of days to determine clusters of days of the week having similar utility consumption;

means for generating an individual profile of utility consumption for each cluster; and

means for copying data from at least one individual profile into the gap to form a second set of data without any gaps,

wherein the means for analyzing comprises a microprocessor programmed to perform a clustering algorithm.

26. The apparatus of claim 25, wherein the clustering algorithm comprises a form of an agglomerative hierarchical clustering method.

27. An apparatus for filling a gap of missing data using source specified data, comprising:

means for analyzing a first set of data representative of utility consumption for a plurality of days to determine clusters of days of the week having similar utility consumption;

means for generating an individual profile of utility consumption for each cluster; and

means for copying data from at least one individual profile into the gap to form a second set of data without any gaps,

wherein the means for copying includes means for adjusting the individual profile using statistics computed from data adjacent to the gap.

28. The apparatus of claim 27, wherein the means for adjusting the individual profile includes a microprocessor programmed to compute at least one statistical Z-score for data adjacent to the gap.

29. The apparatus of claim 28, wherein the microprocessor is programmed to determine a starting Z-score for pre-gap data and an ending Z-score for post-gap data.

30. The apparatus of claim 29, wherein the microprocessor is programmed to determine a plurality of interpolated Z-scores between the starting and ending Z-scores using linear interpolation.

31. The apparatus of claim 28, wherein the data adjacent to the gap comprises at least one of pre-gap data immediately before the gap and post-gap data immediately after the gap.

32. The apparatus of claim 31, wherein the at least one of pre-gap data and post-gap data comprises one day'"'"'s worth of data before or after the gap, or one day each before and after the gap.

33. An apparatus for filling a gap of missing data using source specified data, comprising:

means for analyzing a first set of data representative of utility consumption for a plurality of days to determine clusters of days of the week having similar utility consumption;

means for generating an individual profile of utility consumption for each cluster; and

means for copying data from at least one individual profile into the gap to form a second set of data without any gaps,

wherein the first set of data is a time series of sufficient duration that each individual profile includes about 15 days worth of data prior to the gap and about 15 days worth of data following the gap.

34. A system for filling a gap of missing data using source specified data, the system comprising:

a programmed microprocessor configured to analyze a first set of data representative of utility consumption for a plurality of days to determine clusters of days of the week having similar utility consumption and to generate an individual profile of utility consumption data for each cluster, the microprocessor further configured to,

copy data from at least one individual profile into the gap; and

adjust the copied data up or down to account for local trend information adjacent to the gap.

35. The system of claim 34, wherein the microprocessor is programmed to obtain the local trend information from time series data immediately prior to and/or following the gap.

36. The system of claim 34, wherein the microprocessor is programmed to obtain the local trend information by determining at least one statistical Z-score for data adjacent to the gap.

37. The system of claim 36, wherein the microprocessor is programmed to determine a starting Z-score for pre-gap data and an ending Z-score for post-gap data.

38. The system of claim 37, wherein the microprocessor is programmed to determine a plurality of interpolated Z-scores between the starting and ending Z-scores using linear interpolation.

39. The system of claim 34, wherein the local trend information is based on approximately one day'"'"'s worth of data preceding the gap and approximately one day'"'"'s worth of data following the gap.

40. A method for filling a gap of missing data using source specified data, comprising:

analyzing a first set of data representative of utility consumption for a plurality of days to determine clusters of days of the week having similar utility consumption;

generating an individual profile of utility consumption for each cluster; and

copying data from at least one individual profile into the gap to form a second set of data without any gaps, wherein copying data includes adjusting the individual profile using at least one statistical Z-score computed for data adjacent to the gap.

41. The method of claim 40, wherein computing the at least one statistical Z-score comprises determining a starting Z-score for pre-gap data and determining an ending Z-score for post-gap data.

42. The method of claim 41, further comprising computing a plurality of interpolated Z-scores between the starting and ending Z-scores using linear interpolation.

43. The method of claim 40, wherein the data adjacent to the gap comprises at least one of pre-gap data immediately before the gap and post-gap data immediately after the gap.

44. The method of claim 43, wherein the at least one of pre-gap data and post-gap data comprises one day'"'"'s worth of data before or after the gap, or one day each before and after the gap.

45. A method for filling a gap of missing data using source specified data, comprising:

analyzing a first set of data representative of utility consumption for a plurality of days to determine clusters of days of the week having similar utility consumption, wherein the first set of data is a time series of sufficient duration that each individual profile includes about 15 days worth of data prior to the gap and about 15 days worth of data following the gap;

generating an individual profile of utility consumption for each cluster; and

copying data from at least one individual profile into the gap to form a second set of data without any gaps.

46. The method of claim 45, further including the step of increasing the size of the window prior to the gap when insufficient data is available following the gap.

47. A method for filling a gap of missing data using source specified data, comprising:

analyzing a first set of data representative of utility consumption for a plurality of days to determine clusters of days of the week having similar utility consumption;

generating an individual profile of utility consumption for each cluster;

copying data from at least one individual profile into the gap to form a second set of data without any gaps; and

displaying the second set of data on a display using indicia to distinguish the copied data.

48. The method of claim 47, wherein the distinguishing indicia includes at least one of color, line thickness, labels, and emphasis.

49. A method for filling a gap of missing data using source specified data, comprising:

analyzing a first set of data representative of utility consumption for a plurality of days to determine clusters of days of the week having similar utility consumption;

generating an individual profile of utility consumption for each cluster;

copying data from at least one individual profile into the gap to form a second set of data without any gaps; and

performing summary calculations on the second set of data including the copied data.

50. The method of claim 49, wherein the summary calculations include at least one of a demand charge, a time of day charge, and a consumption charge.

51. The method of claim 49, wherein the summary calculations include at least one of aggregation and averaging.

52. An apparatus for filling a gap of missing data using source specified data, comprising:

means for analyzing a first set of data representative of utility consumption for a plurality of days to determine clusters of days of the week having similar utility consumption;

means for generating an individual profile of utility consumption for each cluster; and

means for copying data from at least one individual profile into the gap to

form a second set of data without any gaps, wherein the means for copying includes means for adjusting the individual profile using statistics computed from data adjacent to the gap, and wherein the means for adjusting the individual profile includes a microprocessor programmed to compute at least one statistical Z-score for data adjacent to the gap.

53. The apparatus of claim 52, wherein the microprocessor is programmed to determine a starting Z-score for pre-gap data and an ending Z-score for post-gap data.

54. The apparatus of claim 53, wherein the microprocessor is programmed to determine a plurality of interpolated Z-scores between the starting and ending Z-scores using linear interpolation.

55. The apparatus of claim 52, wherein the data adjacent to the gap comprises at least one of pre-gap data immediately before the gap and post-gap data immediately after the gap.

56. The apparatus of claim 55, wherein the at least one of pre-gap data and post-gap data comprises one day'"'"'s worth of data before or after the gap, or one day each before and after the gap.

57. An apparatus for filling a gap of missing data using source specified data, comprising:

means for analyzing a first set of data representative of utility consumption for a plurality of days to determine clusters of days of the week having similar utility consumption, wherein the first set of data is a time series of sufficient duration that each individual profile includes about 15 days worth of data prior to the gap and about 15 days worth of data following the gap;

means for generating an individual profile of utility consumption for each cluster; and

means for copying data from at least one individual profile into the gap to form a second set of data without any gaps.

58. A system for filling a gap of missing data using source specified data, the system including a programmed microprocessor configured to analyze a first set of data representative of utility consumption for a plurality of days to determine clusters of days of the week having similar utility consumption and to generate an individual profile of utility consumption data for each cluster, the microprocessor further configured to:

copy data from at least one individual profile into the gap; and

adjust the copied data up or down to account for local trend information adjacent to the gap, wherein the microprocessor is programmed to obtain the local trend information from time series data immediately prior to and/or following the gap.

59. A system for filling a gap of missing data using source specified data, the system including a programmed microprocessor configured to analyze a first set of data representative of utility consumption for a plurality of days to determine clusters of days of the week having similar utility consumption and to generate an individual profile of utility consumption data for each cluster, the microprocessor further configured to:

copy data from at least one individual profile into the gap; and

adjust the copied data up or down to account for local trend information adjacent to the gap, wherein the microprocessor is programmed to obtain the local trend information by determining at least one statistical Z-score for data adjacent to the gap.

60. The system of claim 59, wherein the microprocessor is programmed to determine a starting Z-score for pre-gap data and an ending Z-score for post-gap data.

61. The system of claim 60, wherein the microprocessor is programmed to determine a plurality of interpolated Z-scores between the starting and ending Z-scores using linear interpolation.

62. A system for filling a gap of missing data using source specified data, the system including a programmed microprocessor configured to analyze a first set of data representative of utility consumption for a plurality of days to determine clusters of days of the week having similar utility consumption and to generate an individual profile of utility consumption for each cluster, the microprocessor further configured to:

copy data from at least one individual profile into the gap; and

adjust the copied data up or down to account for local trend information adjacent to the gap, wherein the local trend information is based on approximately one day'"'"'s worth of data preceding the gap and approximately one day'"'"'s worth of data following the gap.