Data pre-processing and indexing for efficient retrieval and enhanced presentation

• US 8,954,377 B1
• Filed: 05/25/2012
• Issued: 02/10/2015
• Est. Priority Date: 03/26/2009
• Status: Active Grant

• Current Assignee: Federal Express Corporation (FedEx Corporation)
• Original Assignee: Moog Incorporated
• Inventors: Turon, Martin A., Kapur, Rahul
• Primary Examiner(s): Mamillapalli, Pavan
• Application Number:
  US13/481,531
• Time in Patent Office: 991 Days
• Field of Search: 707/754, 707/602, 707/603
• US Class Current: 707/602
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24539 using cached or materialised query results
• CPC Class Codes: G06F 16/2477 Temporal data queries

Abstract

A system, a method and a computer readable storage medium for pre-processing data collected from one or more data sources more accurately summarize data. During the pre-processing, multiple raw data are summarized into a pre-processed datum. By using the pre-processed data entries, more accurate trend data may be generated. Alternatively, data entries are indexed and selectively retrieved based on indices. Decimation of data points are performed based on the indices without retrieving all the data sets from the database, reducing the data access time for returning a query result. Additional data sets may also be retrieved efficiently from the database using the indices.

Claims

1. A computer-implemented method for displaying a visual representation of data on a screen, comprising:

receiving a first set of data from a first data source, the first set of data having data entries, each data entry being associated with a different time of a first time interval;

keying data entries of the first set of data, each keyed data entry being associated with a different count-based index;

storing the keyed data entries;

retrieving two or more of the keyed data entries using the count-based indices, and summarizing those two or more of the retrieved data entries for the first set of data received over a predetermined time interval to generate a first pre-processed datum representing a summarized datum for the first data source over the predetermined time interval, the predetermined time interval being shorter than the first time interval;

storing the first pre-processed datum in data storage;

retrieving the first pre-processed datum responsive to receiving a request for the first set of data covering the predetermined time interval; and

displaying a visual representation of the first set of data on the screen based on the first pre-processed datum responsive to retrieving the first pre-processed datum.

2. The method of claim 1, wherein the summarizing comprises performing at least one of averaging the two or more data, selecting a minimum or maximum value of the two or more data, obtaining a value based on polynomial fitting of the two or more data, or selecting a median value of the two or more data.

3. The method of claim 1, wherein the first data source comprises a module for sensing physical properties.

4. The method of claim 1, further comprising:

receiving a second set of data from a second data source associated with different times spanning across a second time interval longer than the predetermined time interval;

summarizing two or more data for the second set of data over the predetermined time interval to generate a second pre-processed datum representing a summarized datum for the second data source over the predetermined time interval;

storing the second pre-processed datum in the data storage; and

retrieving the second pre-processed datum responsive to receiving a request for the second set of data covering the predetermined time interval.

5. The method of claim 1, wherein the summarizing of the two or more data is performed offline before receiving the request for the first set of data.

6. The method of claim 1, wherein the first data source comprises an asset tracking device.

7. The method of claim 1, further comprising processing the first pre-processed data responsive to retrieving the first pre-processed data to display the visual representation.

8. The method of claim 1, further comprising, prior to the retrieving/summarizing step, retrieving two or more of the keyed data entries using the count-based indices, and decimating the retrieved keyed data entries.

9. A computer readable storage medium structured to store instructions executable by a processor in a computing device, the instructions, when executed causing the processor to:

receive a first set of data from a first data source, the first set of data having data entries, each data entry being associated with a different time of a first time interval;

key data entries of the first set of data, each keyed data entry being associated with a different count-based index;

store the keyed data entries;

retrieve two or more of the keyed data entries using the count-based indices, and summarize those two or more of the retrieved data entries for the first set of data received over a predetermined time interval to generate a first pre-processed datum representing a summarized datum for the first data source over the predetermined time interval, the predetermined time interval being shorter than the first time interval;

store the first pre-processed datum in a data storage;

retrieve the first pre-processed datum responsive to receiving a request for the first set of data covering the predetermined time interval; and

display a visual representation of the first set of data on the screen based on the first pre-processed datum responsive to retrieving the first pre-processed datum.

10. The computer readable storage medium of claim 9, wherein the instructions to summarize comprises instructions to perform at least one of averaging the two or more data, selecting a minimum or maximum value of the two or more data, obtaining a value based on polynomial fitting of the two or more data, or selecting a median value of the two or more data.

11. The computer readable storage medium of claim 9, further comprising instructions to:

receive a second set of data from a second data source associated with different times spanning across a second interval longer than the predetermined time interval;

summarize two or more data for the second set of data over the predetermined time interval to generate a second pre-processed datum representing a summarized datum for the second data source over the predetermined time interval;

store the second pre-processed datum in the data storage; and

retrieve the second pre-processed datum responsive to receiving a request for the second set of data covering the predetermined time interval.

12. The computer readable storage medium of claim 9, wherein the first data source comprises a module for sensing physical properties.

13. The computer readable storage medium of claim 9, wherein instructions to summarize the two or more data is executed offline before executing instructions to receive the request for the first set of data.

14. The computer readable storage medium of claim 9, further storing instructions to process the first pre-processed data responsive to retrieving the first pre-processed data to display the visual representation.

15. The storage medium of claim 9, wherein when the instructions are executed, the processor is caused, prior to the retrieving/summarizing step, to retrieve two or more of the keyed data entries using the count-based indices, and decimate those retrieved keyed data entries.

16. A data server for processing and retrieving data for display on a screen, the data server comprising:

an interface configured to receive a first set of data from a first data source, the first set of data having data entries, each data entry being associated with a different time of a first time interval;

a pre-processing engine configured to;

key data entries of the first set of data each keyed data entry being associated with a different count-based index;

store the keyed data entries;

retrieve two or more of the keyed data entries using the count-based indices, and summarize those two or more of the retrieved data entries for the first set of data received over a predetermined time interval to generate a first pre-processed datum representing a summarized datum for the first data source over the predetermined time interval, the predetermined time interval being shorter than the first time interval;

a pre-processed data storage configured to store the first pre-processed datum in a data storage; and

a data processing engine configured to retrieve the first pre-processed datum responsive to receiving a request for the first set of data covering the predetermined time interval, the data processing engine providing datum for use in displaying a visual representation of the first set of data.

17. The data server of claim 16, further comprising a communication module adapted to transmit the retrieved first pre-processed datum to a client device, the client device displaying a visual representation of the first set of data on the screen based on the first pre-processed datum responsive to retrieving the pre-processed datum.

18. The data server of claim 16, wherein the pre-processing engine is configured to summarize the two or more data by performing at least one of averaging the two or more data, selecting a minimum or maximum value of the two or more data, obtaining a value based on polynomial fitting of the two or more data, or selecting a median value of the two or more data.

19. The data server of claim 16, wherein:

the interface is further configured to receive a second set of data from a second data source associated with different times spanning across a second interval longer than the predetermined time interval;

the preprocessing engine is further configured to summarize two or more data for the second set of data over the predetermined time interval to generate a second pre-processed datum representing a summarized datum for the second data source over the predetermined time interval;

the pre-processed data storage is further configured to store the second pre-processed datum in the data storage; and

the data processing engine is further configured to retrieve the second pre-processed datum responsive to receiving a request for the second set of data covering the predetermined time interval.

20. The data server of claim 16, wherein the first data source comprises a module for sensing physical properties.

21. The data server of claim 16, wherein the data processing engine is further configured to process the first pre-processed data responsive to retrieving the first pre-processed data to display the visual representation.

22. The data server of claim 16, wherein the pre-processing engine is configured to summarize two or more data for the first set of data before receiving the request for the first set of data.

23. The data server of claim 16, wherein the pre-processing engine is configured to retrieve, prior to the retrieving/summarizing step, two or more of the keyed data entries using the count-based indices, and decimate those retrieved keyed data entries.