Fast data retrieval based upon contiguous consolidation of records according to frequency of access

• US 6,691,136 B2
• Filed: 12/22/2000
• Issued: 02/10/2004
• Est. Priority Date: 09/28/2000
• Status: Active Grant

• Current Assignee: Fair Isaac Corporation
• Original Assignee: Fair Isaac Corporation
• Inventors: Lee, Walter, Kisslak, George, Lipham, John
• Primary Examiner(s): Mizrahi, Diane D.
• Application Number:
  US09/747,766
• Publication Number:
  US 20020103784A1
• Time in Patent Office: 1,145 Days
• Field of Search: 707/204, 707/100, 707/101, 707/102, 707/205, 707/200, 711/112, 711/111, 711/113, 711/118, 711/129, 709/213, 709/214, 705/10
• US Class Current: 1/1
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

The most and least frequently accessed data records are contiguously stored on static media. Additionally, a buffer in random access memory holds copies of the most recently accessed records. Updated records are copied from the buffer, and contiguously stored in a separate file on static media. Non-accessed records are contiguously consolidated.

Claims

1. A method in a computer system for facilitating quick access of stored data, the method comprising:

storing a plurality of data records on media, in at least one contiguous file;

storing a subset of the data records in electronic computer memory, the subset comprising the most recently accessed data records;

storing an index in said electronic computer memory, said index facilitating access of said stored plurality of data records on media and said subset of said data records stored in said electronic computer memory, said subset of facilitating access comprising keeping track of which of said subset of said data records stored in said electronic computer memory are updated since having been stored from said media;

contiguously writing at least a portion of the subset of data records stored in electronic computer memory to at least one available file on media, wherein a frequency of executing said step of contiguously writing is configurable;

for at least one file on media, classifying as obsolete data records therein that have subsequently been written to another file;

contiguously copying non-obsolete data records from said at least one file on media to at least one available file on media and skipping said obsolete data records, wherein a frequency of executing said step of contiguously copying is configurable; and

classifying said at least one file on media from which non-obsolete data records were copied and wherein said obsolete data records are skipped as available for subsequent contiguous storage of data records from any of said electronic computer memory and said media;

where rein said at least one contiguous fit from which said subset of the data records are stored in electronic computer memory can subsequently comprise both obsolete data an non-obsolete data; and

wherein when any available file is full, a new available file is designated as active and said any full available file is designated as no longer active and no longer available.

2. The method of claim 1, further comprising:

pre-allocating at least one contiguous region of media, for contiguous storage of at least one file.

3. The method of claim 1, further comprising:

pre-allocating n contiguous files of m bytes each on blank media, where n and m are integers greater than or equal to one.

4. The method of claim 1, wherein:

the portion of the subset of the data records in electronic computer memory that is contiguously written to media comprises data records that have not been written to media since having been updated.

5. The method of claim 1, further comprising:

storing the subset of the data records in electronic computer memory in a circular queue.

6. The method of claim 1, further comprising:

storing the subset of the data records in electronic computer memory in a variable sized buffer.

7. The method of claim 1, further comprising:

responsive to an attempt to access a data record, determining whether the data record is in electronic computer memory; and

responsive to a determination that the data record is not in electronic computer memory, loading the data record into electronic computer memory.

8. The method of claim 1, further comprising:

updating an existing data record;

storing the updated data record; and

storing a pointer from the updated data record to the existing data record.

9. The method of claim 8, further comprising:

utilizing the pointer to trace changes to a data record.

10. The method of claim 8, further comprising:

utilizing the pointer to recover a corrupted data record.

11. The method of claim 1, wherein:

data records comprise profiles, a profile being data that summarizes at least one transaction.

12. The method of claim 11, wherein:

at least one profile is used as input for predictive statistical modeling program.

13. The method of claim 11, wherein:

at least one profile comprises output from a predictive statistical modeling program.

14. A computer program product for facilitating quick access of stored data, the computer program product comprising:

program code for storing a plurality of data records on media, in at least one contiguous file;

program code for storing a subset of the data records in electronic computer memory, the subset comprising the most recently accessed data records;

program code for storing an index in said electronic computer memory, said index facilitating access of said stored plurality of data records on media and said subset of said data records stored in said electronic computer memory, said step of facilitating access comprising keeping track of which of said subset of said data records stored in said electronic computer memory are updated since having been stored from said media;

program code for contiguously writing at least a portion of the subset of data records stored in electronic computer memory to at least one available file on media, wherein a frequency of executing said step of contiguously writing is configurable;

program code for classifying as obsolete, for at least one file on media, data records therein that have subsequently been written to another file;

program code for contiguously copying non-obsolete data records from said at least one file on media to at least one available file on media and skipping said obsolete data records, wherein a frequency of executing said step of contiguously copying is configurable;

program code for classifying said at least one file on media from which non-obsolete data records were copied and wherein said obsolete data records are skipped as available for subsequent contiguous storage of data records from any of said electronic computer memory and said media; and

a computer readable medium on which the program codes are stored;

wherein said at least one contiguous file from which said subset of the data records are stored in electronic computer memory can subsequently comprise both obsolete data non-obsolete data; and

wherein when any available file is full, a new available file is designated as active and said any full available file a designated as no longer active and no longer available.

15. The computer program product of claim 14, further comprising:

program code for pre-allocating at least one contiguous region of media, for contiguous storage of at least one file.

16. The computer program product of claim 14, further comprising:

program code for pre-allocating n contiguous files of m bytes each on blank media, where n and m are integers greater than or equal to one.

17. The computer program product of claim 14, further comprising:

program code for contiguously writing data records that have not been written to media since having been updated to at least one file on media.

18. The computer program product of claim 14, further comprising:

program code for storing the subset of the data records in electronic computer memory in a circular queue.

19. The computer program product of claim 14, further comprising:

program code for storing the subset of the data records in electronic computer memory in a variable sized buffer.

20. The computer program product of claim 14, further comprising:

program code for determining, response to an attempt to access a data record, whether the data record is in electronic computer memory; and

program code for loading the data record into electronic computer memory, responsive to a determination that the data record is not in electronic computer memory.

21. The computer program product of claim 14, further comprising:

program code for updating an existing data record;

program code for storing the updated data record; and

program code for storing a pointer from the updated data record to the existing data record.

22. The computer program product of claim 21, further comprising:

program code for utilizing the pointer to trace changes to a data record.

23. The computer program product of claim 21, further comprising:

program code for utilizing the pointer to recover a corrupted data record.

24. The computer program product of claim 14, further comprising:

program code for using at least one data record that summarizes at least one transaction as input for a predictive statistical modeling program.

25. The computer program product of claim 14, further comprising:

program code for receiving at least on data record that summarizes at least one transaction as output from a predictive statistical modeling program.

26. A computer system for facilitating quick access of stored data, the computer system comprising:

a storage module, for storing a plurality of data records on media, in at least one contiguous file;

a sub-set storage module, for storing a subset of the data records in electronic computer memory, the subset comprising the most recently accessed data records, the sub-set storage module being coupled to the storage module;

an index module for facilitating access of said stored plurality of data records on media and said subset of said data records stored in said electronic computer memory, said step of facilitating access comprising keeping track of which of said subset of said data records stored in said electronic computer memory are updated since having been stored from said media, the index module being coupled to the sub-set storage module;

a writing module, for contiguously writing at least a portion of the subset of data records stored in electronic computer memory to at least one available file on media, wherein a frequency of executing said step of contiguously writing is configurable, the writing module being coupled to the sub-set storage module;

a classification module, for classifying as obsolete, for at least one file on media, data records therein that have subsequently been written to another file, the classification module being coupled to the writing module;

a copying module, for contiguously copying non-obsolete data records from said at least one file on media at least one available file on media and skipping said obsolete data records, wherein a frequency of executing said step of contiguously copying is configurable, the copying module being coupled to the classification module; and

an availability-classification module, for classifying said at least one file on media from which non-obsolete data records were copied and wherein said obsolete data records are skipped as available for subsequent contiguous storage of data records from any of said electronic computer memory and said media, the availability-classification module being coupled to the copying module;

wherein said at least one contiguous file from which said subset of the data records are stored in electronic computer memory can subsequently comprise both obsolete data and non-obsolete data; and

wherein when any available file is full, a new available file is designated active, and said any full available file is designated as no longer active and no longer available.

27. The computer system of claim 26, further comprising:

a pre-allocation module, for pre-allocating at least one contiguous region of media, for contiguous storage of at least one file, the pre-allocation module being coupled to the storage module.

28. The computer system of claim 26, further comprising:

a pre-allocation module, for pre-allocating n contiguous files of m bytes each on blank media, where n and m are integers greater than or equal to one, the pre-allocation module being coupled to the storage module.

29. The computer system of claim 26, wherein:

the writing module is further for contiguously writing to media data records in electronic computer memory that have not been written to media since having been updated.

30. The computer system of claim 26, wherein:

the sub-set storage module is further for storing the subset of the data records in electronic computer memory in circular queue.

31. The computer system of claim 26, further comprising:

the sub-set storage module is further for storing the subset of the data records in electronic computer memory in a variable sized buffer.

32. The computer system of claim 26, further comprising:

a determination module, for determining, responsive to an attempt to access a data record, whether the data record is in electronic computer memory, the determination module being coupled to the sub-set storage module; and

a loading module, for loading the data record into electronic computer memory responsive to a determination that the data record is not in electronic computer memory, the loading module being coupled to the determination module.

33. The computer system of claim 26, further comprising:

an updating module, for updating an existing data record, the updating module being coupled to the sub-set storage module;

an updated-record storage module, for storing the updated data record, the updated-record storage module being coupled to the updating module; and

a pointer storage module, for storing a pointer from the updated data record to the existing data record, the pointer storage module being coupled to the updated-record storage module.

34. The computer system of claim 33, further comprising:

a pointer utilization module, for utilizing the pointer to trace changes to a data record, the pointer utilization module being coupled to the pointer storage module.

35. The computer system of claim 33, further comprising:

a pointer utilization module, for utilizing the pointer to recover a corrupted data record, the pointer utilization module being coupled to the pointer storage module.

36. The computer system of claim 26, further comprising:

a data record utilization module, for utilizing at least one data record that summarizes at least one transaction as input for a predictive statistical modeling program, the data record utilization module being coupled to the storage module.

37. The computer system of claim 26, further comprising:

a data record reception module, for receiving at least one data record that summarizes at least one transaction as output from a predictive statistical modeling program, the data record reception module being coupled to the storage module.

38. A method in a computer system for facilitating quick access of stored data, the method comprising:

loading accessed data records into computer memory;

dynamically consolidating data records accessed most recently by contiguously flushing accessed data records from computer memory to a separate file on static media; and

dynamically consolidating data records accessed least recently by writing non-accessed data records to a separate file on static media;

wherein when any of said separate files is full, a new separate file is designated as active and available for subsequent storage; and

wherein when any separate file comprises only obsolete records and non-access records previously copied to another separate file, said an separate file is designated as obsolete and available for subsequent storage.