Method and apparatus for data preheating

• US 9,569,489 B2
• Filed: 05/15/2014
• Issued: 02/14/2017
• Est. Priority Date: 11/18/2011
• Status: Active Grant

• Current Assignee: Huawei Cloud Computing Technologies Company Limited (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Inventors: Dai, Kun, Li, Huaizhou, Song, Jiao
• Primary Examiner(s): Phillips, III, Albert
• Application Number:
  US14/278,707
• Publication Number:
  US 20140250102A1
• Time in Patent Office: 1,006 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 12/0246 in block erasable memory, e.g. flash memory
• CPC Class Codes: G06F 12/0862 with prefetch
• CPC Class Codes: G06F 16/21 Design, administration or maintenance of databases
• CPC Class Codes: G06F 16/2453 Query optimisation

Abstract

Embodiments of the present invention disclose a method and an apparatus for data preheating, which relate to database technologies, so as to increase a responding speed of a database system, reduce frequent IO on a magnetic disk, and improve overall performance of the database system. The method includes: when a database is started, reading hit ratio information from a flash memory device and reading user configuration information from a hard disk; obtaining a union set of a preset hot data table in the user configuration information and a hot data table in the hit ratio information; determining the total number of to-be-loaded hot data blocks from the union set; according to the storage proportions, determining the to-be-loaded hot data block to be stored into a shared buffer and the flash memory device; and separately storing the to-be-loaded hot data block into the shared buffer and the flash memory device.

Claims

1. A method for data preheating, the method comprising:

reading, separately, hit ratio information and user configuration information when a database is started, wherein the hit ratio information comprises a hot data table, and the user configuration information comprises a preset hot data table, storage proportions of a to-be-loaded hot data block stored into a shared buffer and a flash memory device, a capacity of the shared buffer, a storage capability of the flash memory device, and a deviation value of the total number of the to-be-loaded hot data blocks;

obtaining a union set of the preset hot data table in the user configuration information and the hot data table in the hit ratio information;

determining the total number of the to-be-loaded hot data blocks from the union set according to the storage proportions, the capacity of the shared buffer, a capacity of the flash memory device, a size of the hot data block, and the deviation value;

determining, according to the storage proportions, the to-be-loaded hot data block to be stored into the shared buffer and the flash memory device; and

storing, separately, the to-be-loaded hot data block into the shared buffer and the flash memory device.

2. The method for data preheating according to claim 1, wherein determining the total number of the to-be-loaded hot data blocks from the union set according to the storage proportions, the capacity of the shared buffer, the capacity of the flash memory device, the size of the hot data block, and the deviation value comprises:

calculating a threshold of the total number of the to-be-loaded hot data blocks according to the storage proportions, the capacity of the shared buffer, the capacity of the flash memory device, and the size of the hot data block;

calculating the number of candidate hot data tables TCT according to the threshold of the total number of the to-be-loaded hot data blocks, the total number of hot data tables in the union set, and the total number of hot data blocks in the union set, wherein the candidate hot data table is first TCT hot data table whose hit count is the highest in the union set;

calculating the number of candidate hot data blocks in each candidate hot data table according to the threshold of the total number of the to-be-loaded hot data blocks, the total number of hot data blocks in the candidate hot data table, and the total number of hot data blocks in the union set; and

sifting out the to-be-loaded hot data block from the candidate hot data block in the candidate hot data table, and obtaining the total number of the to-be-loaded hot data blocks.

3. The method for data preheating according to claim 2, wherein calculating the threshold of the total number of the to-be-loaded hot data blocks according to the storage proportions, the capacity of the shared buffer, the capacity of the flash memory device, and the size of the hot data block comprises:

calculating the threshold of the total number of the to-be-loaded hot data blocks according to

4. The method for data preheating according to claim 2, wherein calculating the number of candidate hot data tables TCT according to the threshold of the total number of the to-be-loaded hot data blocks, the total number of hot data tables in the union set, and the total number of hot data blocks in the union set comprises:

calculating the number of candidate hot data tables TCT according to

5. The method for data preheating according to claim 2, wherein calculating the number of candidate hot data blocks in each candidate hot data table according to the threshold of the total number of the to-be-loaded hot data blocks, the total number of hot data blocks in the candidate hot data table, and the total number of hot data blocks in the union set comprises:

calculating the number of candidate hot data blocks in each candidate hot data table according to

6. The method for data preheating according to claim 2, wherein sifting out the to-be-loaded hot data block from the candidate hot data block in the candidate hot data table, and obtaining the total number of the to-be-loaded hot data blocks comprises:

calculating a sifting base value of each candidate hot data table according to FB=THN_TCT/2×

BR_BN, wherein FB is the sifting base value of each candidate hot data table, THN_TCT/2 is a hit count of a ┌

TCT/2┐

^th candidate hot data table, and BR_BN is a hit ratio of a BN^th hot data block of hot data blocks whose hit ratios are the highest in each candidate hot data table;

calculating the hit count of the candidate hot data block in each candidate hot data table according to PB=PT×

QB, wherein PB is the hit count of the candidate hot data block in each candidate hot data table, PT is the hit count of each candidate hot data table, and QB is the hit ratio of the candidate hot data block in each candidate hot data table;

adding together all first candidate hot data blocks in the candidate hot data table whose hit counts are greater than or equal to the sifting base value of the candidate hot data table, and obtaining the total number of first sifting hot data blocks; and

determining that the first candidate hot data blocks are the to-be-loaded hot data blocks and obtaining the total number of the to-be-loaded hot data blocks when the total number of the first sifting hot data blocks is within a range of the threshold of the total number of the to-be-loaded hot data blocks plus or minus the deviation value.

7. The method for data preheating according to claim 6, further comprising:

multiplying the sifting base value by a preset multiple, and obtaining a new sifting base value when the total number of the first sifting hot data blocks is not within the range of the threshold of the total number of the to-be-loaded hot data blocks plus or minus the deviation value;

adding together all second candidate hot data blocks in the candidate data table whose hit counts are greater than or equal to the new sifting base value, and obtaining the total number of second sifting hot data blocks; and

determining that the second candidate hot data blocks are the to-be-loaded hot data blocks, and obtaining the total number of the to-be-loaded hot data blocks when the total number of the second sifting hot data blocks is within the range of the threshold of the total number of the to-be-loaded hot data blocks plus or minus the deviation value.

8. The method for data preheating according to claim 1, wherein the method further comprises:

collecting statistics on the hit ratio information of hot data in real time and updating the hit ratio information stored into the flash memory device when the database runs; and

backing up data in the flash memory device to a hard disk, wherein the backup data comprises the updated hit ratio information.

9. An apparatus for data preheating, the apparatus comprising:

a processing device comprising at least one processor, the processing device configured to;

read, separately, hit ratio information and user configuration information when a database is started, wherein the hit ratio information comprises a hot data table, and the user configuration information comprises a preset hot data table, storage proportions of a to-be-loaded hot data block stored into a shared buffer and a flash memory device, a capacity of the shared buffer, a storage capability of the flash memory device, and a deviation value of the total number of the to-be-loaded hot data blocks;

obtain a union set of the preset hot data table in the user configuration information and the hot data table in the hit ratio information;

determine the total number of the to-be-loaded hot data blocks from the union set according to the storage proportions, the capacity of the shared buffer, a capacity of the flash memory device, a size of the hot data block, and the deviation value;

determine the to-be-loaded hot data block to be stored into the shared buffer and the flash memory device according to the storage proportions; and

store, separately, the to-be-loaded hot data block into the shared buffer and the flash memory device.

10. The apparatus for data preheating according to claim 9, wherein the processing device is further configured to:

calculate a threshold of the total number of the to-be-loaded hot data blocks according to the storage proportions, the capacity of the shared buffer, the capacity of the flash memory device, and the size of the hot data block;

calculate the number of candidate hot data tables TCT according to the threshold of the total number of the to-be-loaded hot data blocks, the total number of hot data tables in the union set, and the total number of hot data blocks in the union set, wherein the candidate hot data table is first TCT hot data table whose hit count is the highest in the union set;

calculate the number of candidate hot data blocks in each candidate hot data table according to the threshold of the total number of the to-be-loaded hot data blocks, the total number of hot data blocks in the candidate hot data table, and the total number of hot data blocks in the union set; and

sift out the to-be-loaded hot data block from the candidate hot data block in the candidate hot data table, and obtain the total number of the to-be-loaded hot data blocks.

11. The apparatus for data preheating according to claim 9, wherein the processing device is configured to calculate a threshold of the total number of the to-be-loaded hot data blocks according to

12. The apparatus for data preheating according to claim 9, wherein the processing device is configured to calculate the number of candidate hot data tables TCT according to

13. The apparatus for data preheating according to claim 9, wherein the processing device is configured to calculate the number of candidate hot data blocks in each candidate hot data table according to

14. The apparatus for data preheating according to claim 9, wherein the processing device is further configured to:

calculate a sifting base value of each candidate hot data table according to FB=THN_TCT/2×

BR_BN, wherein FB is the sifting base value of each candidate hot data table, THN_TCT/2 is the hit count of a ┌

TCT/2┐

^th candidate hot data table, and BR_BN is a hit ratio of a BN^th hot data block of hot data blocks whose hit ratios are the highest in each candidate hot data table;

calculate the hit count of the candidate hot data block in each candidate hot data table according to PB=PT×

QB, wherein PB is the hit count of the candidate hot data block in each candidate hot data table, PT is the hit count of each candidate hot data table, and QB is the hit ratio of the candidate hot data block in each candidate hot data table;

add together all first candidate hot data blocks in the candidate data table whose hit counts are greater than or equal to the sifting base value of the candidate hot data table, and obtain the total number of first sifting hot data blocks; and

determine that the first candidate hot data blocks are the to-be-loaded hot data blocks and obtain the total number of the to-be-loaded hot data blocks when the total number of the first sifting hot data blocks is within a range of the threshold of the total number of the to-be-loaded hot data blocks plus or minus the deviation value.

15. The apparatus for data preheating according to claim 9, wherein the processing device is further configured to:

multiply the sifting base value by a preset multiple and obtain a new sifting base value when the total number of the first sifting hot data blocks is not within the range of the threshold of the total number of the to-be-loaded hot data blocks plus or minus the deviation value;

add together all second candidate hot data blocks in the candidate data table whose hit counts are greater than or equal to the new sifting base value, and obtain the total number of second sifting hot data blocks; and

determine that the second candidate hot data blocks are the to-be-loaded hot data blocks and obtain the total number of the to-be-loaded hot data blocks when the total number of the second sifting hot data blocks is within the range of the threshold of the total number of the to-be-loaded hot data blocks plus or minus the deviation value.

16. The apparatus for data preheating according to claim 9, wherein the processing device is further configured to:

collect statistics on the hit ratio information of hot data in real time and update the hit ratio information stored into the flash memory device when the database runs; and

back up data in the flash memory device to a hard disk, wherein the backup data comprises the updated hit ratio information.