Selective data compression for in-memory databases

• US 9,990,308 B2
• Filed: 08/31/2015
• Issued: 06/05/2018
• Est. Priority Date: 08/31/2015
• Status: Active Grant

• Current Assignee: Oracle International Corporation (Oracle Corporation)
• Original Assignee: Oracle International Corporation (Oracle Corporation)
• Inventors: Krishnappa, Chinmayi, Marwah, Vineet, Ganesh, Amit
• Primary Examiner(s): Tran, Anh Tai
• Application Number:
  US14/841,561
• Publication Number:
  US 20170060772A1
• Time in Patent Office: 1,009 Days
• Field of Search: None
• CPC Class Codes: G06F 12/0802 Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
• CPC Class Codes: G06F 12/12 Replacement control
• CPC Class Codes: G06F 16/24552 Database cache management
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: H03M 7/30 Compression speech analysis-synthesis for redundancy reduction G10L19/00; for image communication H04N; Expansion; Suppression of unnecessary data, e.g. redundancy reduction

Abstract

Techniques are provided for maintaining data persistently in one format, but making that data available to a database server in more than one format. For example, one of the formats in which the data is made available for query processing is based on the on-disk format, while another of the formats in which the data is made available for query processing is independent of the on-disk format. Data that is in the format that is independent of the disk format may be maintained exclusively in volatile memory to reduce the overhead associated with keeping the data in sync with the on-disk format copies of the data. Selection of data to be maintained in the volatile memory may be based on various factors. Once selected the data may also be compressed to save space in the volatile memory. The compression level may depend on one or more factors that are evaluated for the selected data. The factors for the selection and compression level of data may be periodically evaluated, and based on the evaluation, the selected data may be removed from the volatile memory or its compression level changed accordingly.

Claims

1. A computer-implemented method comprising:

maintaining, by a database server, a database on persistent storage;

wherein the database includes;

a first data portion, that has been designated as eligible for mirroring within volatile memory; and

a second data portion that has been also designated as eligible for mirroring within the volatile memory;

storing first eligibility data that corresponds to the first data portion, wherein the first eligibility data reflects an estimated benefit of having the first data portion mirrored in the volatile memory;

storing second eligibility data that corresponds to the second data portion, wherein the second eligibility data reflects an estimated benefit of having the second data portion mirrored in the volatile memory;

wherein the first eligibility data and the second eligibility data are determined by the database server based on one or more factors;

based on the first eligibility data, automatically determining that the first data portion should not be loaded into the volatile memory;

based on the second eligibility data, automatically determining that the second data portion should be loaded into the volatile memory; and

in response to a load event that causes data portions to be mirrored within the volatile memory, loading, into the volatile memory, data from the second data portion without loading into volatile memory any data from the first data portion.

2. The method of claim 1, wherein the first data portion is a subset of the second data portion.

3. The method of claim 1, wherein loading, into the volatile memory, the data from the second data portion comprises constructing an in-memory compression unit using the data from the second data portion.

4. The method of claim 3, wherein, in the persistent storage, the data from the second data portion is organized in a row-major format, and, within the in-memory compression unit, the data from the second data portion is organized in a column-major format.

5. The method of claim 1, wherein the one or more factors include at least one of:

access statistics associated with a particular data portion for which particular eligibility data is being determined, a datatype of the particular data portion, operation statistics associated with the particular data portion, or data statistics associated with the particular data portion.

6. The method of claim 5,

wherein the access statistics associated with the particular data portion comprise heat map statistics of the particular data portion;

and

the method further comprising;

evaluating the heat map statistics of the particular data portion,

if the heat map statistics indicate frequent read accesses to the particular data portion, modifying the particular eligibility data to increase likelihood of the particular data portion to be loaded into the volatile memory;

if the heat map statistics indicate infrequent read accesses to the particular data portion, modifying the particular eligibility data to decrease the likelihood of the particular data portion to be loaded into the volatile memory.

7. The method of claim 5:

wherein the access statistics associated with the particular data portion comprise heat map statistics of the particular data portion; and

the method further comprising;

evaluating the heat map statistics of the particular data portion,

if the heat map statistics indicate frequent write accesses to the particular data portion, modifying the particular eligibility data to decrease likelihood of the particular data portion to be loaded into the volatile memory.

8. The method of claim 5, further comprising:

evaluating the datatype of the particular data portion,

if the datatype is a large object type, modifying the particular eligibility data to decrease likelihood of the particular data portion to be loaded into the volatile memory.

9. The method of claim 1, further comprising:

evaluating for a certain data portion of the second data portion at least one of the following;

access statistics of the certain data portion

a data type of the certain data portion,

performance statistics for one or more compressed data portions that are similar to the certain data portion;

based on the evaluating, modifying a compression level for the certain data portion;

compressing data in the certain data portion at the modified compression level; and

loading the compressed data into the volatile memory.

10. The method of claim 9, wherein the performance statistics, for the one or more compressed data portions that are similar to the certain data portion, are evaluated to indicate that a different compression level consumes less resources of the database server;

and modifying the compression level for the certain data portion to approach the different compression level.

11. The method of claim 1, wherein:

wherein the database includes a third data portion, that has been designated as eligible for mirroring within the volatile memory; and

the method further comprises;

storing third eligibility data that corresponds to the third data portion, wherein the third eligibility data reflects an estimated benefit of having the third data portion mirrored in the volatile memory;

while data from the third data portion is mirrored in the volatile memory and the data from the first data portion is not mirrored in the volatile memory, the database server performing the steps of;

recalculating the first eligibility data;

recalculating the third eligibility data;

based on the first eligibility data, automatically determining that the data from the first data portion should be loaded into the volatile memory;

performing a comparison, wherein the comparison is between the third eligibility data and either;

the first eligibility data;

or

a threshold value;

based on the comparison, the database server determining that the data from the third data portion is to be removed from the volatile memory;

in response to a purge event that causes copies of mirrored data portions to be removed from the volatile memory, removing from the volatile memory, the data from the third data portion;

in response to a load event that causes copies of data portions to be mirrored within the volatile memory, loading into the volatile memory the data from the first data portion.

12. One or more non-transitory storage media storing instructions, which when executed by one or more hardware processors, cause:

maintaining, by a database server, a database on persistent storage;

wherein the database includes;

a first data portion, that has been designated as eligible for mirroring within volatile memory; and

a second data portion that has been also designated as eligible for mirroring within the volatile memory;

storing first eligibility data that corresponds to the first data portion, wherein the first eligibility data reflects an estimated benefit of having the first data portion mirrored in the volatile memory;

storing second eligibility data that corresponds to the second data portion, wherein the second eligibility data reflects an estimated benefit of having the second data portion mirrored in the volatile memory;

wherein the first eligibility data and the second eligibility data are determined by the database server based on one or more factors;

based on the first eligibility data, automatically determining that the first data portion should not be loaded into the volatile memory;

based on the second eligibility data, automatically determining that the second data portion should be loaded into the volatile memory; and

in response to a load event that causes data portions to be mirrored within the volatile memory, loading, into the volatile memory, data from the second data portion without loading into volatile memory any data from the first data portion.

13. The one or more non-transitory storage media of claim 12, wherein the first data portion is a subset of the second data portion.

14. The one or more non-transitory storage media of claim 12, wherein instructions for loading, into the volatile memory, the data from the second data portion include instructions, which when executed by said one or more hardware processors further cause constructing an in-memory compression unit using the data from the second data portion.

15. The one or more non-transitory storage media of claim 14, wherein, in the persistent storage, the data from the second data portion is organized in a row-major format, and, within the in-memory compression unit, the data from the second data portion is organized in a column-major format.

16. The one or more non-transitory storage media of claim 12, wherein the one or more factors include at least one of:

access statistics associated with a particular data portion for which particular eligibility data is being determined, a datatype of the particular data portion, operation statistics associated with the particular data portion, or data statistics associated with the particular data portion.

17. The one or more non-transitory storage media of claim 16,

wherein the access statistics associated with the particular data portion comprise heat map statistics of the particular data portion;

and

wherein the instructions further include instructions, which when executed by said one or more hardware processors, further cause;

evaluating the heat map statistics of the particular data portion, if the heat map statistics indicate frequent read accesses to the particular data portion, modifying the particular eligibility data to increase likelihood of the particular data portion to be loaded into the volatile memory;

if the heat map statistics indicate infrequent read accesses to the particular data portion, modifying the particular eligibility data to decrease the likelihood of the particular data portion to be loaded into the volatile memory.

18. The one or more non-transitory storage media of claim 16:

wherein the access statistics associated with the particular data portion comprise heat map statistics of the particular data portion; and

wherein the instructions further include instructions, which when executed by said one or more hardware processors, further cause;

evaluating the heat map statistics of the particular data portion,

if the heat map statistics indicate frequent write accesses to the particular data portion, modifying the particular eligibility data to decrease likelihood of the particular data portion to be loaded into the volatile memory.

19. The one or more non-transitory storage media of claim 16, wherein the instructions further include instructions for causing:

evaluating the datatype of the particular data portion,

if the datatype is a large object type, modifying the particular eligibility data to decrease likelihood of the particular data portion to be loaded into the volatile memory.

20. The one or more non-transitory storage media of claim 12, wherein the instructions further include instructions, which when executed by said one or more hardware processors, further cause:

evaluating for a certain data portion of the second data portion at least one of the following;

access statistics of the certain data portion

a data type of the certain data portion,

performance statistics for one or more compressed data portions that are similar to the certain data portion;

based on the evaluating, modifying a compression level for the certain data portion;

compressing data in the certain data portion at the modified compression level; and

loading the compressed data into the volatile memory.

21. The one or more non-transitory storage media of claim 20, wherein the performance statistics, for the one or more compressed data portions that are similar to the certain data portion, are evaluated to indicate that a different compression level consumes less resources of the database server;

and modifying the compression level for the certain data portion to approach the different compression level.

22. The one or more non-transitory storage media of claim 12, wherein:

wherein the database includes a third data portion, that has been designated as eligible for mirroring within the volatile memory; and

wherein the instructions further include instructions, which when executed by said one or more hardware processors, further cause;

storing third eligibility data that corresponds to the third data portion, wherein the third eligibility data reflects an estimated benefit of having the third data portion mirrored in the volatile memory;

while data from the third data portion is mirrored in the volatile memory and the data from the first data portion is not mirrored in the volatile memory, the database server performing the steps of;

recalculating the first eligibility data;

recalculating the third eligibility data;

based on the first eligibility data, automatically determining that the data from the first data portion should be loaded into the volatile memory;

performing a comparison, wherein the comparison is between the third eligibility data and either;

the first eligibility data;

or

a threshold value;

based on the comparison, the database server determining that the data from the third data portion is to be removed from the volatile memory;

in response to a purge event that causes copies of mirrored data portions to be removed from the volatile memory, removing from the volatile memory, the data from the third data portion;

in response to a load event that causes copies of data portions to be mirrored within the volatile memory, loading into the volatile memory the data from the first data portion.