ADAPTIVE FILE STORAGE METHOD AND APPARATUS

• US 20180239549A1
• Filed: 04/23/2018
• Published: 08/23/2018
• Est. Priority Date: 11/04/2015
• Status: Active Grant

• Current Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Inventors: ZHANG, Shiyue, XIAO, Yulei, TU, Dandan
• Granted Patent: US 11,182,080 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/1076 Parity data used in redundant arrays of independent storages, e.g. in RAID systems
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/1824 implemented using Network-attached Storage [NAS] architecture distributed or networked storage systems G06F3/067; protocols for distributed storage of data in a network H04L67/1097
• CPC Class Codes: G06F 3/0613 in relation to throughput
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0626 Reducing size or complexity of storage systems

Abstract

An adaptive file storage method and apparatus is disclosed. The method includes determining a cold and hot attribute of a file, and performing coding storage processing or transcoding storage processing on the file according to the cold and hot attribute of the file. Therefore, a requirement of the cold and hot attribute of the file for storage overheads and restoration costs can be fully considered. In addition, the used coding technology has high reliability and a high coding speed. Therefore, comprehensive performance in multiple dimensions of storage overheads, restoration costs, reliability, and an coding speed can be improved.

Claims

1. A method of adaptively storing a file, the method comprising:

determining a cold and hot attribute of the file, wherein the cold and hot attribute of the file represents a frequency of accessing the file; and

performing coding storage processing or transcoding storage processing on the file according to the cold and hot attribute of the file.

2. The method according to claim 1, wherein the determining a cold and hot attribute of the file comprises:

when the access frequency of the file is greater than a first frequency threshold, or the average access time interval of the file is less than a first time interval, determining that the cold and hot attribute of the file is hot;

or

when the access frequency of the file is not greater than a first frequency threshold, or the average access time interval of the file is not less than a first time interval, determining that the cold and hot attribute of the file is cold.

3. The method according to claim 1, wherein the performing coding storage processing or transcoding storage processing on the file according to the cold and hot attribute of the file comprises:

when the cold and hot attribute of the file is hot, performing coding storage processing on the file by using a first coding algorithm;

or

when the cold and hot attribute of the file is cold, performing coding storage processing on the file by using a second coding algorithm, wherein

restoration costs of the first coding algorithm are less than a restoration costs threshold, and storage overheads of the second coding algorithm are less than a storage overheads threshold.

4. The method according to claim 1, wherein the performing coding storage processing or transcoding storage processing on the file according to the cold and hot attribute of the file comprises:

performing transcoding storage processing on the file according to a change of the cold and hot attribute of the file.

5. The method according to claim 4, wherein the performing transcoding storage processing on the file according to a change of the cold and hot attribute of the file comprises:

when the cold and hot attribute of the file changes from cold to hot, performing, on the file, transcoding storage processing from a first source coding algorithm to a first destination coding algorithm;

or

when the cold and hot attribute of the file changes from hot to cold, performing, on the file, transcoding storage processing from a second source coding algorithm to a second destination coding algorithm,

wherein restoration costs of the first destination coding algorithm are less than a restoration costs threshold, restoration costs of the first source coding algorithm are not less than the restoration costs threshold, storage overheads of the second destination coding algorithm are less than a storage overheads threshold, and storage overheads of the second source coding algorithm are not less than the storage overheads threshold.

6. The method according to claim 5, wherein the file comprises N data blocks, the performing, on the file, transcoding storage processing from a first source coding algorithm to a first destination coding algorithm comprises:

determining, a first destination verification code corresponding to M data blocks in the N data blocks by subtracting a first difference verification code from a first source verification code, wherein

the first source verification code is generated by encoding the N data blocks based on a first encoding function, the first difference verification code is generated by encoding data blocks other than the M data blocks in the N data blocks based on the first encoding function, the first encoding function is an encoding function based on the first source coding algorithm, N and M are positive integers, and N>

M.

7. The method according to claim 5, wherein the file comprises N data blocks, the performing, on the file, transcoding storage processing from a second source coding algorithm to a second destination coding algorithm comprises:

determining, a second destination verification code corresponding to the N data blocks by adding a second source verification code to a second difference verification code, wherein

the second source verification code is generated by encoding M data blocks in the N data blocks based on a second encoding function, the second difference verification code is generated by encoding data blocks other than the M data blocks in the N data blocks based on the second encoding function, the second encoding function is an encoding function based on the second source coding algorithm, N and M are positive integers, and N>

M.

8. The method according to claim 3, wherein before the determining a cold and hot attribute of the file, the method further comprises:

determining that a storage time of the file reaches a first time threshold and a quantity of access times of the file is less than a first quantity threshold.

9. An adaptive file storage apparatus, comprising:

a processor, and a memory storing instructions thereon that when executed by the processor, cause the processor to;

determine a cold and hot attribute of a file, wherein the cold and hot attribute of the file represents frequency of accessing the file; and

perform coding storage processing or transcoding storage processing on the file according to the cold and hot attribute of the file determined.

10. The apparatus according to claim 9, wherein when the access frequency of the file is greater than a first frequency threshold, or the average access time interval of the file is less than a first time interval, the cold and hot attribute of the file is hot;

or

when the access frequency of the file is not greater than a first frequency threshold, or the average access time interval of the file is not less than a first time interval, the cold and hot attribute of the file is cold.

11. The apparatus according to claim 9, wherein the processor is configured to:

when the cold and hot attribute of the file is hot, perform coding storage processing on the file by using a first coding algorithm;

or

when the cold and hot attribute of the file is cold, perform coding storage processing on the file by using a second coding algorithm, wherein

restoration costs of the first coding algorithm are less than a restoration costs threshold, and storage overheads of the second coding algorithm are less than a storage overheads threshold.

12. The apparatus according to claim 9, wherein the processor is configured to:

perform transcoding storage processing on the file according to a change of the cold and hot attribute of the file.

13. The apparatus according to claim 12, wherein the processor is configured to:

when the cold and hot attribute of the file changes from cold to hot, perform, on the file, transcoding storage processing from a first source coding algorithm to a first destination coding algorithm;

or

when the cold and hot attribute of the file changes from hot to cold, perform, on the file, transcoding storage processing from a second source coding algorithm to a second destination coding algorithm, wherein

restoration costs of the first destination coding algorithm are less than a restoration costs threshold, restoration costs of the first source coding algorithm are not less than the restoration costs threshold, storage overheads of the second destination coding algorithm are less than a storage overheads threshold, and storage overheads of the second source coding algorithm are not less than the storage overheads threshold.

14. The apparatus according to claim 13, wherein the file comprises N data blocks, the processor is configured to:

determine a first destination verification code corresponding to M data blocks in the N data blocks by subtracting a first difference verification code from a first source verification code, wherein

the file comprises N data blocks, the first source verification code is generated by encoding the N data blocks based on a first encoding function, the first difference verification code is generated by encoding data blocks other than the M data blocks in the N data blocks based on the first encoding function, the first encoding function is an encoding function based on the first source coding algorithm, N and M are positive integers, and N>

M.

15. The apparatus according to claim 13, wherein the file comprises N data blocks, the processor is configured to:

determine a second destination verification code corresponding to the N data blocks by adding a second source verification code to a second difference verification code, wherein

the second source verification code is generated by encoding M data blocks in the N data blocks based on a second encoding function, the second difference verification code is generated by encoding data blocks other than the M data blocks in the N data blocks based on the second encoding function, the second encoding function is an encoding function based on the second source coding algorithm, N and M are positive integers, and N>

M.

16. A non-transitory computer readable storage medium storing instructions which, when executed by a processor, cause the processor to:

determine a cold and hot attribute of a file, wherein the cold and hot attribute of the file represents frequency of accessing the file; and

perform coding storage processing or transcoding storage processing on the file according to the cold and hot attribute of the file determined.

17. The non-transitory computer readable storage medium according to claim 16, wherein when the access frequency of the file is greater than a first frequency threshold, or the average access time interval of the file is less than a first time interval, the cold and hot attribute of the file is hot;

or

when the access frequency of the file is not greater than a first frequency threshold, or the average access time interval of the file is not less than a first time interval, the cold and hot attribute of the file is cold.

18. The non-transitory computer readable storage medium according to claim 16, wherein

when the cold and hot attribute of the file is hot, coding storage processing is performed on the file by using a first coding algorithm;

or

when the cold and hot attribute of the file is cold, coding storage processing is performed on the file by using a second coding algorithm, wherein

restoration costs of the first coding algorithm are less than a restoration costs threshold, and storage overheads of the second coding algorithm are less than a storage overheads threshold.

19. The non-transitory computer readable storage medium according to claim 16, wherein transcoding storage processing is performed on the file according to a change of the cold and hot attribute of the file.

20. The non-transitory computer readable storage medium according to claim 19, wherein

when the cold and hot attribute of the file changes from cold to hot, transcoding storage processing is performed on the file from a first source coding algorithm to a first destination coding algorithm;

or

when the cold and hot attribute of the file changes from hot to cold, transcoding storage processing is performed on the file from a second source coding algorithm to a second destination coding algorithm, wherein

restoration costs of the first destination coding algorithm are less than a restoration costs threshold, restoration costs of the first source coding algorithm are not less than the restoration costs threshold, storage overheads of the second destination coding algorithm are less than a storage overheads threshold, and storage overheads of the second source coding algorithm are not less than the storage overheads threshold.