SYSTEMS, METHODS, AND APPARATUSES FOR SIMPLIFYING FILESYSTEM OPERATIONS UTILIZING A KEY-VALUE STORAGE SYSTEM

• US 20190087431A1
• Filed: 09/21/2017
• Published: 03/21/2019
• Est. Priority Date: 09/21/2017
• Status: Active Grant

• Current Assignee: Alibaba Group Holding Ltd.
• Original Assignee: Alibaba Group Holding Ltd.
• Inventors: QIU, Sheng, LIU, Fei, LI, Shu
• Granted Patent: US 11,016,932 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31
• CPC Class Codes: G06F 16/16 File or folder operations, e.g. details of user interfaces specifically adapted to file systems
• CPC Class Codes: G06F 16/188 Virtual file systems

Abstract

Systems and methods for providing key-value based filesystem operations are disclosed. In one embodiment, the method comprises receiving at least one file operation from an application, the file operation including a file path and a requested operation; generating a key-value operation based on the file operation, the key-value operation including a key including the file path; executing the key-value operation at a key-value storage engine; receiving, from the key-value storage engine, a result of the key-value operation; generating a result value based on the result of the key-value operation; and returning the result value to the application.

Claims

1. A method comprising:

receiving at least one file operation from an application, the file operation including a file path and a requested operation;

generating a key-value operation based on the file operation, the key-value operation including a key including the file path;

executing the key-value operation at a key-value storage engine;

receiving, from the key-value storage engine, a result of the key-value operation;

generating a result value based on the result of the key-value operation; and

returning the result value to the application.

2. The method of claim 1 wherein the at least one file operation comprises a write operation including a number of bytes to write, and wherein generating a key-value operation based on the file operation further comprises generating a PUT request using the file path as a key and the number of bytes as a value.

3. The method of claim 1, wherein the at least one file operation comprises a transaction operation, the transaction operation including a plurality of sub-operations, wherein generating a key-value operation based on the file operation further comprises generating a key-value transaction including a plurality of key-value operations based on the sub-operations, wherein executing the key-value operation at a key-value storage engine comprises executing either all of the key-value operations or none of the key-value operations.

4. The method of claim 3, further comprising rolling back the plurality of sub-operations upon detecting an error in executing one of the plurality of sub-operations.

5. The method of claim 1, wherein the at least one file operation comprises a multi-command operation including an array of file paths and wherein generating a key-value operation based on the file operation further comprises generating a plurality of key-value instructions for each of the file paths in the array of file paths, the plurality of key-value instructions generated based on the type of the multi-command operation.

6. The method of claim 5 wherein generating a result value based on the result of the key-value operation comprises populating a result buffer with the result of each of the plurality of key-value instructions.

7. The method of claim 1 wherein the at least one file operation comprises a search operation, the search operation including a list of directories, a list of file attributes, and a query result buffer, and wherein the method further comprises:

identifying a plurality of file paths responsive to the file list of directories; and

generating a metadata key based on the plurality of file paths and the list of file attributes.

8. The method of claim 1, wherein prior to generating a key-value operation based on the file operation, the method further comprises verifying that the application has permission to access a file identified by the file path.

9. The method of claim 1, further comprising performing a filesystem operation on a file represented by the file path based on the file operation, wherein performing a filesystem operation based on the file operation comprises issuing a system call to an underlying hierarchal filesystem.

10. A device comprising:

a processor; and

a non-transitory memory storing computer-executable instructions therein that, when executed by the processor, cause the device to;

receive at least one file operation from an application, the file operation including a file path and a requested operation;

generate a key-value operation based on the file operation, the key-value operation including a key including the file path;

execute the key-value operation at a key-value storage engine;

receive, from the key-value storage engine, a result of the key-value operation;

generate a result value based on the result of the key-value operation; and

return the result value to the application.

11. The device of claim 10, wherein the at least one file operation comprises a write operation including a number of bytes to write, and wherein generating a key-value operation based on the file operation further comprises generating a PUT request using the file path as a key and the number of bytes as a value.

12. The device of claim 10, wherein the at least one file operation comprises a transaction operation, the transaction operation including a plurality of sub-operations and wherein generating a key-value operation based on the file operation further comprises generating a key-value transaction including a plurality of key-value operations based on the sub-operations, wherein executing the key-value operation at a key-value storage engine comprises executing either all of the key-value operations or none of the key-value operations.

13. The device of claim 12, wherein the computer-executable instructions further comprise instructions causing the device to roll back the plurality of sub-operations upon detecting an error in executing one of the plurality of sub-operations.

14. The device of claim 10, wherein the at least one file operation comprises a multi-command operation including an array of file paths and wherein generating a key-value operation based on the file operation further comprises generating a plurality of key-value instructions for each of the file paths in the array of file paths, the plurality of key-value instructions generated based on the type of the multi-command operation.

15. The device of claim 14, wherein generating a result value based on the result of the key-value operation comprises populating a result buffer with the result of each of the plurality of key-value instructions.

16. The device of claim 10, wherein the at least one file operation comprises a search operation, the search operation including a list of directories, a list of file attributes, and a query result buffer, and wherein the computer-executable instructions further comprise instructions causing the device to:

identify a plurality of file paths responsive to the file list of directories; and

generate a metadata key based on the plurality of file paths and the list of file attributes.

17. The device of claim 10, wherein prior to generating a key-value operation based on the file operation, the computer-executable instructions further comprise instructions causing the device to verify that the application has permission to access a file identified by the file path.

18. The device of claim 10, wherein the instructions further cause the device to perform a filesystem operation based on the file operation, performing a filesystem operation based on the file operation comprising issuing a system call to an underlying hierarchal filesystem.

19. A system comprising:

one or more applications executing on a processor;

a storage device storing a plurality of files;

a key-value storage engine configured to store a plurality of keys with associated values, the keys corresponding to file paths representing the plurality of files; and

a key-value filesystem module configured to;

receive at least one file operation from an application, the file operation including a file path and a requested operation;

generate a key-value operation based on the file operation, the key-value operation including a key including the file path;

execute the key-value operation at a key-value storage engine;

receive, from the key-value storage engine, a result of the key-value operation;

generate a result value based on the result of the key-value operation; and

return the result value to the application.

20. The system of claim 19, wherein the at least one file operation comprises one of a write operation, a transaction operation, a multi-command operation, or a search operation.