OFFLOADING UPLOAD PROCESSING OF A FILE IN A DISTRIBUTED SYSTEM USING A KEY THAT INCLUDES A HASH CREATED USING ATTRIBUTE(S) OF A REQUESTOR AND/OR THE FILE

• US 20200137141A1
• Filed: 10/24/2018
• Published: 04/30/2020
• Est. Priority Date: 10/24/2018
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Mannengal, Arun Ramadasan, Basran, Ashish, Katib, Jawad Ahmed Ibrahim, Chandru, Avinash, Datta, Shreeja Subrata
• Granted Patent: US 11,184,423 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/122 using management policies backup systems G06F11/1446; file migration policies for HSM systems G06F16/185
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: G06F 16/1865 Transactional file systems
• CPC Class Codes: G06F 2209/509 Offload
• CPC Class Codes: G06F 9/5005 to service a request
• CPC Class Codes: G06F 9/541 via adapters, e.g. between incompatible applications
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 9/0643 Hash functions, e.g. MD5, SHA, HMAC or f9 MAC
• CPC Class Codes: H04L 9/0819 Key transport or distribution, i.e. key establishment techniques where one party creates or otherwise obtains a secret value, and securely transfers it to the other(s) network architectures or network communication protocols for key distribution in a packet data network H04L63/062
• CPC Class Codes: H04L 9/0861 Generation of secret information including derivation or calculation of cryptographic keys or passwords
• CPC Class Codes: H04L 9/088 Usage controlling of secret information, e.g. techniques for restricting cryptographic keys to pre-authorized uses, different access levels, validity of crypto-period, different key- or password length, or different strong and weak cryptographic algorithms network architectures or network communication protocols for using time-dependent keys in a packet data network H04L63/068

Abstract

Techniques are described herein that are capable of offloading upload processing of a file in a distributed system. A request is received from a requestor to upload a file to a transactional database of a DBMS. Information regarding the requestor and/or the file is extracted from the request. A determination is made that the file is to be uploaded to a non-indexing file storage system in lieu of the transactional database based at least in part on the extracted information satisfying one or more criteria. A key that includes a hash is generated. The hash is created using attribute(s) of the requestor and/or the file from the extracted information. The key is provided to the requestor. The key and at least a portion of the file are received from the requestor. Uploading of the file to the non-indexing file storage system in lieu of the transactional database is initiated.

Claims

1. A system to offload upload processing of a file in a distributed system, the system comprising:

memory; and

one or more processors coupled to the memory, the one or more processors configured to;

extract information regarding at least one of a requestor or the file from a request that is received from the requestor, the request requesting upload of the file to a transactional database of a database management system in the distributed system;

determine that the file is to be uploaded to a non-indexing file storage system in lieu of the transactional database based at least in part on the extracted information satisfying one or more criteria;

generate a key that includes a hash by converting specified information, which includes at least one of (1) at least one attribute of the requestor or (2) at least one attribute of the file, from the extracted information and a secret to create the hash;

provide the key to the requestor based at least in part on a determination that the file is to be uploaded to the non-indexing file storage system in lieu of the transactional database; and

initiate uploading of the file to the non-indexing file storage system in lieu of the transactional database based at least in part on receipt of the key and at least a portion of the file from the requestor.

2. The system of claim 1, wherein the one or more processors are configured to:

determine that the file is to be uploaded to the non-indexing file storage system in lieu of the transactional database based at least in part on at least one of the following;

a file name of the file and a reference file semantically matching;

a type of the file and a reference type being same;

a size of the file being greater than or equal to a reference file size;

content of the file including reference content;

content of the file including a reference type of content;

an estimated workload associated with uploading the file to the transactional database being greater than or equal to a threshold workload;

a location of the file and a reference location being same.

3. The system of claim 1, wherein the one or more processors are configured to:

determine that the file is to be uploaded to the non-indexing file storage system in lieu of the transactional database based at least in part on at least one of the following;

an entity with which the file is associated and a reference entity being same;

a credential of the requestor and a reference credential being same;

an organization to which the requestor belongs and a reference entity being same.

4. The system of claim 1, wherein the one or more processors are configured to:

generate the key by converting the specified information, which includes at least one of a name of the file or a size of the file, from the extracted information and the secret to create the hash.

5. The system of claim 1, wherein the one or more processors are configured to execute a file upload API of the transactional database and a DBMS-agnostic API;

wherein the file upload API is configured to;

generate the key based at least in part on receipt of the request; and

forward the key to the DBMS-agnostic API; and

wherein the DBMS-agnostic API is configured to;

store the key; and

validate the key by confirming that the key was generated by the file upload API.

6. The system of claim 1, wherein the one or more processors are configured to:

configure the key to expire after a finite period of time; and

initiate uploading of the file to the non-indexing file storage system in lieu of the transactional database based at least in part on receipt of the key from the requestor within the finite period of time.

7. The system of claim 6, wherein the one or more processors are configured to:

persist the key in a distributed data store and store the key in a cache;

validate the key that is received from the requestor by (1) comparing the key that is received from the requestor and at least one of the key persisted in the distributed data store or the key stored in the cache to determine that the key that is received from the requestor and the at least one of the key persisted in the distributed data store or the key stored in the cache are same and (2) verifying that the key is received from the requestor within the finite period of time; and

upload at least the portion of the file to the non-indexing file storage system based at least in part on validation of the key that is received from the requestor.

8. The system of claim 7, wherein the one or more processors are configured to:

initiate the uploading of the file to the non-indexing file storage system in lieu of the transactional database based at least in part on the receipt of the key and at least the portion of the file from the requestor during a first period of time;

validate the key that is received from the requestor during a second period of time that is subsequent to the first period of time based at least in part on receipt of the key and at least a second portion of the file from the requestor during the second period of time by (1) comparing the key that is received from the requestor during the second period of time and at least one of the key persisted in the distributed data store or the key stored in the cache to determine that the key that is received from the requestor during the second period of time and the at least one of the key persisted in the distributed data store or the key stored in the cache are same and (2) verifying that the key that is received from the requestor during the second period of time is received from the requestor within the finite period of time; and

upload at least the second portion of the file to the non-indexing file storage system based at least in part on validation of the key that is received from the requestor during the second period of time.

9. The system of claim 1, wherein the one or more processors are configured to:

upload the file to a storage location in the non-indexing file storage system;

commit the file in the non-indexing file storage system; and

provide a pointer in the transactional database, the pointer pointing to the storage location in the non-indexing file storage system.

10. The system of claim 1, wherein the one or more processors are configured to:

analyze content of the file to determine whether the content includes disallowed content; and

not commit the file in the non-indexing file storage system based at least in part on the content including the disallowed content.

11. The system of claim 1, wherein the one or more processors are configured to:

upload a plurality of portions of the file in parallel to the non-indexing file storage system in lieu of the transactional database based at least in part on the receipt of the key from the requestor.

12. The system of claim 1, wherein the one or more processors are configured to:

initiate uploading of a plurality of portions of the file to the non-indexing file storage system in lieu of the transactional database;

determine whether a portion of the plurality of portions failed to upload; and

provide a notification to the requestor based at least in part on the portion failing to upload, the notification distinguishing the portion from the other portions and instructing the requestor to cause uploading of the portion to be re-initiated.

13. The system of claim 1, wherein the non-indexing file storage system is a distributed non-indexing file storage system;

and

wherein the one or more processors are configured to;

upload the file in a distributed manner to the distributed non-indexing file storage system.

14. The system of claim 1, wherein the one or more processors are configured to:

provide analytics regarding the file to the requestor as the file is being uploaded to the non-indexing file storage system in lieu of the transactional database.

15. A method of offloading upload processing of a file in a distributed system, the method comprising:

receiving a request from a requestor to upload the file to a transactional database of a database management system in the distributed system;

extracting information regarding at least one of the requestor or the file from the request;

determining that the file is to be uploaded to a non-indexing file storage system in lieu of the transactional database based at least in part on the extracted information satisfying one or more criteria;

generating a key that includes a hash by converting specified information, which includes at least one of (1) at least one attribute of the requestor or (2) at least one attribute of the file, from the extracted information and a secret to create the hash;

providing the key to the requestor based at least in part on a determination that the file is to be uploaded to the non-indexing file storage system in lieu of the transactional database;

receiving the key and at least a portion of the file from the requestor; and

initiating uploading of the file to the non-indexing file storage system in lieu of the transactional database based at least in part on receipt of the key from the requestor.

16. The method of claim 15, wherein generating the key comprises:

generating the key by converting the specified information, which includes at least one of a name of the file or a size of the file, from the extracted information and the secret to create the hash.

17. The method of claim 15, wherein receiving the request comprises:

receiving the request at a file upload API of the transactional database;

wherein generating the key comprises;

generating the key by the file upload API; and

wherein the method further comprises;

forwarding the key from the file upload API of the transactional database to a DBMS-agnostic API, which stores the key; and

validating the key by the DBMS-agnostic API by confirming that the key was generated by the file upload API.

18. The method of claim 15, wherein generating the key comprises:

configuring the key to expire after a finite period of time; and

wherein initiating uploading of the file comprises;

initiating uploading of the file to the non-indexing file storage system in lieu of the transactional database based at least in part on receipt of the key from the requestor within the finite period of time.

19. The method of claim 18, further comprising:

persisting the key in a distributed data store and storing the key in a cache; and

validating the key that is received from the requestor, the validating comprising;

comparing the key that is received from the requestor and at least one of the key persisted in the distributed data store or the key stored in the cache to determine that the key that is received from the requestor and the at least one of the key persisted in the distributed data store or the key stored in the cache are same; and

verifying that the key is received from the requestor within the finite period of time;

wherein initiating the uploading of the file comprises;

uploading at least the portion of the file to the non-indexing file storage system based at least in part on validation of the key that is received from the requestor.

20. A computer program product comprising a computer-readable storage medium having instructions recorded thereon for enabling a processor-based system to perform operations to offload upload processing of a file in a distributed system, the operations comprising:

extract information regarding at least one of a requestor or the file from a request that is received from the requestor, the request requesting upload of the file to a transactional database of a database management system in the distributed system;

determine that the file is to be uploaded to a non-indexing file storage system in lieu of the transactional database based at least in part on the extracted information satisfying one or more criteria;

generate a key that includes a hash by converting specified information, which includes at least one of (1) at least one attribute of the requestor or (2) at least one attribute of the file, from the extracted information and a secret to create the hash;

provide the key to the requestor based at least in part on a determination that the file is to be uploaded to the non-indexing file storage system in lieu of the transactional database; and

initiate uploading of the file to the non-indexing file storage system in lieu of the transactional database based at least in part on receipt of the key and at least a portion of the file from the requestor.