File processing method, system and server-clustered system for cloud storage

• US 9,774,564 B2
• Filed: 05/31/2012
• Issued: 09/26/2017
• Est. Priority Date: 06/17/2011
• Status: Active Grant

• Current Assignee: Alibaba Group Holding Ltd.
• Original Assignee: Alibaba Group Holding Ltd.
• Inventors: Quan, Xiao Fei, Xue, Quirong, Zhuo, Lei, Song, Nan, Song, Wanli, Cheng, Han
• Primary Examiner(s): Gillis, Brian J
• Assistant Examiner(s): Ling, Amy
• Application Number:
  US13/576,213
• Publication Number:
  US 20130073691A1
• Time in Patent Office: 1,944 Days
• Field of Search: 709219
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: H04L 61/457 containing identifiers of data entities on a computer, e.g. file names
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]

Abstract

A server receives a request to upload a file form a user device. The server may generate a unique file ID associated with the file based on the request, and transmit the file ID to the user device. The user device may divide the file to generate multiple slice files and their corresponding slice IDs. From the user device, the server receives the multiple slice files and the slice IDs. Based on the slice IDs, the server may determine storage addresses at which to store the multiple slice files in a distributed storage system using a consistent hash algorithm. The multiple slice files may be stored based on the corresponding storage addresses of the distrusted storage system.

Claims

1. One or more computer-readable media storing computer-executable instructions that, when executed by one or more processors associated with a server, instruct the one or more processors to perform acts comprising:

receiving an upload request to upload a file from a user device, the upload request including at least a user identifier (ID), a file name, and location information of the file in the user device;

generating a file ID for the file using sequentially increased numbers and the location information of the file in the user device that is included in the upload request;

transmitting the file ID to the user device;

receiving a duplicate determination request including the file ID and multiple slice IDs from the user device, wherein each slice ID of the multiple slice IDs is generated and associated with a respective slice file of multiple slice files, and the multiple slice files are generated by dividing the file;

determining whether one or more duplicate slice IDs exist in the multiple slice IDs that individually correspond to the multiple slice files associated with the file in response to receiving the duplicate determination request from the user device;

sending respective one or more slice IDs of one or more non-duplicate slice files associated with the file to the user device after the determining; and

receiving a slice file package including at least the file ID and the one or more non-duplicate slice files from the user device.

2. The one or more computer-readable media of claim 1, wherein the acts further comprise storing the slice file package such that the file ID is associated with the respective one or more slice IDs of the one or more non-duplicate slice files, and storing the slice file package includes:

computing a storage address of an individual slice file of the one or more non-duplicate slice files using a consistent hash algorithm for the respective one or more slice IDs; and

storing the individual slice file based on the storage address.

3. The one or more computer-readable media of claim 1, wherein the one or more non-duplicate slice files comprise divisions of the slice file package that are of a predetermined size.

4. The one or more computer-readable media of claim 1, wherein the acts further comprise:

receiving an additional upload request to upload an additional file associated with the file ID, the additional upload request being associated with an additional user ID;

determining that the additional file comprises a duplicate of an individual slice file of the multiple slice files;

assigning a count number to an individual slice ID associated with the individual slice file of the slice files, the count number being set as zero;

determining that the duplicate of the individual slice file is associated with the individual slice ID; and

resetting the count number to be one after the duplicate of the individual slice file is stored in response to the additional upload request.

5. The one or more computer-readable media of claim 1, wherein each slice file of the multiple slice files includes a slice ID and data associated with the slice ID.

6. The one or more computer-readable media of claim 5, wherein each slice file of the multiple slice files is assigned a respective slice ID using a hash function.

7. The one or more computer-readable media of claim 1, wherein the acts further comprise:

setting reference counters for the multiple slice IDs of the multiple slice files, wherein a count value of each reference counter is initially set at zero; and

modifying a count value of a corresponding reference counter of an individual slice ID of the multiple slice IDs based on one or more of the duplicate determination request or a deletion request.

8. The one or more computer-readable media of claim 1, wherein the acts further comprise recording an association among the file ID, the user ID, the file name and the location information of the file in the user device that are included in the upload request.

9. A computer-implemented method comprising:

receiving a request to upload a file from a user device, the request including at least a user identifier (ID), a file name, and location information of the file in the user device;

generating a file ID for the file using sequentially increased numbers and the location information of the file in the user device that is included in the request;

transmitting the file ID to the user device;

receiving a duplicate determination request including the file ID and multiple slice IDs from the user device, wherein each slice ID of the multiple slice IDs is generated and associated with a respective slice file of multiple slice files, and the multiple slice files are generated by dividing the file;

determining whether one or more duplicate slice IDs exist in the multiple slice IDs that individually correspond to the multiple slice files associated with the file in response to receiving the duplication determination request from the user device;

sending respective one or more slice IDs of one or more non-duplicate slice files associated with the file to the user device after the determining; and

receiving a slice file package including at least the file ID and the one or more non-duplicate slice files from the user device.

10. The computer-implemented method of claim 9, further comprising computing a storage address for an individual slice file of the multiple slice files, wherein computing the storage address comprises computing the storage address using a consistent hash algorithm for the multiple slice IDs associated with the multiple slice files.

11. The computer-implemented method of claim 10, further comprising storing the individual slice file based on the storage address, wherein the storing the individual slice file includes associating the individual slice file with the file ID of the file.

12. The computer-implemented method of claim 9, wherein the one or more non-duplicate slice files of the multiple slice files comprise divisions of the slice file package that are of a predetermined size.

13. The computer-implemented method of claim 9, further comprising:

determining, at least partly responsive to receiving the duplicate determination request, that an individual slice file associated with an individual slice ID of the multiple slice IDs is not stored; and

storing the individual slice file based on the individual slice ID.

14. The computer-implemented method of claim 9, further comprising:

receiving a download request from an individual user device to download the file;

computing storage addresses of the multiple slice files using a consistent hash algorithm for the multiple slice IDs corresponding to the multiple slice files;

generating the file based on the multiple slice files; and

transmitting the file to the individual user device.

15. The computer-implemented method of claim 9, further comprising:

receiving an additional upload request to upload an additional file associated with the file ID, the additional upload request being associated with an additional user ID;

determining that the additional file comprises a duplicate of an individual slice file of the multiple slice files;

assigning a count number to an individual slice ID associated with the individual slice file of the multiple slice files, the count number being set as zero;

determining that the duplicate of the individual slice file is associated with the individual slice ID;

resetting the count number to be one after the duplicate of the slice file is stored in response to the additional upload request.

16. The computer-implemented method of claim 9, further comprising:

assigning a count number to an individual slice ID associated with an individual slice file of the multiple slice files, the count number being set as zero;

receiving a delete request to delete the individual slice file; and

deleting the individual slice file based at least in part on receiving the delete request determining that the count number associated with the individual slice ID is zero.

17. A user device comprising:

one or more processors;

memory storing executable instructions that, when executed by the one or more processors, to cause the one or more processors to perform acts comprising;

transmitting a upload request to upload a file to a server, the upload request including at least a user identifier (ID), a file name, and location information of the file in the user device;

receiving a file ID associated with the file from the server, the file ID being generated based at least in part on sequentially increased numbers and the location information of the file in the user device included in the upload request;

dividing the file into multiple slice files;

generating multiple slice IDs and associating each of the multiple slice IDs with a respective slice file of the multiple slice files;

transmitting a duplicate determination request including the file ID and the multiple slice IDs to the server;

receiving duplication information including respective one or more slice IDs of one or more non-duplicate slice files associated with the file, wherein one or more of the multiple slice IDs are determined to be duplicate slice IDs that correspond to duplicate slice files of the multiple slice files; and

transmitting a slice file data packet including at least the file ID and the one or more non-duplicate slice files of the multiple slice files.

18. The system of claim 17, wherein each slice file of the multiple slice files includes data associated with the slice ID.

19. The system of claim 17, wherein the upload request further includes a title of the file.

20. The system of claim 17, wherein the slice file data package further includes respective one or more slice IDs of the one or more non-duplicate slice files, and respective data of the one or more non-duplicate slice files.