Large scale file storage in cloud computing

• US 9,262,423 B2
• Filed: 09/27/2012
• Issued: 02/16/2016
• Est. Priority Date: 09/27/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Borshack, Ronen, Thomas, Anil Francis, Einav, Erez, Taron, Philip Ernst
• Primary Examiner(s): Trujillo, James
• Assistant Examiner(s): TESSEMA, AIDA Z
• Application Number:
  US13/629,053
• Publication Number:
  US 20140089273A1
• Time in Patent Office: 1,237 Days
• Field of Search: 707/822, 707/698
• US Class Current: 1/1
• CPC Class Codes: G06F 16/10 File systems; File servers
• CPC Class Codes: G06F 16/137 Hash-based content-based indexing of textual data G06F16/31

Abstract

Storing and retrieving files based on hashes for the files. One method for storing files includes: identifying a file; identifying a hash calculated based on the file; renaming the file based on the hash based on the file; and storing the file in a particular location based on the hash calculated based on the file. Another method for retrieving files includes: identifying a hash for a given file; using the hash, traversing a hierarchical file structure to find a location where the given file should be stored; determining that the file is at the location; and as a result, retrieving the file.

Claims

1. In a computing environment, a method of storing files, the method comprising:

identifying a file;

determining, based on a header of the file, that the file is a compressed container file that includes a plurality of compressed files;

identifying a hash calculated based on content of the compressed container file;

using a first portion of the hash, identifyig a particular file storage account from among a plurality of file storage accounts under which the compressed container file will be stored;

using a second portion of the hash, identifying a particular blob container from among a plurality of blob containers within the particular file storage account into which the compressed container file will be stored;

renaming the compressed container file based on the hash, the compressed container file being renamed a file name that includes one or more portions of the hash;

storing the compressed container file in the particular blob container within the particular file storage account;

decompressing each of the plurality of compressed files from the compressed container file, to obtain a plurality of decompressed files;

for each of the plurality of decompressed files;

identifying a hash corresponding to said decompressed file, the hash calculated based on content of said decompressed file;

using the corresponding hash, identifying an appropriate blob container from among the plurality of blob containers within an appropriate file storage account from among the plurality of file storage accounts into which said decompressed file will be stored;

renaming said decompressed file based on the corresponding hash, said decompressed file being renamed a file name that includes one or more portions of the corresponding hash; and

storing said decompressed file in the appropriate blob container within the appropriate file storage account; and

storing metadata linking each of the plurality of decompressed files to the compressed container file.

2. The method of claim 1, wherein the hash is a hexadecimal string converted from a binary value of the compressed container file, and wherein a single letter from the hexadecimal string is used as the first portion of the hash to identify the particular file storage location and one or more different letters from the hexadecimal string are used as the second portion of the hash to identify the particular blob container within the file storage location.

3. The method of claim 2, further comprising using different portions of the hash, excluding the first and second portions, as the file name.

4. The method of claim 1, further comprising:

identifying a second file;

identifying a hash calculated based on the second file;

determining that the hash calculated based on the second file is the same as the hash calculated based on the compressed container file; and

as a result, determining to not store the second file due to the second file being redundant to the compressed container file.

5. The method of claim 1, further comprising storing an original file name of the file as metadata of the file.

6. The method of claim 1, further comprising storing at least one of a file size of the file, a creation date of the file, or an edit date of the file as metadata of the file.

7. The method of claim 1, further comprising practicing the method in a distributed computing environment to handle a large set of files and to reduce or eliminate duplication of files in the large set of files.

8. The method of claim 1, wherein storing metadata linking each of the plurality of decompressed files to the compressed container file comprises storing the metadata in a relationships table of a database.

9. A computing system for retrieving files, the computing system comprising:

one or more processors; and

one or more computer readable media, wherein the one or more computer readable media comprise computer executable instructions that are executable by at least one of the one or more processors and that configure at least one of the one or more processors to perform the following;

identify a hash for a given file;

access metadata linking the hash to a compressed container file that is stored in a storage system and that includes a plurality of compressed files, and identify a plurality of additional hashes that each correspond to a different one of a plurality of decompressed files that are also stored in the storage system, each of the plurality of decompressed files corresponding to a different one of the plurality of compressed files; and

for each of the plurality of additional hashes;

using said additional hash, traverse a hierarchical file structure of the storage system to find a location where the corresponding decompressed file should be stored, by at least using a first portion of said additional hash to identify a file storage account under which the corresponding decompressed file is stored, and using a second portion of said additional hash to identify a blob container within the file storage account into which the corresponding decompressed file is stored;

determine that the corresponding decompressed file is at the location defined by the blob container within the file storage account;

as a result, retrieve the corresponding decompressed file from the storage system.

10. The computing system of claim 9, wherein the hash is a hexadecimal string converted from a binary value of the given file, and wherein a single letter from the hexadecimal string is used as the first portion of the hash and one or more other letters from the hexadeximal string is used as the second portion of the hash.

11. The computing system of claim 9, wherein the executable instructions also configure at least one of the one or more processors to retrieve an original file name of each plurality of decompressed files from the metadata.

12. The computing system of claim 9, wherein the executable instructions also configure at least one of the one or more processors to retrieve, for at least one of the plurality of decompressed files, at least one of file size, creation date, or edit date from the metadata.

13. The computing system of claim 9, wherein the computing system is in a distributed computing environment, to handle a large set of files and to reduce or eliminate duplication of files in the large set of files.

14. A computing system for storing files, the computing system comprising:

one or more processors; and

one or more computer readable media, wherein the one or more computer readable media comprise computer executable instructions that are executable by at least one of the one or more processors and that configure at least one of the one or more processors to perform the following;

identify a file;

determine, based on a header of the file, that the file is a compressed container file that includes a plurality of compressed files;

identify a hash calculated based on content of the compressed container file;

using a first portion of the hash, identify a particular file storage account in a distributed computing environment in which the file will be stored;

using a second portion of the hash, identify a particular blob container hierarchically below the particular file storage account;

rename the compressed container file based on the hash, the compressed container file being renamed a file name that includes one or more portions of the hash;

store the compressed container file in the blob container

decompress each of the plurality of compressed files from the compressed container file, to obtain a plurality of decompressed files;

for each of the plurality of decompressed files;

identify a hash corresponding to said decompressed file, the hash calculated based on content of said decompressed file;

using the corresponding hash, identify an appropriate blob container within an appropriate file storage account into which said decompressed file will be stored;

rename said decompressed file based on the corresponding hash, said decompressed file being renamed a file name that includes one or more portions of the corresponding hash; and

store said decompressed file in the appropriate blob container within the appropriate file storage account; and

store metadata linking each of the plurality of decompressed files to the compressed container file.

15. The computing system of claim 14, wherein the executable instructions also configure at least one of the one or more processors to store an original file name of the file as metadata of the file.

16. The computing system of claim 14, wherein the executable instructions also configure at least one of the one or more processors to store at least one of file size of the file, creation date of the file, or edit date of the file as metadata of the file.

17. The computing system of claim 14, wherein storing metadata linking each of the plurality of decompressed files to the compressed container file comprises storing the metadata in a relationships table of a database.

18. The computing system of claim 14, wherein the executable instructions also configure at least one of the one or more processors to perform the following:

identify a second file;

identify a hash calculated based on the second file;

determine that the hash calculated based on the second file is the same as the hash calculated based on the compressed container file; and

as a result, determine to not store the second file due to the second file being redundant to the compressed container file.

19. The computing system of claim 14, wherein the executable instructions also configure at least one of the one or more processors to use different portions of the hash, excluding the first and second portions, as the file name.

20. The computing system of claim 14, wherein the hash is a hexadecimal string converted from a binary value of the compressed container file, and wherein a single letter from the hexadecimal string is used as the first portion of the hash to identify the particular file storage location and one or more different letters from the hexadecimal string are used as the second portion of the hash to identify the particular blob container within the file storage location.