Parallel upload and download of large files using bittorrent

• US 9,374,395 B2
• Filed: 01/05/2012
• Issued: 06/21/2016
• Est. Priority Date: 03/08/2011
• Status: Expired due to Fees

• Current Assignee: Rackspace US Incorporated (Apollo Global Management, Inc.)
• Original Assignee: Rackspace US Incorporated (Apollo Global Management, Inc.)
• Inventors: Barton, Michael, Reese, Will, Dickinson, John A., Payne, Jay B., Thier, Charles B., Holt, Gregory
• Primary Examiner(s): Wang, Liangche A
• Assistant Examiner(s): AGUIAR, JOHNNY B
• Application Number:
  US13/344,014
• Publication Number:
  US 20120233293A1
• Time in Patent Office: 1,629 Days
• Field of Search: 709/203, 709/211, 709/213, 709/219, 709/226, 709/231
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1004 to protect a block of data words, e.g. CRC or checksum G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/2219 Large Object storage; Management thereof
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9566 URL specific, e.g. using aliases, detecting broken or misspelled links
• CPC Class Codes: G06F 2201/85 Active fault masking without idle spares
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0626 Reducing size or complexity of storage systems
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 43/106 using time related information in packets, e.g. by adding timestamps
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/5651 : Reducing the amount or size of exchanged application data
• CPC Class Codes: H04L 69/04 : Protocols for data compression, e.g. ROHC

Abstract

Several different embodiments of a segmented object storage system are described. The object storage system divides files into a number of object segments, each segment corresponding to a portion of the object, and stores each segment individually in the cloud storage system. The system also generates and stores a manifest file describing the relationship of the various segments to the original data file. Requests to retrieve the segmented file are fulfilled by consulting the manifest file and using the information from the manifest to reconstitute the original data file from the constituent segments. Modifying, appending to, or truncating the object is accomplished by manipulating individual segments and the manifest file. In further embodiments, manipulation of the individual object segments and/or the manifest is used to implement copy-on-write, snapshotting, software transactional memory, and peer-to-peer transmission of the large file.

Claims

1. A method for storing large object data, comprising:

receiving a request to upload an object from a user device to network-accessible storage over a network;

determining the object includes an object size exceeding a predetermined size;

dividing the object into a plurality of object segments, wherein each object segment includes a segment size below the predetermined size, wherein the segment size for a respective object segment is equal to or less than a size of a cache of an object storage entity receiving the respective object segment, and wherein each object segment has a file name, the file name including a timestamp identifying a creation time of the respective object segment and a size of the respective object segment;

generating a manifest file related to the object, wherein the manifest file allows the plurality of object segments to be retrieved from the network-accessible storage and provided as the object, and wherein the manifest file comprises data verification information for the object and file name information for each object segment;

providing the manifest file to a plurality of object storage entities, wherein a total number of object storage entities in the plurality of object storage entities is equal to a total number of object segments;

uploading the plurality of object segments to the network-accessible storage over the network, wherein each of the plurality of object storage entities stores a different uploaded object segment in the plurality of object segments, wherein each respective uploaded object segment in the plurality of object segments is uploaded to a respective object storage entity in a first zone, wherein the each respective uploaded object segment in the first zone is subject to a correlated loss of access based on a single event; and

replicating the each respective uploaded object segment in the first zone to a respective object storage entity in a second zone,

wherein the first zone and the second zone for the each respective uploaded object segment is determined based on a hash operation performed on an identifier associated with the each respective uploaded object segment.

2. The method of claim 1, wherein each of the plurality of object storage entities stores a different object segment directly from the user device.

3. The method of claim 1, further comprising:

uploading the object to a storage management server over the network, wherein each of the plurality of object storage entities uploads a different object segment from the storage management server.

4. The method of claim 3, further comprising:

deleting the object from the storage management server.

5. The method of claim 1, wherein the plurality of object segments are uploaded to the network-accessible storage in parallel.

6. The method of claim 1, further comprising:

uploading each object segment with a corresponding object storage entity and storing each object segment in the cache of the corresponding object storage entity;

writing each object segment to a disk in the corresponding object storage entity.

7. The method of claim 1, wherein the manifest file includes a torrent.

8. The method of claim 1, wherein the manifest file includes descriptor information related to each object segment, the descriptor information enabling a peer-to-peer file sharing protocol.

9. A file storage system, comprising:

a plurality of object storage entities;

a storage management server coupled to the plurality of object storage entities and a network, wherein the storage management server includes at least one processor and a computer-readable medium coupled to the at least one processor, wherein the computer-readable medium includes instructions which, when executed by the at least one processor, cause the at least one processor;

receive a request to upload an object from a user device to the plurality of object storage entities over a network;

divide the object into a plurality of object segments, wherein each object segment includes a segment size below a predetermined size, wherein the segment size for a respective object segment is equal to or less than a size of a cache in an object storage entity receiving the respective object segment, and wherein a total number of object storage entities is equal to a total number of object segments, and wherein each object segment has a file name including a timestamp identifying a creation time of the respective object segment and a size of the respective object segment;

generate a manifest file related to the object, wherein the manifest file allows the plurality of object segments to be retrieved from the plurality of object storage entities and provided over the network as the object, and wherein the manifest file comprises data verification information for the object;

provide the manifest file to the plurality of object storage entities;

upload the plurality of object segments to the plurality of object storage entities over the network, wherein each of the plurality of object storage entities stores a different uploaded object segment in the plurality of object segments, wherein each respective uploaded object segment in the plurality of object segments is uploaded to a respective object storage entity in a first zone, and wherein the each respective uploaded object segment in the first zone is subject to a correlated loss of access based on a single event; and

replicate the each respective object uploaded segment in the first zone to a respective object storage entity in a second zone,

wherein the first zone and the second zone for the each respective uploaded object segment is determined based on a hash operation performed on an identifier associated with the each respective uploaded object segment.

10. The system of claim 9, wherein each of the plurality of object storage entities uploads a different object segment directly from the user device.

11. The system of claim 9, wherein the computer-readable medium includes instructions that, when executed by the at least one processor, cause the at least one processor to:

upload the object to the storage management server over the network, wherein each of the plurality of object storage entities uploads a different object segment from the storage management server; and

delete the object from the storage management server.

12. The system of claim 9, wherein the plurality of object segments is uploaded to the object storage entities in parallel.

13. The system of claim 9, wherein the computer-readable medium includes instructions that, when executed by the at least one processor, cause the at least one processor to:

upload each object segment with the corresponding object storage entity and storing each object segment in the cache of the corresponding object storage entity;

write each object segment to a disk in the corresponding object storage entity.

14. The system of claim 9, wherein the manifest file includes a torrent.

15. The system of claim 9, wherein the manifest file includes descriptor information related to each object segment and enables a peer-to-peer file sharing protocol.