Method for handling large object files in an object storage system

• US 8,990,257 B2
• Filed: 12/23/2011
• Issued: 03/24/2015
• Est. Priority Date: 03/08/2011
• Status: Expired due to Fees

• Current Assignee: Rackspace US Incorporated (Apollo Global Management, Inc.)
• Original Assignee: Rackspace US Incorporated (Apollo Global Management, Inc.)
• Inventors: Barton, Michael, Reese, Will, Dickinson, John A., Payne, Jay B., Thier, Charles B., Holt, Gregory
• Primary Examiner(s): GIRMA, ANTENEH B
• Application Number:
  US13/336,175
• Publication Number:
  US 20120233522A1
• Time in Patent Office: 1,187 Days
• Field of Search: None
• US Class Current: 707/792
• CPC Class Codes: G06F 11/1004 to protect a block of data words, e.g. CRC or checksum G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/2219 Large Object storage; Management thereof
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9566 URL specific, e.g. using aliases, detecting broken or misspelled links
• CPC Class Codes: G06F 2201/85 Active fault masking without idle spares active fault masking without idle spare hardware where processing functionality is redundant G06F11/2035
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0626 Reducing size or complexity of storage systems
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 43/106 using time related information in packets, e.g. by adding timestamps
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/5651 : Reducing the amount or size of exchanged application data
• CPC Class Codes: H04L 69/04 : Protocols for data compression, e.g. ROHC

Abstract

Several different embodiments of a segmented object storage system are described. The object storage system divides files into a number of object segments, each segment corresponding to a portion of the object, and stores each segment individually in the cloud storage system. The system also generates and stores a manifest file describing the relationship of the various segments to the original data file. Requests to retrieve the segmented file are fulfilled by consulting the manifest file and using the information from the manifest to reconstitute the original data file from the constituent segments. Modifying, appending to, or truncating the object is accomplished by manipulating individual segments and the manifest file. In further embodiments, manipulation of the individual object segments and/or the manifest is used to implement copy-on-write, snapshotting, software transactional memory, and peer-to-peer transmission of the large file.

Claims

1. A method for storing object data in a cloud storage system, the method comprising:

receiving, by one or more processors, a request to store a computer readable information object in network-accessible storage;

receiving, by one or more processors, a first data range associated with the computer readable information object; and

evaluating, by one or more processors, whether to segment the computer readable information object;

wherein, based on the evaluating, if the computer readable information object is to be segmented, further performing using one or more processors;

segmenting the computer readable information object into a plurality of object segments, each segment corresponding to a portion of the computer readable information object;

storing each of the plurality of object segments individually in network-accessible storage;

generating a manifest file related to the plurality of object segments; and

storing the manifest file in computer readable storage;

wherein the manifest file allows the plurality of object segments to be retrieved from network-accessible storage and to be reconstituted into a data range equal to the first data range; and

wherein if, based on the evaluating, the computer readable information object is not to be segmented, storing, by one or more processors, the first data range associated with the computer readable information object in network-accessible storage.

2. The method of claim 1, wherein segmenting the computer readable information object is performed before the data corresponding to the object is received.

3. The method of claim 1, wherein the step of segmenting the computer readable information object occurs as the data corresponding to the computer readable information object is received.

4. The method of claim 1, further comprising the step of balancing data stored in a plurality of storage pools provided in the network-accessible storage, wherein the balancing includes storing a first of the plurality of object segments in a first of the plurality of storage pools and storing a second of the plurality of object segments in a second of the plurality of storage pools.

5. The method of claim 1, further comprising the step of replicating the plurality of object segments in a plurality of storage pools provided in the network-accessible storage, wherein the replicating includes replicating a first of the plurality of object segments in a first of the plurality of storage pools and replicating a second of the plurality of object segments in a second of the plurality of storage pools.

6. The method of claim 1, further comprising the steps of performing a data verification operation on a first of the plurality of object segments to produce a data verification result;

and associating the data verification result with the first of the plurality of object segments in the manifest file.

7. The method of claim 1, further comprising the steps of performing a forward error correction calculation on a first of the plurality of object segments to produce an error correction result;

and associating the error correction result with the first of the plurality of object segments in the manifest file.

8. The method of claim 1, wherein evaluating whether to segment the computer readable information object includes the step of determining that the computer readable information object size exceeds a predetermined size.

9. The method of claim 1, wherein segmenting the computer readable information object into a plurality of object segments includes the step of determining each segment does not exceed a predetermined size.

10. The method of claim 1, wherein evaluating whether to segment the computer readable information object includes the step of examining the request for an indicia that the object should be segmented.

11. The method of claim 1, further comprising the step of receiving a second data range associated with the computer readable information object, and wherein evaluating whether to segment the computer readable information object occurs after receiving the second data range.

12. The method of claim 1, further comprising the step of receiving a second data range associated with the computer readable information object, and wherein a first segment of the plurality of object segments corresponds to the first data range, and wherein a second segment of the plurality of object segments corresponds to the second data range.

13. The method of claim 1, wherein segmenting the computer readable information object into a plurality of object segments is performed by separating out distinct byte ranges from the computer readable information object.

14. The method of claim 1, wherein segmenting the object into a plurality of object segments is performed by segmenting the computer readable information object based upon information content within the computer readable information object.

15. The method of claim 14, wherein the computer readable information object is a structured information repository, and wherein segmenting the computer readable information object into a plurality of object segments is based upon information content within the computer readable information object is performed by identifying selected records within the structured information repository and creating a second structured information repository containing only the selected records.

16. A method for providing object data from a cloud storage system, the method comprising:

storing each of a plurality of computer readable information object segments individually, wherein each of the plurality of computer readable information object segments correspond to a portion of the data from a first data source;

receiving a computer readable request to download object data corresponding to one or more of the object segments; and

responding to the request with object data from the one or more of the computer readable information object segments;

wherein if the requested object data corresponds to less than all the data from the first data source, the step of responding to the request with object data further includes retrieving the object segments corresponding only to the requested object data and returning the requested data; and

wherein if the requested object data corresponds to all the data from the first data source, the step of responding to the request with object data further includes the steps of receiving a manifest file describing the relationships of the object segments to the first data source; and

reconstituting the requested object data from the plurality of object segments.

17. The method of claim 16, wherein if the requested object data corresponds to less than all the data from the first data source, but more than one object segment, further including the step of receiving a manifest file describing the relationships of the object segments to the first data source;

and reconstituting the requested object data from the plurality of object segments.

18. The method of claim 16, wherein reconstituting the requested object data from the plurality of computer readable information object segments is performed by concatenating byte ranges from the plurality of computer readable information object segments.

19. The method of claim 16, wherein reconstituting the requested object data from the plurality of computer readable information object segments is performed by recreating a logical structure equivalent to the first data source.

20. A system for storing a file, comprising:

a storage management server, the storage management server further comprising a processor, a file storage module, and a file retrieval module;

at least one computer-readable store coupled to the storage management server;

wherein the file storage module further includes a file segmenter, a manifest generator, and a storage writer;

wherein the file segmenter receives a first data file for storage and segments the first data file into a plurality of segment data files, each segment data file corresponding to a portion of the first data file;

the manifest generator records the relationship of the plurality of segmented data files to the first data file in a manifest; and

the file writer stores the manifest and each segment data file to the computer-readable store as a separate file object;

wherein the file retrieval module further includes a file reader and a file desegmenter;

wherein the file reader retrieves the manifest and the plurality of segment data files from the computer-readable store and provides them to the file desegmenter; and

the file desegmenter reconstitute reconstitutes the first data file using the provided manifest and segmented data files.