Appending to files via server-side chunking and manifest manipulation

• US 9,306,988 B2
• Filed: 12/23/2011
• Issued: 04/05/2016
• Est. Priority Date: 03/08/2011
• Status: Active Grant

• Current Assignee: Rackspace US Incorporated (Apollo Global Management, Inc.)
• Original Assignee: Rackspace US Incorporated (Apollo Global Management, Inc.)
• Inventors: Barton, Michael, Reese, Will, Dickinson, John A., Payne, Jay B., Thier, Charles B., Holt, Gregory
• Primary Examiner(s): MENG, JAU SHYA
• Application Number:
  US13/336,095
• Publication Number:
  US 20120233228A1
• Time in Patent Office: 1,565 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1004 to protect a block of data words, e.g. CRC or checksum G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/2219 Large Object storage; Management thereof
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9566 URL specific, e.g. using aliases, detecting broken or misspelled links
• CPC Class Codes: G06F 2201/85 Active fault masking without idle spares active fault masking without idle spare hardware where processing functionality is redundant G06F11/2035
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0626 Reducing size or complexity of storage systems
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 43/106 using time related information in packets, e.g. by adding timestamps
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/5651 : Reducing the amount or size of exchanged application data
• CPC Class Codes: H04L 69/04 : Protocols for data compression, e.g. ROHC

Abstract

A segmented object storage system is an object storage system that divides files into a number of object segments, each segment corresponding to a portion of an object, and stores each segment individually in the storage system. The system also generates and stores a manifest file describing the relationship of the various segments to the original data file. Requests to retrieve the segmented file are fulfilled by consulting the manifest file and using the information from the manifest to reconstitute the original data file from the constituent segments. Modifying, appending to, or truncating the object is accomplished by manipulating individual segments and the manifest file. Manipulation of the individual object segments and/or the manifest can be used to implement copy-on-write, snapshotting, software transactional memory, and peer-to-peer transmission of the large file.

Claims

1. A method for modifying object data in a cloud storage system, the method comprising:

storing, using one or more processors, a segmented file as a plurality of individually stored object segments and a manifest, wherein each of the plurality of individually stored object segments corresponds to a portion of data in the segmented file, and wherein the manifest describes the relationship of the plurality of individually stored object segments to the segmented file sufficient to reconstitute the segmented file,

wherein the storing a segmented file as a plurality of individually stored object segments includes storing a first individually stored object segment of the plurality of individually stored object segments into a first storage pool located in a first zone, and data stored within the first storage pool are subject, based on a single event, to a correlated loss of access;

receiving a first data range and an instruction to include object data from the first data range in the segmented file;

storing information from the first data range in at least one new object segment;

calculating a first checksum for the first individually stored object segment, the first checksum being associated with the first individually stored object segment in the manifest;

replicating the first individually stored object segment;

calculating a second checksum for the replicated first individually stored object segment;

storing the replicated first individually stored object segment into a second storage pool different from the first storage pool in response to determining that the first checksum matches the second checksum, wherein the second storage pool is located in a second zone different from the first zone, and data stored within the second storage pool are subject to a correlated loss of access; and

modifying the manifest to reference the object data from the first data range as well as information from the plurality of preexisting object segments, and the replicated first individually stored object segment is stored in the second storage pool rather than the first individually stored object segment, wherein a reconstructed segmented file is based on the modified manifest.

2. The method of claim 1, wherein the instruction to include object data is an append instruction, and wherein modifying the manifest includes adding relational information to the manifest that indicates the at least one new object segment follows the plurality of preexisting object segments.

3. The method of claim 1, wherein the instruction to include object data is an insert instruction, and wherein modifying the manifest includes adding relational information to the manifest that indicates the at least one new object segment is between two of the plurality of preexisting object segments.

4. The method of claim 1, wherein the instruction to include object data is a modify instruction, and wherein the modifying the manifest includes adding relational information to the manifest that indicates the at least one new object segment is to be included in place of at least one of the plurality of preexisting object segments.

5. The method of claim 1, wherein the method further includes determining that at least one of the plurality of preexisting object segments should not be included in the segmented file;

and modifying the manifest to identify a second reconstruction of the segmented file, wherein the second reconstruction does not include information from the at least one of the non-included preexisting object segments.

6. The method of claim 5, wherein the method further includes after determining that at least one of the plurality of preexisting object segments should not be included in the segmented file and modifying the manifest to identify a third reconstruction of the segmented file, wherein the third reconstruction does not include information from the at least one of the non-included preexisting object segments, deleting at least one of the non-included preexisting object segments.

7. The method of claim 5, wherein determining that at least one of the plurality of preexisting object segments should not be included in the segmented file further includes identifying at least one preexisting object segment older than a threshold time.

8. The method of claim 5, wherein determining that at least one of the plurality of preexisting object segments should not be included in the segmented file further includes evaluating whether including the object data from the first data range in the segmented file will cause the segmented file to become larger than a threshold size.

9. The method of claim 5, wherein determining that at least one of the plurality of preexisting object segments should not be included in the segmented file further includes identifying a window of interest in the segmented file, the window of interest corresponding to one or more individually stored object segments;

and not including data outside the window of interest.

10. The method of claim 1, wherein the segmented file is a structured data file, and wherein each of the plurality of individually stored object segments is a structured data file including a portion of the information represented by the segmented data file.

11. The method of claim 10, further including:

reconstituting the segmented file, wherein reconstituting the segmented file includes performing a reduce operation to merge information from the plurality of individually stored object segments into a single result.

12. The method of claim 10, wherein the segmented file can be partially reconstituted to give a query result including less than the entire contents of the segmented file, wherein the query result is created by performing a reduce operation to merge information from the plurality of individually stored object segments into a single result.

13. The method of claim 1, wherein an individually stored object segment of the plurality of individually stored object segments includes a header that includes the manifest.

14. The method of claim 1, further including:

determining to retrieve the replicated first individually stored object segment stored in the second storage pool rather than the first object segment stored in the first storage pool, wherein the modifying the manifest includes modifying the manifest based on the determination to retrieve the replicated first individually stored object segment stored in the second storage pool rather than the first individually stored object segment stored in the first storage pool.

15. A method for implementing localized changes to a data object in a cloud storage system, the method comprising:

storing, using one or more processors, a segmented file as a plurality of individually stored object segments and a first manifest, wherein each of the plurality of individually stored object segments corresponds to a portion of data in the segmented file, wherein a first stored object segment of the plurality of individually stored object segments is stored in a first storage pool, wherein the first manifest describes the relationship of the plurality of individually stored object segments to the segmented file sufficient to reconstitute the segmented file, and wherein the first storage pool is located in a first zone, and data stored within the first storage pool are subject, based on a single event, to a correlated loss of access;

receiving a request for write access to the segmented file;

creating a copy-on-write segmented file responsive to the request for write access to the segmented file, wherein creating the copy-on-write segmented file includes creating a second manifest, wherein the second manifest initially describes the relationship of the plurality of individually stored object segments to the segmented file in a manner identical to the first manifest;

receiving a first data range and an instruction to include object data from the first data range in the copy-on-write segmented file;

storing information from the first data range in at least one new object segment in a second storage pool, wherein second storage pool is located in a second zone different from the first zone, and data stored within the second storage pool are subject to a correlated loss of access;

calculating a first checksum for the first stored object segment, the first checksum being associated with the first stored object segment in the first manifest;

replicating the first stored object segment;

calculating a second checksum for the replicated first stored object segment;

storing the replicated first stored object segment into a third storage pool different from the first storage pool in response to determining that the first checksum matches the second checksum; and

modifying the second manifest to reference the object data from the first data range as well as information from the plurality of preexisting object segments, and the replicated first stored object segment is stored in the third storage pool rather than the first stored object segment, wherein a reconstructed segmented file is based on the modified second manifest.

16. The method of claim 15, wherein the method further includes evaluating whether there are any conflicting changes to the first manifest between the time of the creation of the second manifest, and selecting the first manifest if there are conflicting changes, and selecting the second manifest if there are no conflicting changes.

17. The method of claim 15, wherein the method further includes evaluating whether there are any conflicting changes to the first manifest between the time of the creation of the second manifest, and selecting the first manifest if there are conflicting changes, and merging the first and second manifests to create a third merged manifest if there are no conflicting changes.

18. The method of claim 15, wherein the method further includes receiving a second request for write access to the segmented file;

creating a second copy-on-write segmented file responsive to the second request to interact with the segmented file, including creating a third manifest;

receiving a second data range and an instruction to include object data from the second data range in the second copy-on-write segmented file;

storing information from the second data range in at least a second new object segment; and

modifying the third manifest to identify a second reconstruction of the second copy-on-write segmented file, wherein the second reconstruction includes the object data from the second data range as well as information from the plurality of preexisting object segments.