APPENDING TO FILES VIA SERVER-SIDE CHUNKING AND MANIFEST MANIPULATION

• US 20120233228A1
• Filed: 12/23/2011
• Published: 09/13/2012
• Est. Priority Date: 03/08/2011
• Status: Active Grant

• Current Assignee: Rackspace US Incorporated (Apollo Global Management, Inc.)
• Original Assignee: Rackspace US Incorporated (Apollo Global Management, Inc.)
• Inventors: Barton, Michael, Reese, Will, Dickinson, John A., Payne, Jay B., Thier, Charles B., Holt, Gregory
• Granted Patent: US 9,306,988 B2
• Time in Patent Office: Days
• US Class Current: 707/827
• CPC Class Codes: G06F 11/1004 to protect a block of data words, e.g. CRC or checksum G06F11/1076 takes precedence; security arrangements for protecting computers or computer systems against unauthorized activity G06F21/00
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/178 Techniques for file synchronisation in file systems
• CPC Class Codes: G06F 16/2219 Large Object storage; Management thereof
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9566 URL specific, e.g. using aliases, detecting broken or misspelled links
• CPC Class Codes: G06F 2201/85 Active fault masking without idle spares active fault masking without idle spare hardware where processing functionality is redundant G06F11/2035
• CPC Class Codes: G06F 3/0619 in relation to data integrity, e.g. data losses, bit errors
• CPC Class Codes: G06F 3/0626 Reducing size or complexity of storage systems
• CPC Class Codes: G06F 3/0644 Management of space entities, e.g. partitions, extents, pools
• CPC Class Codes: G06F 3/065 Replication mechanisms
• CPC Class Codes: G06F 3/067 Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
• CPC Class Codes: H04L 43/106 using time related information in packets, e.g. by adding timestamps
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04L 67/1095 : Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/1097 : for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/5651 : Reducing the amount or size of exchanged application data
• CPC Class Codes: H04L 69/04 : Protocols for data compression, e.g. ROHC

Abstract

Several different embodiments of a segmented object storage system are described. The object storage system divides files into a number of object segments, each segment corresponding to a portion of the object, and stores each segment individually in the cloud storage system. The system also generates and stores a manifest file describing the relationship of the various segments to the original data file. Requests to retrieve the segmented file are fulfilled by consulting the manifest file and using the information from the manifest to reconstitute the original data file from the constituent segments. Modifying, appending to, or truncating the object is accomplished by manipulating individual segments and the manifest file. In further embodiments, manipulation of the individual object segments and/or the manifest is used to implement copy-on-write, snapshotting, software transactional memory, and peer-to-peer transmission of the large file.

Claims

1. A method for modifying object data in a cloud storage system, the method comprising:

storing a segmented file as a plurality of individual stored object segments and a manifest, wherein each of the plurality of individually stored object segments correspond to a portion of the data in the segmented file, and wherein the manifest describes the relationship of the plurality of individually stored object segments to the segmented file sufficient to reconstitute the segmented file;

receiving a first data range and an instruction to include object data from the first data range in the segmented file;

storing information from the first data range in at least one new object segment;

modifying the manifest such that a reconstruction of the segmented file includes the object data from the first data range as well as information from the plurality of preexisting object segments.

2. The method of claim 1, wherein the instruction to include object data is an append instruction, and wherein modifying the manifest includes adding relational information to the manifest file that indicates the at least one new object segment follows the plurality of preexisting object segments.

3. The method of claim 1, wherein the instruction to include object data is an insert instruction, and wherein modifying the manifest includes adding relational information to the manifest file that indicates the at least one new object segment is between two of the plurality of preexisting object segments.

4. The method of claim 1, wherein the instruction to include object data is a modify instruction, and wherein the modifying the manifest includes adding relational information to the manifest file that indicates the at least one new object segment is to be included in place of at least one of the plurality of preexisting object segments.

5. The method of claim 1, wherein the method further includes the steps of determining that at least one of the plurality of preexisting object segments should not be included in the segmented file;

and modifying the manifest such that a reconstruction of the segmented does not include information from the at least one non-included preexisting object segments.

6. The method of claim 5, wherein the method further includes the step of, after determining that at least one of the plurality of preexisting object segments should not be included in the segmented file and modifying the manifest such that a reconstruction of the segmented does not include information from the at least one non-included preexisting object segments, deleting at least one non-included preexisting object segments.

7. The method of claim 5, wherein the step of determining that at least one of the plurality of preexisting object segments should not be included in the segmented file further includes the step of identifying at least one preexisting object segment older than a threshold time.

8. The method of claim 5, wherein the step of determining that at least one of the plurality of preexisting object segments should not be included in the segmented file further includes the step of evaluating whether including the object data from the first data range in the segmented file will cause the segmented file to become larger than a threshold size.

9. The method of claim 5, wherein the step of determining that at least one of the plurality of preexisting object segments should not be included in the segmented file further includes the step of identifying a window of interest in the segmented file, the window of interest corresponding to one or more individually stored object segments;

and not including data outside the window of interest.

10. The method of claim 1, wherein the segmented file is a structured data file, and wherein each of the plurality of individual stored object segments is a structured data file including a portion of the information represented by the segmented data file.

11. The method of claim 10, wherein reconstituting the segmented file includes performing a reduce operation to merge information from the plurality of individual stored object segments into a single result.

12. The method of claim 10, wherein the segmented file can be partially reconstituted to give a query result including less than the entire contents of the segmented file, wherein the query result is created by performing a reduce operation to merge information from the plurality of individual stored object segments into a single result.

13. A method for implementing localized changes to a data object in a cloud storage system, the method comprising:

storing a segmented file as a plurality of individual stored object segments and a first manifest, wherein each of the plurality of individually stored object segments correspond to a portion of the data in the segmented file, and wherein the first manifest describes the relationship of the plurality of individually stored object segments to the segmented file sufficient to reconstitute the segmented file;

receiving a request for write access to the segmented file;

creating a copy-on-write segmented file responsive to the request for write access to the segmented file, wherein creating the copy-on-write segmented file includes creating a second manifest file wherein the second manifest initially describes the relationship of the plurality of individually stored object segments to the segmented file in a manner identical to the first manifest;

receiving a first data range and an instruction to include object data from the first data range in the copy-on-write segmented file;

storing information from the first data range in at least one new object segment;

modifying the second manifest such that a reconstruction of the copy-on-write segmented file includes the object data from the first data range as well as information from the plurality of preexisting object segments.

14. The method of claim 13, wherein the method further includes the steps of evaluating whether there were any conflicting changes to the first manifest between the time of the creation of the second manifest, and selecting the first manifest if there were conflicting changes, and selecting the second manifest if there were no conflicting changes.

15. The method of claim 13, wherein the method further includes the steps of evaluating whether there were any conflicting changes to the first manifest between the time of the creation of the second manifest, and selecting the first manifest if there were conflicting changes, and merging the first and second manifests to create a third merged manifest if there were no conflicting changes.

16. The method of claim 13, wherein the method further includes the steps of receiving a second request for write access to the segmented file;

creating a second copy-on-write segmented file responsive to the second request to interact with the segmented file, including creating a third manifest;

receiving a second data range and an instruction to include object data from the second data range in the second copy-on-write segmented file;

storing information from the second data range in at least a second new object segment; and

modifying the third manifest such that a reconstruction of the second copy-on-write segmented file includes the object data from the second data range as well as information from the plurality of preexisting object segments.

17. A network file system, the system including:

a storage management server, the storage management server further comprising a processor, a network interface and a filesystem export module;

at least one computer-readable storage coupled to the storage management server;

wherein the computer-readable storage includes a plurality of individual stored object segments and a first manifest, wherein each of the plurality of individually stored object segments correspond to a portion of the data in the filesystem, and wherein the first manifest describes the relationship of the plurality of individually stored object segments to the filesystem sufficient for the filesystem export module to present a filesystem interface in response to communications received over the network interface.

18. The system of claim 17, wherein the system further comprises a file writer module and a snapshot module, wherein the snapshot module is operable to create a second manifest responsive to a request for write access to the filesystem, and wherein the file writer is operable to store at least one new individually stored object segments containing the differential changes caused by the request for write access.

19. The system of claim 18, wherein the system is operable to present a filesystem interface using either the first manifest or the second manifest.

20. The system of claim 17, wherein the plurality of individual stored object segments correspond to one or more blocks from a block device.

21. The system of claim 17, wherein the plurality of individual stored object segments correspond to one or more inodes from a filesystem.

22. The system of claim 17, wherein the plurality of individual stored object segments correspond to one or more files from a filesystem.