Content based file chunking

• US 8,909,657 B2
• Filed: 09/30/2011
• Issued: 12/09/2014
• Est. Priority Date: 01/14/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Mensch, James L., Birse, Cameron Stuart, Misra, Ronnie G., Carlson, Eric Olaf, Giampaolo, Dominic B.
• Primary Examiner(s): LE, DEBBIE M
• Application Number:
  US13/250,504
• Publication Number:
  US 20120185448A1
• Time in Patent Office: 1,166 Days
• Field of Search: 707/755, 707/636, 707/695, 707/701
• US Class Current: 707/755
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 16/1834 implemented based on peer-to-peer networks, e.g. gnutella p2p communication protocols H04L67/104
• CPC Class Codes: G06F 16/1873 Versioning file systems, temporal file systems, e.g. file system supporting different historic versions of files
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: H04L 65/75 Media network packet handling
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for transferring electronic data. In general, one aspect of the subject matter described in this specification can be embodied in methods that include the actions of identifying a data item to be chunked; determining the type of the data item; determining whether the type of the data item is one of a specified one or more types; if it is determined that the type of the data item is not one of the specified one or more types, performing a first chunking of the data item; and if it is determined that the type of the data item is one of the specified one or more types, performing a second chunking of the data item that is based on the particular content portions of the data item.

Claims

1. A method performed by data processing apparatus, the method comprising:

identifying a data item to be chunked;

determining a data type of the data item;

determining that the data type of the data item is one of a specified one or more data types; and

in response to determining that the data type of the data item is one of the specified one or more data types;

identifying particular content portions that are included within the data item; and

performing a chunking of the data item that is based on the particular content portions that are included within the data item.

2. The method of claim 1, where performing the chunking includes performing the chunking of the data item based on a likelihood that at least some content portions of the particular content portions will be different than a version of the data item stored at a recipient device.

3. The method of claim 1, where performing the chunking includes:

introspecting the data item;

generating a data map of the particular content portions based on the introspection;

using the generated data map to define content based chunking boundaries; and

chunking the data item based on the data map.

4. The method of claim 3, where generating the data map includes identifying the different particular content portions within the data item.

5. The method of claim 1, where identifying the data type of the data item includes identifying a file extension associated with the data item.

6. The method of claim 4, where chunking the data item based on the data map includes separately chunking different particular content portions.

7. The method of claim 2, further comprising:

sending the chunks to a destination.

8. The method of claim 6, further comprising:

encrypting each chunk prior to sending.

9. The method of claim 1, further comprising:

in response to receiving a request for the data item;

sending a list of chunks, each chunk having a chunk identifier, to the requestor;

receiving a request for one or more chunks of the data item from the list of chunks, the requested one or more chunks being chunks that have changed from an earlier version of the data item; and

sending the requested one or more chunks.

10. A method performed by data processing apparatus, the method comprising:

receiving a data item to be chunked;

identifying a data type associated with the data item;

using the identified data type to introspect the data of the data item and build a content based map of the data item;

using the content based map to identify particular content portions included in the data item;

using the content based map to identify a separate chunking to be performed for the particular content portions in the data item; and

chunking the data item based on the particular content portions in the data item.

11. The method of claim 10, where building the content based map includes identifying different types of content within the data item.

12. The method of claim 10, where chunking the data item includes performing separate chunking operations on the particular content portions identified within the data item.

13. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

identifying a data item to be chunked;

determining that a data type the data item is one of a specified one or more data types;

in response to determining that the data type of the data item is one of the specified one or more data types;

identifying particular content portions included within the data item;

identifying a first content portion included within the data item that is likely to change relative to an earlier version of the data item and a second content portion included within the data item that is likely to be unchanged relative to an earlier version of the data item;

chunking the first content portion; and

chunking the second content portion separately from the first content portion.

14. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

identifying a first data item to be chunked;

determining that the first data item is of a first data type;

in response to determining that the first data item is of the first data type, chunking the first data item based on the size of the data item;

identifying a second data item to be chunked;

determining that the second data item is of a second data type; and

in response to determining that the second data item is of a second data type;

identifying particular content portions included in the second data item; and

chunking the second data item based on the likelihood that at least some portions of the particular content portions will be different than an earlier version of the second data item.

15. A system comprising:

one or more computing devices operable to perform operations comprising;

identifying a data item to be chunked;

determining a data type of the data item;

determining that the data type of the data item is one of a specified one or more data types; and

in response to determining that the data type of the data item is one of the specified one or more data types;

identifying particular content portions that are included within the data item; and

performing a chunking of the data item that is based on the particular content portions that are included within the data item.

16. The system of claim 15, where performing the chunking of the data item that is based on the particular content portions that are included within the data item includes:

identifying a first content portion that is likely to change between versions;

chunking the first content portion using a variable chunk size;

identifying a second content portion that is likely to be unchanged between versions; and

chunking the second content portion using a fixed chunk size.

17. The system of claim 15, where performing the chunking of the data item that is based on the particular content portions that are included within the data item includes:

introspecting the data item;

generating a data map of the particular content portions included within the data item based on the introspection; and

using the generated data map to define content based chunking boundaries to chunk the particular content portions.

18. The system of claim 17, where generating the data map includes identifying different types of content within the data item.

19. The system of claim 18, where identifying a type of data item includes identifying a file extension associated with the data item.

20. The system of claim 18, where chunking the data item based on the data map includes separately chunking different types of content.

21. The system of claim 16, further operable to perform operations comprising:

sending the chunks to a destination.

22. The system of claim 21, further operable to perform operations comprising:

encrypting each chunk prior to sending.

23. The system of claim 15, further operable to perform operations comprising:

in response to receiving a request for the data item;

sending a list of chunks, each chunk having a chunk identifier, to the requestor;

receiving a request for one or more chunks of the data item from the list of chunks, the requested one or more chunks being chunks that have changed from an earlier version of the data item; and

sending the requested one or more chunks.

24. A system comprising:

one or more computing devices operable to perform operations comprising;

receiving a data item to be chunked;

identifying a data type associated with the data item;

using the identified data type to introspect the data of the data item and build a content based map of the data item;

using the content based map to identify particular content portions included in the data item;

using the content based map to identify a separate chunking to be performed for the particular content portions in the data item; and

chunking the data item based on the particular content portions in the data item.

25. The system of claim 24, where building the content based map includes identifying different types of content within the data item.

26. The system of claim 24, where chunking the data item includes chunking based on a likelihood that at least some content portions of the particular content portions will be different than a version of the data item stored at a recipient device.