CONTENT BASED FILE CHUNKING

• US 20120185448A1
• Filed: 09/30/2011
• Published: 07/19/2012
• Est. Priority Date: 01/14/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Mensch, James L., Birse, Cameron Stuart, Misra, Ronnie G., Carlson, Eric Olaf
• Granted Patent: US 8,909,657 B2
• Time in Patent Office: Days
• US Class Current: 707/693
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 16/1834 implemented based on peer-to-peer networks, e.g. gnutella p2p communication protocols H04L67/104
• CPC Class Codes: G06F 16/1873 Versioning file systems, temporal file systems, e.g. file system supporting different historic versions of files
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: H04L 65/75 Media network packet handling
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for transferring electronic data. In general, one aspect of the subject matter described in this specification can be embodied in methods that include the actions of identifying a data item to be chunked; determining the type of the data item; determining whether the type of the data item is one of a specified one or more types; if it is determined that the type of the data item is not one of the specified one or more types, performing a first chunking of the data item; and if it is determined that the type of the data item is one of the specified one or more types, performing a second chunking of the data item that is based on the particular content portions of the data item.

Claims

1. A method performed by data processing apparatus, the method comprising:

identifying a data item to be chunked;

determining the type of the data item;

determining whether the type of the data item is one of a specified one or more types;

if it is determined that the type of the data item is not one of the specified one or more types, performing a first chunking of the data item; and

if it is determined that the type of the data item is one of the specified one or more types, performing a second chunking of the data item that is based on the particular content portions of the data item.

2. The method of claim 1, where performing the second chunking includes:

introspecting the data item;

generating a data map of content portions within the data item based on the introspection; and

chunking the data item based on the data map.

3. The method of claim 2, where performing the second chunking includes using the generated data map to define content based chunking boundaries.

4. The method of claim 2, where generating the data map includes identifying different types of content within the data item.

5. The method of claim 4, where identifying a type of data item includes identifying a file extension associated with the data item.

6. The method of claim 4, where chunking the data item based on the data map includes separately chunking different types of content.

7. The method of claim 2, further comprising:

sending the chunks to a destination.

8. The method of claim 6, further comprising:

encrypting each chunk prior to sending.

9. The method of claim 1, further comprising:

in response to receiving a request for the data item;

sending a list of chunks, each chunk having a chunk identifier, to the requestor;

receiving a request for one or more chunks of the data item from the list of chunks, the requested one or more chunks being chunks that have changed from an earlier version of the data item; and

sending the requested one or more chunks.

11. The method of claim 9, where building the content based map includes identifying different types of content within the data item.

12. The method of claim 9, where chunking the data item includes performing separate chunking operations on one or more types of content identified within the data item.

10. A method performed by data processing apparatus, the method comprising:

receiving a data item to be chunked;

identifying a type associated with the data item;

using the identified type to introspect the data of the data item and build a content based map of the data item;

using the content based map to identify a separate chunking to be performed for different content portions in the data item; and

chunking the data item based on the content in the data item.

13. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

identifying a data item to be chunked;

determining the type of the data item;

determining whether the type of the data item is one of a specified one or more types;

if it is determined that the type of the data item is not one of the specified one or more types, performing a first chunking of the data item; and

if it is determined that the type of the data item is one of the specified one or more types, performing a second chunking of the data item that is based on the particular content portions of the data item.

14. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

receiving a data item to be chunked;

identifying a type associated with the data item;

using the identified type to introspect the data of the data item and build a content based map of the data item;

using the content based map to identify a separate chunking to be performed for different content portions in the data item; and

chunking the data item based on the content in the data item.

15. A system comprising:

one or more computing devices operable to perform operations comprising;

identifying a data item to be chunked;

determining the type of the data item;

determining whether the type of the data item is one of a specified one or more types;

if it is determined that the type of the data item is not one of the specified one or more types, performing a first chunking of the data item; and

if it is determined that the type of the data item is one of the specified one or more types, performing a second chunking of the data item that is based on the particular content portions of the data item.

16. The system of claim 15, where performing the second chunking includes:

introspecting the data item;

generating a data map of content portions within the data item based on the introspection; and

chunking the data item based on the data map.

17. The system of claim 16, where performing the second chunking includes using the generated data map to define content based chunking boundaries.

18. The system of claim 16, where generating the data map includes identifying different types of content within the data item.

19. The system of claim 18, where identifying a type of data item includes identifying a file extension associated with the data item.

20. The system of claim 18, where chunking the data item based on the data map includes separately chunking different types of content.

21. The system of claim 16, further operable to perform operations comprising:

sending the chunks to a destination.

22. The system of claim 21, further operable to perform operations comprising:

encrypting each chunk prior to sending.

23. The system of claim 15, further operable to perform operations comprising:

in response to receiving a request for the data item;

sending a list of chunks, each chunk having a chunk identifier, to the requestor;

receiving a request for one or more chunks of the data item from the list of chunks, the requested one or more chunks being chunks that have changed from an earlier version of the data item; and

sending the requested one or more chunks.

24. A system comprising:

one or more computing devices operable to perform operations comprising;

receiving a data item to be chunked;

identifying a type associated with the data item;

using the identified type to introspect the data of the data item and build a content based map of the data item;

using the content based map to identify a separate chunking to be performed for different content portions in the data item; and

chunking the data item based on the content in the data item.

25. The system of claim 24, where building the content based map includes identifying different types of content within the data item.

26. The system of claim 24, where chunking the data item includes performing separate chunking operations on one or more types of content identified within the data item.