Content based file chunking

• US 9,305,008 B2
• Filed: 12/08/2014
• Issued: 04/05/2016
• Est. Priority Date: 01/14/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Mensch, James L., Birse, Cameron Stuart, Misra, Ronnie G., Carlson, Eric Olaf, Giampaolo, Dominic B.
• Primary Examiner(s): Le, Debbie
• Application Number:
  US14/563,967
• Publication Number:
  US 20150095385A1
• Time in Patent Office: 484 Days
• Field of Search: 707/755, 707/756, 707/793, 707/701
• US Class Current: 1/1
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/1752 based on file chunks
• CPC Class Codes: G06F 16/183 Provision of network file services by network file servers, e.g. by using NFS, CIFS network file access protocols H04L67/1097
• CPC Class Codes: G06F 16/1834 implemented based on peer-to-peer networks, e.g. gnutella p2p communication protocols H04L67/104
• CPC Class Codes: G06F 16/1873 Versioning file systems, temporal file systems, e.g. file system supporting different historic versions of files
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: H04L 65/75 Media network packet handling
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for transferring electronic data. In general, one aspect of the subject matter described in this specification can be embodied in methods that include the actions of identifying a data item to be chunked; determining the type of the data item; determining whether the type of the data item is one of a specified one or more types; if it is determined that the type of the data item is not one of the specified one or more types, performing a first chunking of the data item; and if it is determined that the type of the data item is one of the specified one or more types, performing a second chunking of the data item that is based on the particular content portions of the data item.

Claims

1. A method, comprising:

identifying, by a data processing apparatus, a data item to be chunked;

determining, by the data processing apparatus, a data item type of the data item;

determining, by the data processing apparatus, that the data item type is one of a specified one or more data types;

in response to determining that the data item type is one of the specified one or more data types;

identifying, by the data processing apparatus, a first content portion and a second content portion included within the data item;

identifying, by the data processing apparatus, a first type of chunking process to be performed on the first content portion;

identifying, by the data processing apparatus, a second type of chunking process to be performed on the second content portion, where the first type of chunking process is different than the second type of chunking process;

performing the first type of chunking process on the first content portion; and

performing the second type of chunking process on the second content portion.

2. The method of claim 1, wherein determining the data item type includes identifying one of a file extension, header or data pattern indicative of the data item type.

3. The method of claim 1, wherein the first type of chunking process is a fixed length chunking and the second type of chunking process is a variable length chunking.

4. The method of claim 1, wherein identifying the first content portion and the second content portion comprises:

introspecting the data item to identify the first and the second content portions; and

generating a map based on the introspection that indicates that the first or second content portions of the data item are likely or not likely to change between versions of the data item.

5. The method of claim 4, wherein the second content portion is embedded in the first content portion, the introspecting identifies a bound of the second content portion within the first content portion, and the second type of chunking process is applied to the second content portion according to the identified bound.

6. The method of claim 1, wherein the data item is an image file, the first content portion is image data and the second content portion is image tag data.

7. The method of claim 1, wherein the data item is an audio file, the first content portion is audio data and the second content portion is audio tag data.

8. A data processing apparatus, comprising:

one or more processors;

a computer-readable medium coupled to the one or more processors and configured to store instructions, which, when executed by the one or more processors, causes the one or more processors to perform operations comprising;

identifying a data item to be chunked;

determining a data item type of the data item;

determining that the data item type is one of a specified one or more data types;

in response to determining that the data item type is one of the specified one or more data types;

identifying a first content portion and a second content portion included within the data item;

identifying a first type of chunking process to be performed on the first content portion;

identifying a second type of chunking process to be performed on the second content portion, where the first type of chunking process is different than the second type of chunking process;

performing the first type of chunking process on the first content portion to produce first chunks; and

performing the second type of chunking process on the second content portion to produce second chunks.

9. The data processing apparatus of claim 8, wherein determining the data item type includes identifying one of a file extension, header or data pattern indicative of the data item type.

10. The data processing apparatus of claim 8, wherein the first type of chunking process is a fixed length chunking and the second type of chunking process is a variable length chunking.

11. The data processing apparatus of claim 8, wherein identifying the first and the second content portions further comprises:

introspecting the data item to identify the first and the second content portions; and

generating a map based on the introspection that indicates a first likelihood and a second likelihood that the first and the second content portions, respectively, are likely to change between versions of the data item.

12. The data processing apparatus of claim 11, wherein the second content portion is embedded in the first content portion, the introspecting identifies a bound of the second content portion within the first content portion, and the second type of chunking process is applied to the second content portion according to the identified bound.

13. The data processing apparatus of claim 8, wherein the data item is an image file, the first content portion is image data and the second content portion is image tag data.

14. The data processing apparatus of claim 8, wherein the data item is an audio file, the first content portion is audio data and the second content portion is audio tag data.

15. The data processing apparatus of claim 8, wherein the one or more processors perform operations comprising:

generating a list of chunks resulting from the first or the second type of chunking process; and

sending the list of chunks to a destination.

16. The data processing apparatus of claim 15, wherein the one or more processors perform the operations comprising:

identifying encrypted chunks that were previously stored at the destination; and

sending to the destination only an encrypted chunk that was not previously stored at the destination.

17. The data processing apparatus of claim 15, wherein the one or more processors perform the operations comprising:

generating chunk identifiers for the first chunks and for the second chunks; and

sending to the destination only those chunks of the first chunks and the second chunks that were not previously stored at the destination or that have undergone a change since a last transmission or synchronization of the data item, where the change is indicated by a chunk identifier associated with a chunk which has changed.

18. A method of splitting a first version of an audio file, the method comprising:

by a first computing device;

receiving, from a second computing device, a request for the audio file;

identifying a first portion of the audio file;

identifying a second portion of the audio file;

splitting the audio file into a first chunk corresponding to the first portion and a second chunk corresponding to the second portion; and

sending the first and the second chunks to the second computing device, wherein the first portion is likely to change in a second version of the audio file and the second portion is unlikely to change in the second version of the audio file.

19. The method of claim 18, wherein the first portion comprises metadata.

20. The method of claim 18, wherein the second portion comprises audio data.