Cloud library de-duplication

• US 9,449,012 B2
• Filed: 05/30/2014
• Issued: 09/20/2016
• Est. Priority Date: 05/30/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Jalon, Julien J., Woods, Nicholas J., Wong, Raymond Wai-Keung
• Primary Examiner(s): Le, Hung
• Application Number:
  US14/292,231
• Publication Number:
  US 20150347444A1
• Time in Patent Office: 844 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1748 De-duplication implemented within the file system, e.g. based on file segments de-duplication techniques in storage systems for the management of data blocks G06F3/0641
• CPC Class Codes: G06F 16/489 using time information

Abstract

Disclosed herein are systems, methods, and non-transitory computer-readable storage media identifying duplicate media items that occur during a batch upload process from a client device to the cloud media library and performing media de-duplication and re-mapping of duplicate media items.

Claims

1. A computer-implemented method of identifying duplicate media items in a cloud media library comprising:

receiving a media file from a client device;

inspecting one or more data objects in the media file;

identifying, in a data object, a reference to a master asset fingerprint and a secondary fingerprint;

determining that a master asset described by the master asset fingerprint is stored by the cloud media library;

gathering one or more additional data objects in the cloud media library describing other media files stored by the cloud media library and including a reference to the master asset fingerprint of the received media file;

determining that the secondary fingerprint in the data object associated with the media file matches a secondary fingerprint of an additional data object for an additional media file stored by the cloud media library;

selecting the media file received from the client device or the additional media file based on the secondary fingerprint of the media file received and the secondary fingerprint of the additional media file; and

performing a de-duplication action.

2. The computer-implemented method of claim 1, wherein each secondary fingerprint describes one or more adjustments that have been made to the master asset.

3. The computer-implemented method of claim 2, wherein determining that the secondary fingerprint in the data object associated with the media file matches the secondary fingerprint of the additional data object for the other media file stored by the cloud media library, further comprises:

inspecting the one or more adjustments described in the secondary fingerprints; and

determining that the adjustments described in the secondary fingerprints fall within a predefined group of adjustments considered acceptably different for a purpose of de-duplication.

4. The computer-implemented method of claim 2, wherein selecting the media file received from the client device or the additional media file further comprises:

examining the secondary fingerprint of the received media file for a presence of adjustments made to the received media file;

examining a time stamp described in a data object of the received media file to determine when the received media file was last modified;

examining the secondary fingerprint of the additional media file for a presence of adjustments made to the additional media file;

examining a time stamp described in a data object of the additional media file to determine when the additional media file was last modified; and

selecting the media file received from the client device or the additional media file based on a de-duplication policy that factors the presence and time stamps of adjustments made to the media file received from the client device and the additional media file.

5. The computer-implemented method of claim 1, wherein each of the media file received from the client device and the additional media file is associated with a container data object describing a relationship with another media file in the cloud media library.

6. The computer-implemented method of claim 5, wherein performing a de-duplication action further comprises:

re-mapping the container data object of the non-selected media file to describe a relationship with the selected media file.

7. The computer-implemented method of claim 1, wherein the cloud media library stores media files in an external content repository.

8. A system for identifying duplicate media items in a cloud media library comprising:

a content ingestion module configured to receive a media file from a client device, inspect one or more data objects in the media file, and identify, in a data object, a reference to a master asset fingerprint and a secondary fingerprint;

a de-duplication module configured to determine that a master asset described by the master asset fingerprint is stored by the cloud media library, gather one or more additional data objects in the cloud media library describing other media files stored by the cloud media library and including a reference to the master asset fingerprint of the received media file, determine that the secondary fingerprint in the data object associated with the media file matches a secondary fingerprint of an additional data object for an additional media file stored by the cloud media library, and select the media file received from the client device or the additional media file based on the secondary fingerprint of the media file received and the secondary fingerprint of the additional media file; and

a re-mapping module configured to perform a de-duplication action.

9. The system of claim 8, wherein each secondary fingerprint describes one or more adjustments that have been made to the master asset.

10. The system of claim 9, wherein the de-duplication module is configured to inspect the one or more adjustments described in the secondary fingerprints and determine that the adjustments described in the secondary fingerprints fall within a predefined group of adjustments considered acceptably different for a purpose of de-duplication.

11. The system of claim 9, the de-duplication module is further configured to examine the secondary fingerprint of the received media file for a presence of adjustments made to the received media file, examine a time stamp described in a data object of the received media file to determine when the received media file was last modified, examine the secondary fingerprint of the additional media file for a presence of adjustments made to the additional media file, examine a time stamp described in a data object of the additional media file to determine when the additional media file was last modified, and select the media file received from the client device or the additional media file based on a de-duplication policy that factors the presence and time stamps of adjustments made to the media file received from the client device and the additional media file.

12. The system of claim 8, wherein each of the media file received from the client device and the additional media file is associated with a container data object describing a relationship with another media file in the cloud media library.

13. The system of claim 12, wherein the re-mapping module performs a de-duplication action that involves re-mapping the container data object of the non-selected media file to describe a relationship with the selected media file.

14. The system of claim 8, wherein the cloud media library stores media files in an external content repository.

15. A non-transitory computer-readable storage medium comprising:

a medium configured to store computer-readable instructions thereon; and

the computer-readable instructions that, when executed by a processing device cause the processing device to perform a method of identifying duplicate media items in a cloud media library comprising;

receiving a media file from a client device;

inspecting one or more data objects in the media file;

identifying, in a data object, a reference to a master asset fingerprint and a secondary fingerprint;

determining that a master asset described by the master asset fingerprint is stored by the cloud media library;

gathering one or more additional data objects in the cloud media library describing other media files stored by the cloud media library and including a reference to the master asset fingerprint of the received media file;

determining that the secondary fingerprint in the data object associated with the media file matches a secondary fingerprint of an additional data object for an additional media file stored by the cloud media library;

selecting the media file received from the client device or the additional media file based on the secondary fingerprint of the media file received and the secondary fingerprint of the additional media file; and

performing a de-duplication action.

16. The non-transitory computer-readable storage medium of claim 15, wherein each secondary fingerprint describes one or more adjustments that have been made to the master asset.

17. The non-transitory computer-readable storage medium of claim 16, wherein determining that the secondary fingerprint in the data object associated with the media file matches the secondary fingerprint of the additional data object for the other media file stored by the cloud media library, further comprises:

inspecting the one or more adjustments described in the secondary fingerprints; and

determining that the adjustments described in the secondary fingerprints fall within a predefined group of adjustments considered acceptably different for a purpose of de-duplication.

18. The non-transitory computer-readable storage medium of claim 16, wherein selecting the media file received from the client device or the additional media file further comprises:

examining the secondary fingerprint of the received media file for a presence of adjustments made to the received media file;

examining a time stamp described in a data object of the received media file to determine when the received media file was last modified;

examining the secondary fingerprint of the additional media file for a presence of adjustments made to the additional media file;

examining a time stamp described in a data object of the additional media file to determine when the additional media file was last modified; and

selecting the media file received from the client device or the additional media file based on a de-duplication policy that factors the presence and time stamps of adjustments made to the media file received from the client device and the additional media file.

19. The non-transitory computer-readable storage medium of claim 15, wherein each of the media file received from the client device and the additional media file is associated with a container data object describing a relationship with another media file in the cloud media library.

20. The non-transitory computer-readable storage medium of claim 19, wherein performing a de-duplication action further comprises:

re-mapping the container data object of the non-selected media file to describe a relationship with the selected media file.