METHOD AND APPARATUS FOR PROVIDING MULTIMEDIA CONTENT OPTIMIZATION

• US 20090089326A1
• Filed: 09/28/2007
• Published: 04/02/2009
• Est. Priority Date: 09/28/2007
• Status: Abandoned Application

• Current Assignee: Oath Inc. (Verizon Communications Inc.)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Balasubramanian, Srinivasan
• Application Number:
  US11/864,370
• Publication Number:
  US 20090089326A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking

Abstract

Methods, system and computer readable medium for detecting duplicate content in a pair of media files prior to publication on a webpage include generating fingerprints for the contents of each of the pair of media files. The fingerprints of one of the pair of media file are then compared with the fingerprints of another of the pair of media files to compute a similarity score. The similarity score is compared against an established threshold. If the similarity score exceeds the established threshold, it is determined that the two media files are substantial duplicate of one another.

Claims

1. A method for detecting duplicate content in a pair of media files prior to publication on a webpage, the method comprising:

generating fingerprint for the content of each of the pair of media files, the fingerprint defining a feature set for each media file;

comparing the fingerprint of the pair of media files to obtain a similarity score; and

declaring the media files as substantial duplicates when the similarity score exceeds an established threshold.

2. The method for detecting duplicate content in a pair of media files prior to publication on a webpage of claim 1, wherein the media files are normalized prior to generating the fingerprint for the content of the media files.

3. The method for detecting duplicate content in a pair of media files prior to publication on a webpage of claim 2, wherein normalizing media files includes one or more of converting text to lower case, eliminating whitespaces, eliminating special characters and eliminating stop words.

4. The method for detecting duplicate content in a pair of media files prior to publication on a webpage of claim 1, wherein defining the feature set further comprising:

establishing a sliding window size;

segmenting each of the pair of media files into a plurality of strings, the width of each segment of the media files determined by the established sliding window size;

incrementing the sliding window over the length of the media files by a pre-determined length at a time; and

computing hash value over each segment of the pair of the media files.

5. The method for detecting duplicate content in a pair of media files prior to publication on a webpage of claim 4, wherein the similarity score is obtained by comparing the hash values for each segment of one media file with the hash value for each segment of another media file.

6. The method for detecting duplicate content in a pair of media files prior to publication on a webpage of claim 5, wherein the similarity score is obtained by comparing only distinct fingerprints of each of the pair of media files.

7. The method for detecting duplicate content in a pair of media files prior to publication on a webpage of claim 4, wherein the pre-determined length for incrementing the sliding window is one byte.

8. The method for detecting duplicate content in a pair of media files prior to publication on a webpage of claim 1, wherein the established threshold for the similarity score to exceed is configurable.

9. The method for detecting duplicate content in a pair of media files prior to publication on a webpage of claim 1, further comprising generating fingerprint for the title of each of the media files and determining similarity score for the titles of the pair of media files.

10. The method for detecting duplicate content in a pair of media files prior to publication on a webpage of claim 1, wherein the media files include one or more of text files, graphic user interface files, audio files, video files, or any other multimedia content files.

11. The method for detecting duplicate content in a pair of media files prior to publication on a webpage of claim 1, wherein the content of the media files are defined by one or more metadata and wherein the fingerprints are generated for the pair of media files using the metadata.

12. A method for publishing media files on a webpage, comprising:

identifying media files to be published on the webpage;

computing similarity scores for the identified media files based on contents of the media files;

determining a containment percentage score for the identified media files, the containment percentage score is associated with the content of the media files;

ranking the media files based on pre-defined metric using the similarity scores and containment scores of the identified media files, the pre-defined metric including one or more established thresholds for comparing the similarity scores and containment scores of the media files to determine the ranking of the media files; and

publishing the appropriate content of the media files based on the ranking of the media files.

13. The method for publishing media files on a webpage of claim 12, wherein computing similarity scores further including:

defining a sliding window size based on the content of the media files;

segmenting the content of each of the media files into a plurality of sections, the width of each section defined by the sliding window size, the plurality of sections obtained by incrementing the sliding window by one byte across the length of each of the media file; and

computing hash value for each section of the media file, the hash value used in determining the similarity score of each of the media file.

14. The method for publishing media files on a webpage of claim 12, wherein computing containment percentage score further including:

determining size of each of a pair of the media files;

calculating containment percentage of the contents of the media files when the size of a first media files exceeds the size of a second media file by a pre-defined threshold.

15. The method for publishing media files on a webpage of claim 12, wherein the contents of the media files are normalized, the normalization including one or more of converting text characters within the media files to lower case, eliminating whitespaces, eliminating special characters, and eliminating stop words.

16. The method for publishing media files on a webpage of claim 12, wherein the contents of the media files are defined by one or more metadata.

17. A system for detecting duplicate content in a pair of multimedia files prior to publication on a webpage, the system comprising:

a backend server to receive the multimedia files from a plurality of content providers, the backend server receiving the multimedia files over a communication network;

a duplicate detection software module available to the backend server, the duplicate detection software module configured to compute a similarity score for the content of the media files received from the plurality of content providers; and

a publish server to publish the multimedia files over the internet as a webpage.

18. The system for detecting duplicate content in a pair of multimedia files prior to publication on a webpage of claim 17, further comprising a list generator available to the backend server, the list generator configured to generate headlines for the available multimedia files at the backend server.

19. A computer readable medium in which program instructions are stored, the program instructions when read by a server of a computing system, cause the server to perform a method for detecting duplicate content in a pair of media files prior to publication on a webpage, the method comprising:

generating fingerprint for the content of each of the pair of media files, the fingerprint defining a feature set for each media file;

comparing the fingerprint of the pair of media files to obtain a similarity score; and

declaring the media files as substantial duplicates when the similarity score exceeds an established threshold.

20. The computer readable medium of claim 19, further comprising program instructions to normalize each of the pair of the media files prior to generating the fingerprint for the content of each media file, wherein normalization includes one or more of converting text in each media file to lower case, eliminating stop words, eliminating whitespaces and eliminating special characters.

21. The computer readable medium of claim 19, further comprising generating fingerprint for the title of each of the pair of media files and computing the similarity score for the titles of the pair of media files by comparing the fingerprints for the title.