Methods for identifying audio or video content

• US 8,688,999 B2
• Filed: 07/09/2013
• Issued: 04/01/2014
• Est. Priority Date: 12/23/2005
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Conwell, William Y.
• Primary Examiner(s): Barron, Jr., Gilberto
• Assistant Examiner(s): Le, David
• Application Number:
  US13/937,995
• Publication Number:
  US 20130297942A1
• Time in Patent Office: 266 Days
• Field of Search: None
• US Class Current: 713/176
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/489 using time information
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06N 5/048 Fuzzy inferencing
• CPC Class Codes: G06Q 10/00 Administration; Management
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06Q 50/18 Legal services
• CPC Class Codes: G06Q 50/265 Personal security, identity or safety
• CPC Class Codes: H04L 63/08 for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 9/3247 involving digital signatures

Abstract

Methods and systems for identifying audio and video entertainment content are provided. Certain shortcomings of fingerprint-based content identification can be redressed through use of human-reviewers in a first party'"'"'s social networking site environment. Access to a copy of a possible fingerprint-based match of user uploaded content is provided to a second party via a queue of an online interface provided by the first party. In response to providing access to the copy of the user uploaded content, manual human assessment data is obtained from the second party through the online interface. The manual human assessment data indicating a result of a comparison of the copy of the user uploaded content and data associated with reference content by one or more human reviewer(s). The manual human assessment data is used in concluding that the reference content corresponds with the user uploaded content.

Claims

1. A method comprising:

i. receiving user uploaded video or audio content, the video or audio content having been uploaded by a user to a first party'"'"'s social networking site for distribution from the first party'"'"'s social networking site to the public, in which the user uploaded video or audio content has been altered relative to a corresponding version of the user uploaded video or audio content;

ii. processing the user uploaded video or audio content;

iii. generating fingerprints from the processed, user uploaded video or audio content;

iv. identifying reference content, from a universe of reference content, that is a possible match of the user uploaded video or audio content, said identifying being based at least on a generated fingerprint and being performed by a computer system configured to perform such act;

v. providing a computer controlled online interface associated with the first party'"'"'s social networking site through which a result of manual human review of a queue generated by the first party'"'"'s social networking site is obtained, the queue for review by a second party, different than the first party;

vi. providing access to a copy of the user uploaded video or audio content via the queue of the computer controlled online interface;

vii. in response to providing access to the copy of the user uploaded video or audio content via the queue of the computer controlled online interface, obtaining manual human assessment data from the second party through the computer controlled online interface associated with the first party'"'"'s social networking site, the manual human assessment data indicating a result of a comparison of the copy of the user uploaded video or audio content and data associated with the reference content by one or more human reviewer(s);

viii. using the manual human assessment data in concluding that the reference content corresponds with the user uploaded video or audio content; and

ix. controlling distribution of the user uploaded video or audio content from the first party'"'"'s social networking site to the public based on the concluding.

2. The method of claim 1 in which the generated fingerprints include audio fingerprints or video fingerprints, further comprising determining scoring that indicates a relative correspondence between the generated fingerprints and fingerprints of the reference content, in which manual human review is only solicited through the computer controlled online interface when the scoring indicates an unreliability.

3. The method of claim 1 in which the computer controlled online interface comprises a graphical interface to facilitate access to the queue by the second party.

4. The method of claim 1 in which said providing access also provides access to a copy of the reference content via a link in the queue of the computer controlled online interface, and the manual human assessment data indicates a result of an audible or visual comparison of the copy of the user uploaded video or audio content and the copy of the reference content by one or more human reviewer(s).

5. The method of claim 1 in which said processing comprises transforming the user uploaded video or audio content, and said generating utilizes hashing, in which the possible match is determined by a distance function comparing plural hashes of the user uploaded video or audio content and plural hashes of the reference content.

6. The method of claim 1 in which the data associated with the reference content comprises the reference content, and providing access to the copy of the user uploaded video or audio content is provided via a link.

7. The method of claim 1 in which said identifying reference content identifies several possible matches.

17. The method of claim 1 in which the manual human assessment data comprises subjective human judgment of one or more human reviewer(s) pre-qualified as knowledgeable about the audio or video content.

18. The method of claim 1 in which the manual human assessment data is automatically stored once obtained through the computer controlled online interface.

19. The method of claim 1 in which said concluding considers both the manual human assessment data and a result of said identifying reference content, but gives greater weight to the manual human assessment data relative to the result of said identifying reference content.

22. The method of claim 1 further comprising determining a confidence level that indicates a relative correspondence between data representing perceptually relevant features of the user uploaded video or audio content and data representing perceptually relevant features of the reference content, the confidence level indicating an unreliability in the possible match between the reference content and the user uploaded video or audio content, in which manual human review is only solicited through the computer controlled online interface when the confidence level indicates unreliability.

23. The method of claim 1 in which a programmed computer causes said act of identifying to generate a score associated with the possible match, in which, when the score comprises a first score above a first measure of confidence, then the programmed computer operates in a first mode in which the manual human review is not requested and distribution of the user uploaded video or audio content is controlled without the programmed computer executing acts v-ix, and when the score comprises a second score indicating a lower confidence than the first measure of confidence in the possible match, then the programmed computer operates in a second mode in which the programmed computer executes acts v-ix.

24. The method of claim 22 in which the confidence level comprises a metric based on a distance function.

25. The method of claim 1 in which said identifying reference content utilizes a Viterbi function.

8. One or more computer servers associated with a first party'"'"'s social networking site, comprising:

an input to receive user uploaded video or audio content, the video or audio content having been uploaded by a user to a first party'"'"'s social networking site for distribution from the first party'"'"'s social networking site to the public, in which the user uploaded video or audio content has been altered relative to a corresponding version of the user uploaded video or audio content;

one or more processors programmed for;

i. processing the user uploaded video or audio content;

ii. generating fingerprints from the processed, user uploaded video or audio content;

iii. identifying reference content, from a universe of reference content, that is a possible match of the user uploaded video or audio content, said identifying being based at least on a generated fingerprint;

iv. providing a computer controlled online interface associated with the first party'"'"'s social networking site through which a result of manual human review of a queue generated by the first party'"'"'s social networking site is obtained, the queue for review by a second party, different than the first party;

v. providing access to a copy of the user uploaded video or audio content via the queue of the computer controlled online interface;

vi. in response to providing access to the copy of the user uploaded video or audio content via the queue of the computer controlled online interface, obtaining manual human assessment data from the second party through the computer controlled online interface associated with the first party'"'"'s social networking site, the manual human assessment data indicating a result of a comparison of the copy of the user uploaded video or audio content and data associated with the reference content by one or more human reviewer(s);

vii. using the manual human assessment data in concluding that the reference content corresponds with the user uploaded video or audio content; and

viii. controlling distribution of the user uploaded video or audio content from the first party'"'"'s social networking site to the public based on the concluding.

9. The one or more computer servers of claim 8 in which the fingerprint includes an audio fingerprint or a video fingerprint, in which said one or more processors are programmed for determining scoring that indicates a relative correspondence between the generated fingerprints and fingerprints of the reference content, in which manual human review is only solicited through the computer controlled online interface when the scoring indicates an unreliability.

10. The one or more computer servers of claim 8 in which said one or more processors control the online interface to provide a graphical interface to allow access by the second party for reviewing the queue.

11. The one or more computer servers of claim 8 in which said providing access also provides access to a copy of the reference content via a link in the queue of the online interface, and the manual human assessment data indicates a result of an audible or visual comparison of the copy of the user uploaded video or audio content and the copy of the reference content by one or more human reviewer(s).

12. The one or more computer servers of claim 8 in which said processing comprises transforming the user uploaded video or audio content, and said generating utilizes hashing, in which the identifying determines the possible match with a distance function comparing plural hashes of the user uploaded video or audio content and plural hashes of the reference content.

13. The one or more computer servers of claim 8 in which the data associated with the reference content comprises the reference content, and providing access to the copy of the user uploaded video or audio content is provided via a link.

20. The one or more computer servers of claim 8 further comprising storage for automatically storing the manual human assessment data once obtained through the computer controlled online interface.

21. The one or more computer servers of claim 8 in which the concluding considers both the manual human assessment data and a result of the identifying reference content, but gives greater weight to the manual human assessment data relative to the result of the identifying reference content.

26. The one or more computer servers of claim 8 in which said one or more processors are programmed for determining a confidence level that indicates a relative correspondence between data representing perceptually relevant features of the user uploaded video or audio content and data representing perceptually relevant features of the reference content, the confidence level indicating an unreliability in the possible match, in which manual human review is only solicited through the computer controlled online interface when the confidence level indicates unreliability.

27. The one or more computer servers of claim 8 in which the identifying generates a score associated with the possible match, and in which said one or more computer servers are configured to operate in a first mode and second mode, in which, when the score comprises a first score above a first measure of confidence, then the one or more computer servers operate in the first mode and the manual human review is not requested and distribution of the user uploaded video or audio content is controlled without said one or more processors executing said iv-viii, and when the score comprises a second score indicating a lower confidence than the first measure of confidence in the possible match, then said one or more computer servers operate in the second mode and said one or more processors execute said iv-viii.

28. The one or more computer servers of claim 26 in which the confidence level comprises a metric based on a distance function.

29. The one or more computer servers of claim 8 in which the identifying reference content utilizes a Viterbi function.

14. A non-transitory computer readable medium comprising instructions stored thereon to cause one or more processors to perform the following:

processing user uploaded video or audio content, the video or audio content having been uploaded by a user to a first party'"'"'s social networking site for distribution from the first party'"'"'s social networking site to the public, in which the user uploaded video or audio content has been altered relative to a corresponding version of the user uploaded video or audio content;

generating fingerprints from the processed, user uploaded video or audio content;

identifying reference content, from a universe of reference content, that is a possible match of the user uploaded video or audio content, said identifying being based at least on a generated fingerprint;

providing an computer controlled online interface associated with the first party'"'"'s social networking site through which a result of manual human review of a queue generated by the first party'"'"'s social networking site is obtained, the queue for review by a second party, different than the first party;

providing access to a copy of the user uploaded video or audio content via the queue of the computer controlled online interface;

in response to providing access to the copy of the user uploaded video or audio content via the queue of the computer controlled online interface, obtaining manual human assessment data from the second party through the computer controlled online interface associated with the first party'"'"'s social networking site, the manual human assessment data indicating a result of a comparison of the copy of the user uploaded video or audio content and data associated with the reference content by one or more human reviewer(s);

using the manual human assessment data in concluding that the reference content corresponds with the user uploaded video or audio content; and

controlling distribution of the user uploaded video or audio content from the first party'"'"'s social networking site to the public based on the concluding.

15. The non-transitory computer readable medium of claim 14 in which the processing comprises transforming the user uploaded video or audio content, and the generating utilizes hashing, in which the identifying determines the possible match with a distance function comparing the generated fingerprints and fingerprints of the reference content.

16. The non-transitory computer readable medium of claim 14 in which the providing access also provides access to a copy of the reference content via the queue of the computer controlled online interface, and the manual human assessment data indicates a result of an audible or visual comparison of the copy of the user uploaded video or audio content and the copy of the reference content by one or more human reviewer(s).

30. The non-transitory computer readable medium of claim 14 further comprising instructions to perform the following:

determining a confidence level that indicates a relative correspondence between data representing perceptually relevant features of the user uploaded video or audio content and data representing perceptually relevant features of the reference content, in which manual human review is only solicited through the computer controlled online interface when the confidence level indicates an unreliability, in which the confidence level comprises a metric based on a distance function.