Methods for identifying audio or video content

• US 8,868,917 B2
• Filed: 06/04/2013
• Issued: 10/21/2014
• Est. Priority Date: 12/23/2005
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Conwell, William Y.
• Primary Examiner(s): McNally, Michael S
• Application Number:
  US13/909,834
• Publication Number:
  US 20140156691A1
• Time in Patent Office: 504 Days
• Field of Search: 713/176, 797/758
• US Class Current: 713/176
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/489 using time information
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06N 5/048 Fuzzy inferencing
• CPC Class Codes: G06Q 10/00 Administration; Management
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06Q 50/18 Legal services
• CPC Class Codes: G06Q 50/265 Personal security, identity or safety
• CPC Class Codes: H04L 63/08 for authentication of entities cryptographic mechanisms or cryptographic arrangements for entity authentication H04L9/32
• CPC Class Codes: H04L 9/3247 involving digital signatures

Abstract

The present disclosure relates to analyzing user uploaded visual or audio content. One method comprises: generating fingerprint data from user uploaded visual or audio content, the user uploaded visual or audio content having been uploaded by a user to a first party'"'"'s online video service for distribution from the first party'"'"'s online video service to the public; identifying plural items of reference content, from a universe of reference content, that are a possible match of the user uploaded visual or audio content, said identifying being based on the fingerprint data and being performed by a computer system configured to perform such act; assessing a score for each of the identified plural items of reference content, each score indicating a likely correspondence of the user uploaded visual or audio content with a respective item of reference content; for items in the plural items of reference content that fall within a predetermined score range, queuing for review by a second party, different than the first party, the user uploaded visual or audio content item and data associated with the items in the plural items of reference content that fall within a predetermined score range; receiving assessment data from the second party, the assessment data indicating a result of a subjective comparison of the user uploaded visual or audio content and the items in the plural items of reference content that fall within a predetermined score range; using the assessment data and a score assessed in said act of assessing in concluding that at least one item in the identified plural items of reference content corresponds with the user uploaded visual or audio content; and limiting distribution of the user uploaded visual or audio content from the first party'"'"'s online video service to the public based on the concluding. Of course other methods and combinations are disclosed as well.

Claims

1. A method comprising:

generating fingerprint data from user uploaded visual or audio content, the user uploaded visual or audio content having been uploaded by a user to a first party'"'"'s online video service for distribution from the first party'"'"'s online video service to the public;

identifying plural items of reference content, from a universe of reference content, that are a possible match of the user uploaded visual or audio content, said identifying being based on the fingerprint data and being performed by a computer system configured to perform such act;

assessing a score for each of the identified plural items of reference content, each score indicating a likely correspondence of the user uploaded visual or audio content with a respective item of reference content;

for items in the plural items of reference content that fall within a predetermined score range, queuing for review by a second party, different than the first party, the user uploaded visual or audio content item and data associated with the items in the plural items of reference content that fall within a predetermined score range;

receiving assessment data from the second party, the assessment data indicating a result of a subjective comparison of the user uploaded visual or audio content and the items in the plural items of reference content that fall within a predetermined score range;

using the assessment data and a score assessed in said act of assessing in concluding that at least one item in the identified plural items of reference content corresponds with the user uploaded visual or audio content; and

limiting distribution of the user uploaded visual or audio content from the first party'"'"'s online video service to the public based on the concluding.

2. The method of claim 1 in which the user uploaded visual or audio content has been intentionally altered to avoid fingerprint matching.

3. The method of claim 2 in which the user uploaded visual or audio content has been rotated.

4. The method of claim 1 further comprising updating the universe of reference content with the user uploaded visual or audio content.

5. The method of claim 1 in which said limiting distribution comprises interrupting delivery to a public user.

6. The method of claim 5 further comprising providing an explanatory message to the public user.

7. The method of claim 1 in which the first party provides a graphical interface to allow access by the second party for reviewing queued items.

8. The method of claim 1 in which said queuing includes providing a second party a link to the plural items of reference content.

9. The method of claim 1 in which said generating fingerprint data comprises comprising analyzing a transform domain representation of the user uploaded visual or audio content.

10. The method of claim 1 in which the data associated with the items in the plural items of reference content that fall within a predetermined score range comprises at least a portion of the plural items of reference content.

11. A computer system comprising:

one or more processors;

a non-transitory computer-readable readable medium storing executable code, the code when executed by the one or more processors causes the computer system to perform the following;

generating fingerprint data from user uploaded visual or audio content, the user uploaded visual or audio content having been uploaded by a user to a first party'"'"'s online video service for distribution from the first party'"'"'s online video service to the public;

identifying plural items of reference content, from a universe of reference content, that are a possible match of the user uploaded visual or audio content, said identifying being based on the fingerprint data and being performed by a computer system configured to perform such act;

assessing a score for each of the identified plural items of reference content, each score indicating a likely correspondence of the user uploaded visual or audio content with a respective item of reference content;

for items in the plural items of reference content that fall within a predetermined score range, queuing for review by a second party, different than the first party, the user uploaded visual or audio content item and data associated with the items in the plural items of reference content that fall within a predetermined score range;

receiving assessment data from the second party, the assessment data indicating a result of a subjective comparison of the user uploaded visual or audio content and the items in the plural items of reference content that fall within a predetermined score range;

using the assessment data and a score assessed in said assessing in concluding that at least one item in the identified plural items of reference content corresponds with the user uploaded visual or audio content; and

limiting distribution of the user uploaded visual or audio content from the first party'"'"'s online video service to the public based on the concluding.

12. The computer system of claim 11 in which the user uploaded visual or audio content has been intentionally altered to avoid fingerprint matching.

13. The computer system of claim 12 in which the user uploaded visual or audio content has been rotated.

14. The computer system of claim 11 further comprising code that when executed by the one or more processors causes the computer system to update the universe of reference content with the user uploaded visual or audio content.

15. The computer system of claim 11 in which the limiting distribution comprises interrupting delivery to a public user.

16. The computer system of claim 15 further comprising code that when executed by the one or more processors causes the computer system to provide an explanatory message to the public user.

17. The computer system of claim 11 further comprising code that when executed by the one or more processors causes the computer system to provide a graphical interface to allow access by the second party for reviewing queued items.

18. The computer system of claim 11 in which said queuing includes providing the second party a link to the plural items of reference content.

19. The computer system of claim 11 that further comprises code that when executed by the one or more processors causes analyzing a transform domain representation of the user uploaded visual or audio content.

20. The computer system of claim 11 in which the data associated with the items in the plural items of reference content that fall within a predetermined score range comprises at least a portion of the plural items of reference content.

21. A computer system comprising:

an input to receive user uploaded audio-visual content, the user uploaded audio-visual content having been uploaded by a user to a first party'"'"'s online video service for distribution from the first party'"'"'s online video service to the public; and

one or more processors configured for;

generating fingerprint data from data representing audio or video data associated with the user uploaded audio-visual content;

identifying at least one item of reference content, from a universe of reference content, that is a possible match of the user uploaded audio-visual content, the identifying being based on the fingerprint data;

determining a correspondence between the user uploaded audio-visual content and the at least one item of reference content;

providing a computer interface for access by a second party, the second party being different than the first party;

upon a condition in which the correspondence falls within a predetermined range;

queuing a comparison for review by the second party via the computer interface, the computer interface providing access to the user uploaded audio-visual content and access to the at least one item of reference content,

receiving assessment data from the second party, through the computer interface,

the assessment data being associated with a result of the comparison of the user uploaded audio-visual content and the at least one item of reference content, and

controlling distribution of the user uploaded audio-visual content on the first party'"'"'s online video service based at least in part on the assessment data, in which the user uploaded audio-visual content remains publically accessible pending receipt of the assessment data; and

upon a condition in which the correspondence falls above the predetermined range;

controlling distribution of the user uploaded audio-visual content from the first party'"'"'s online video service to the public per a rights holder'"'"'s request.

22. The computer system of claim 21 in which the user uploaded audio-visual content has been altered to avoid fingerprint matching prior to uploading.

23. The computer system of claim 21 in which the user uploaded audio-visual content has been rotated prior to uploading.

24. The computer system of claim 21 in which the one or more processors are configured to cause the computer system to update the universe of reference content with the user uploaded audio-visual content.

25. The computer system of claim 21 in which the controlling distribution of the user uploaded audio-visual content on the first party'"'"'s online video service based at least in part on the assessment data comprises interrupting delivery to a public user.

26. The computer system of claim 25 in which the one or more processors are configured to cause the computer system to provide an explanatory message to the public user in connection with the controlling distribution.

27. The computer system of claim 21 in which the computer interface provides a graphical interface to allow access by the second party for reviewing queued items.

28. The computer system of claim 21 in which the computer interface provides access to the user uploaded audio-visual content and access to the at least one item of reference content through links to content.

29. The computer system of claim 21 in which the one or more processors are configured to analyze a transform domain representation of the user uploaded audio-visual content, and generate the fingerprint data from the transform domain representation.

30. The computer system of claim 21 in which the one or more processors are configured to cause the computer system to associate the fingerprint data with the universe of reference content.

31. The computer system of claim 21 in which the possible match comprises a match that is less than conclusive.

32. The computer system of claim 21 in which the predetermined range comprises a range below a conclusive match and above a no-match condition.

33. One or more computer servers associated with a first party'"'"'s online video distribution site, comprising:

an input to receive user uploaded audio-visual content, the user upload audio-visual content having been uploaded by a user to the first party'"'"'s online video distribution site for distribution from the first party'"'"'s online video distribution site to the public, in which the user uploaded audio-visual content has been altered relative to a corresponding version of the user uploaded audio-visual content;

one or more processors programmed for;

i. generating fingerprint data from data representing audio or video data associated with the user uploaded audio-visual content;

ii. identifying reference content, from a universe of reference content, that is a possible match of the user uploaded audio-visual content, the identifying being based at least on the generated fingerprint data;

iii. determining a relative correspondence between the generated fingerprint data and fingerprint data associated with the reference content;

iv. providing a computer controlled interface associated with the first party'"'"'s online video distribution site through which a result of manual human review of a queue generated by the first party'"'"'s online video distribution site is obtained, the queue for review by a second party, different than the first party;

v. soliciting manual human review through the computer controlled interface upon a condition in which the relative correspondence indicates an unreliability, and providing access to a copy of the user uploaded audio-visual content via the queue of the computer controlled interface,

vi. in response to providing access to the copy of the user uploaded audio-visual content via the queue of the computer controlled interface, obtaining manual human assessment data from the second party through the computer controlled interface associated with the first party'"'"'s online video distribution site, the manual human assessment data indicating a result of a comparison of the copy of the user uploaded audio-visual content and data associated with the reference content by one or more human reviewer(s), and

vii. controlling distribution of the user uploaded audio-visual content from the first party'"'"'s online video distribution site to the public based at least in part on the manual human assessment data, in which the user uploaded audio-visual content remains publically accessible on the first party'"'"'s online video distribution site pending receipt of the manual human assessment data; and

viii. upon a condition in which the relative correspondence indicates reliability, controlling distribution of the user uploaded audio-visual content from the first party'"'"'s online video distribution site to the public per a rights holder'"'"'s request.

34. The one or more computer servers of claim 33 in which the determining a relative correspondence between the generated fingerprint data and the fingerprint data associated with the reference content utilizes a scoring such that the manual human review is solicited through the computer controlled interface upon a condition in which the scoring indicates the unreliability.

35. The one or more computer servers of claim 33 in which the user uploaded audio-visual content has been altered to avoid fingerprint matching prior to uploading.

36. The one or more computer servers of claim 33 in which the user uploaded audio-visual content has been rotated prior to uploading.

37. The one or more computer servers of claim 33 in which the one or more processors are configured to cause said one or more computer servers to update the universe of reference content with the user uploaded audio-visual content.

38. The one or more computer servers of claim 33 in which the controlling distribution of the user uploaded audio-visual content from the first party'"'"'s online video distribution site to the public based on at least in part on the manual human assessment data comprises interrupting delivery to a public user.

39. The one or more computer servers of claim 38 in which the one or more processors are configured to provide an explanatory message to the public user in connection with the controlling distribution of the user uploaded audio-visual content from the first party'"'"'s online video distribution site to the public based on at least in part on the manual human assessment data.

40. The one or more computer servers of claim 33 in which the computer interface provides a graphical interface to allow access by the second party for reviewing queued items.

41. The one or more computer servers of claim 33 in which the computer interface provides access to the user uploaded audio-visual content and access to the at least one item of reference content through links to content.

42. The one or more computer servers of claim 33 in which the one or more processors are configured to analyze a transform domain representation of the user uploaded audio-visual content.

43. The one or more computer servers of claim 33 in which the one or more processors are configured to cause the one or more computer servers to associate the generated fingerprint data with the universe of reference content.

44. The one or more computer servers of claim 33 in which the possible match comprises a match that is less than conclusive.

45. The one or more computer servers of claim 33 in which the relative correspondence indicates an unreliability when the possible match falls within a predetermined range, the predetermined range comprising a range below a conclusive match and above a no-match condition.

46. The one or more computer servers of claim 33 in which the generated fingerprint data comprises first plural hashes and the fingerprint data associated with the reference content comprises second plural hashes, in which the possible match is determined by a distance function comparing the first plural hashes with the second plural hashes.

47. The one or more computer servers of claim 33 in which the assessment data comprises subjective human judgment of one or more human reviewer(s) pre-qualified as knowledgeable about the audio-visual content.

48. The one or more computer servers of claim 33 in which the controlling distribution of the user uploaded audio-visual content from the first party'"'"'s online video distribution site to the public based on at least in part on the manual human assessment data considers both the manual human assessment data and a result of the identifying reference content, but gives greater weight to the manual human assessment data relative to the result of the identifying reference content.

49. The one or more computer servers of claim 33 in which the identifying reference content utilizes a Viterbi function.