Method and system for an automatic content recognition abstraction layer

• US 9,172,994 B2
• Filed: 12/28/2012
• Issued: 10/27/2015
• Est. Priority Date: 02/07/2012
• Status: Active Grant

• Current Assignee: Turner Broadcasting System Incorporated (Warner Bros. Discovery, Inc.)
• Original Assignee: Turner Broadcasting System Incorporated (Warner Bros. Discovery, Inc.)
• Inventors: Sinha, Nishith Kumar, Loheide, Donald Jude, Giles, Matthew Paul
• Primary Examiner(s): GORADIA, SHEFALI DINESH
• Application Number:
  US13/730,352
• Publication Number:
  US 20130202150A1
• Time in Patent Office: 1,033 Days
• Field of Search: 382/100
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24558 Binary matching operations
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06T 1/0021 Image watermarking
• CPC Class Codes: H04N 21/00 Selective content distribution, e.g. interactive television or video on demand [VOD] real-time bi-directional transmission of motion video data H04N7/14 ; broadcast or conference over packet switching networks H04L12/18
• CPC Class Codes: H04N 21/251 Learning process for intelligent management, e.g. learning user preferences for recommending movies details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08
• CPC Class Codes: H04N 21/25866 Management of end-user data customer care in data networks H04L41/5077
• CPC Class Codes: H04N 21/4147 PVR [Personal Video Recorder] H04N5/76 takes precedence ; arrangements for broadcast with accumulation-type receivers H04H20/40
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/432 Content retrieval operation from a local storage medium, e.g. hard-disk details of retrieval of video data and associated meta data in video databases G06F16/739
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/441 Acquiring end-user identification , e.g. using personal code sent by the remote control or by inserting a card
• CPC Class Codes: H04N 21/442 Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk arrangements for monitoring broadcast services or broadcast-related services H04H60/29; arrangements for identifying or recognising characteristics with a direct linkage to broadcast information H04H60/35; monitoring of user activities for profile generation for accessing a video database G06F16/739; monitoring in wireless networks H04W24/00
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/4667 Processing of monitored end-user data, e.g. trend analysis based on the log file of viewer selections
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content
• CPC Class Codes: H04N 21/478 Supplemental services, e.g. displaying phone caller identification, shopping application
• CPC Class Codes: H04N 21/4784 receiving rewards
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02

Abstract

A method and system for an automatic content recognition (ACR) abstraction layer are described in which real-time event manager in an ACR system may be utilized to receive interactive event identifiers and to assign those identifiers to sets of video fingerprints generated by multiple real-time fingerprint servers (RTFSs). The video fingerprints may be generated from a network television feed by utilizing video fingerprint technologies that are different for each RTFS. The interactive event identifiers may be determined from different sources. When determined from content ingested by the ACR system, for example, the interactive event identifiers may be based on a video fingerprinting process separate from those utilized by the RTFSs. The RTFSs may communicate the sets of video fingerprints and interactive event identifiers to corresponding fingerprint match systems, which in turn may communicate with devices that utilize the same video fingerprint technology as the corresponding RTFS.

Claims

1. A method, comprising:

in a real-time event manager;

receiving one or more interactive event identifiers from one or more sources, wherein the one or more interactive event identifiers are utilized to trigger one or more interactive events; and

triggering timing of assignment of the one or more interactive event identifiers to different sets of video fingerprints, wherein;

a different set of video fingerprints is generated from same portions of a network television feed by a respective one of a plurality of fingerprint servers;

each of the plurality of fingerprint servers utilize a video fingerprint technology that is different from a video fingerprint technology utilized by any of the other fingerprint servers; and

identical interactive event identifiers are assigned to the different set of video fingerprints that are generated from the same portions of the network television feed by the different fingerprint technologies.

2. The method according to claim 1, comprising assigning to a set of audio fingerprints the one or more interactive event identifiers, the set of audio fingerprints being generated from the network television feed by an audio server utilizing an audio fingerprint technology.

3. The method according to claim 2, wherein the audio server is operable to communicate the audio fingerprints and the assigned one or more interactive event identifiers to a corresponding fingerprint match system, the fingerprint match system being operable to communicate with a plurality of devices that utilize the same audio fingerprint technology utilized by the audio server.

4. The method according to claim 1, comprising synchronizing the real-time event manager to the plurality of fingerprint servers using a network time protocol reference clock.

5. The method according to claim 1, comprising:

generating in a monitor application module communicatively coupled to the real-time event manager, a set of video fingerprints in real time from the network television feed;

comparing the generated set of video fingerprints to a set of video fingerprints generated from pre-recorded content; and

determining at least a portion of the interactive event identifiers based on the comparison.

6. The method according to claim 5, comprising:

generating a timeline from the pre-recorded content, the timeline comprising a plurality of interactive event identifiers, each of which corresponds to a particular time in the timeline; and

determining from the plurality of interactive event identifiers in the timeline, and based on the comparison, at least a portion of the one or more interactive event identifiers assigned to the different sets of video fingerprints.

7. The method according to claim 6, comprising:

determining a version of the pre-recorded content; and

generating the timeline of the pre-recorded content based on one or more of the version of the pre-recoded content, a set of video fingerprints generated from a master of the pre-recorded content, and a timeline generated from the master of the prerecorded content.

8. The method according to claim 1, comprising:

receiving one or more inputs from a user or live trigger; and

generating based on the one or more inputs, a fingerprint-to-event association that comprises at least a portion of the one or more interactive event identifiers assigned to the different sets of video fingerprints.

9. The method according to claim 1, comprising:

receiving a network television schedule; and

generating based on the network television schedule, at least a portion of the one or more interactive event identifiers assigned to the different sets of video fingerprints.

10. The method according to claim 1, comprising communicating network time reference information to the plurality of fingerprint servers, each of the plurality of fingerprint servers being operable to communicate messages comprising the network time reference information to a corresponding fingerprint match system.

11. The method according to claim 1, comprising communicating to one or both of a television system and a web system the one or more interactive event identifiers, wherein:

the one or more interactive event identifiers are utilized by one or both of the television system and the web system to trigger interactive events; and

the communication is accomplished via one or more of an enhanced television binary interchange format (EBIF) triggering format for legacy cable systems, ID3 tags in web based devices supporting hypertext transfer protocol live streaming (HLS), and satellite network iTV protocols.

12. The method according to claim 1, wherein the assignment of the one or more interactive event identifiers to the different sets of video fingerprints is based on a media time corresponding to each identified interactive event.

13. The method according to claim 1, wherein each of the plurality of fingerprint servers is operable to communicate its corresponding set of video fingerprints and the assigned one or more interactive event identifiers to a corresponding fingerprint match system, the fingerprint match system being operable to communicate with a plurality of devices that utilize the same video fingerprint technology utilized by the fingerprint server that corresponds to that fingerprint match system.

14. A system, comprising:

a real-time event manager that is operable to;

receive one or more interactive event identifiers from one or more sources, wherein the one or more interactive event identifiers are utilized to trigger one or more interactive events; and

trigger timing of assignment of the one or more interactive event identifiers to different sets of video fingerprints;

wherein;

a different set of video fingerprints is generated from same portions of a network television feed by a respective one of a plurality of fingerprint servers;

each of the plurality of fingerprint servers utilize a video fingerprint technology that is different from a video fingerprint technology utilized by any of the other fingerprint servers; and

identical interactive event identifiers are assigned to the different set of video fingerprints that are generated from the same portions of the network television feed by the different fingerprint technologies.

15. The system according to claim 14, wherein the real-time event manager is operable to assign to a set of audio fingerprints the one or more interactive event identifiers, the set of audio fingerprints being generated from the network television feed by an audio server utilizing an audio fingerprint technology.

16. The system according to claim 15, wherein the audio server is operable to communicate the audio fingerprints and the assigned one or more interactive event identifiers to a corresponding fingerprint match system, the fingerprint match system being operable to communicate with a plurality of devices that utilize the same audio fingerprint technology utilized by the audio server.

17. The system according to claim 14, wherein the real-time event manager is operable to be synchronized with the plurality of fingerprint servers using a network time protocol reference clock.

18. The system according to claim 14, comprising a monitor application module that is communicatively coupled to the real-time event manager, the monitor application module being operable to:

generate a set of video fingerprints in real time from the network television feed;

compare the generated set of video fingerprints to a set of video fingerprints generated from pre-recorded content; and

determine at least a portion of the interactive event identifiers based on the comparison.

19. The system according to claim 18, wherein the monitor application module is operable to:

generate a timeline from the pre-recorded content, the timeline comprising a plurality of interactive event identifiers, each of which corresponds to a particular time in the timeline; and

determine from the plurality of interactive event identifiers in the timeline, and based on the comparison, at least a portion of the one or more interactive event identifiers assigned to the different sets of video fingerprints.

20. The system according to claim 19, wherein the monitor application module is operable to:

determine a version of the pre-recorded content; and

generate the timeline of the pre-recorded content based on one or more of the version of the pre-recoded content, a set of video fingerprints generated from a master of the pre-recorded content, and a timeline generated from the master of the prerecorded content.

21. The system according to claim 14, wherein the real-time event manager is

operable to:

receive one or more inputs from a user or live trigger; and

generate based on the one or more inputs, event triggers that comprises at least a portion of the one or more interactive event identifiers assigned to the different sets of video fingerprints via the corresponding RTFS'"'"'s.

22. The system according to claim 14, wherein the real-time event manager is operable to:

receive a network television schedule; and

generate based on the network television schedule, at least a portion of the one or more interactive event identifiers assigned to the different sets of video fingerprints.

23. The system according to claim 14, wherein the real-time event manager is operable to communicate network time reference information to the plurality of fingerprint servers, each of the plurality of fingerprint servers being operable to communicate messages comprising the network time reference information to a corresponding fingerprint match system.

24. The system according to claim 14, wherein the real-time event manager is operable to communicate to one or both of a television system and a web system the one or more interactive event identifiers, wherein:

the one or more interactive event identifiers are utilized by one or both of the television system and the web based system to trigger interactive events;

communication is accomplished via one or more of an enhanced television binary interchange format (EBIF) triggering mechanism for legacy cable systems, ID3 tags in web based devices supporting hypertext transfer protocol live streaming (HLS), and satellite network iTV protocols.

25. The system according to claim 14, wherein the assignment of the one or more interactive event identifiers to the different sets of video fingerprints is based on a media time corresponding to each identified interactive event.

26. The system according to claim 14, wherein each of the plurality of fingerprint servers is operable to communicate its corresponding set of video fingerprints and the assigned one or more interactive event identifiers to a corresponding fingerprint match system, the fingerprint match system being operable to communicate with a plurality of devices that utilize the same video fingerprint technology utilized by the fingerprint server that corresponds to that fingerprint match system.

27. A non-transitory computer readable media having stored thereon a computer program having at least one code section for processing data, the at least one code section being executable by a computer to cause the computer to perform steps comprising:

in a real-time event manager;

receiving one or more interactive event identifiers from one or more sources, wherein the one or more interactive event identifiers are utilized to trigger one or more interactive events; and

triggering timing of assignment of the one or more interactive event identifiers to different sets of video fingerprints, wherein;

a different sets of video fingerprints is generated from same portions of a network television feed by a respective one of a plurality of fingerprint servers;

each of the plurality of fingerprint servers being operable to utilize a video fingerprint technology that is different from a video fingerprint technology utilized by any of the other fingerprint servers; and

identical interactive event identifiers are assigned to the different set of video fingerprints that are generated from the same portions of the network television feed by the different fingerprint technologies.

28. The non-transitory computer readable media according to claim 27, wherein the at least one code section comprises code for assigning to a set of audio fingerprints the one or more interactive event identifiers, the set of audio fingerprints being generated from the network television feed by an audio server utilizing an audio fingerprint technology.

29. The non-transitory computer readable media according to claim 28, wherein the audio server is operable to communicate the audio fingerprints and the assigned one or more interactive event identifiers to a corresponding fingerprint match system, the fingerprint match system being operable to communicate with a plurality of devices that utilize the same audio fingerprint technology utilized by the audio server.

30. The non-transitory computer readable media according to claim 27, wherein the at least one code comprises code for synchronizing the real-time event manager to the plurality of servers using a network time protocol reference clock.

31. The non-transitory computer readable media according to claim 27, wherein the at least one code section comprises code for:

generating in a monitor application module communicatively coupled to the real-time event manager, a set of video fingerprints in real time from the network television feed;

comparing the generated set of video fingerprints to a set of video fingerprints generated from pre-recorded content; and

determining at least a portion of the interactive event identifiers based on the comparison.

32. The non-transitory computer readable media according to claim 31, wherein the at least one code section comprises code for:

generating a timeline from the pre-recorded content, the timeline comprising a plurality of interactive event identifiers, each of which corresponds to a particular time in the timeline; and

determining from the plurality of interactive event identifiers in the timeline, and based on the comparison, at least a portion of the one or more interactive event identifiers assigned to the different sets of video fingerprints.

33. The non-transitory computer readable media according to claim 32, wherein the at least one code section comprises code for:

determining a version of the pre-recorded content; and

generating the timeline of the pre-recorded content based on one or more of the version of the pre-recoded content, a set of video fingerprints generated from a master of the pre-recorded content, and a timeline generated from the master of the prerecorded content.

34. The non-transitory computer readable media according to claim 27, wherein the at least one code section comprises code for:

receiving one or more inputs from a user or live trigger; and

generating based on the one or more inputs, a fingerprint-to-event association that comprises at least a portion of the one or more interactive event identifiers assigned to the different sets of video fingerprints.

35. The non-transitory computer readable media according to claim 27, wherein the at least one code section comprises code for:

receiving a network television schedule; and

generating based on the network television schedule, at least a portion of the one or more interactive event identifiers assigned to the different sets of video fingerprints.

36. The non-transitory computer readable media according to claim 27, wherein the at least one code section comprises code for communicating network time reference information to the plurality of fingerprint servers, each of the plurality of fingerprint servers being operable to communicate messages comprising the network time reference information to a corresponding fingerprint match system.

37. The non-transitory computer readable media according to claim 27, wherein the at least one code section comprises code for communicating to one or both of a television system and a web system the one or more interactive event identifiers, wherein:

the one or more interactive event identifiers are utilized by one or both of the television system and the web system to trigger interactive events; and

the communication is accomplished via one or more of an enhanced television binary interchange format (EBIF) triggering mechanism for legacy cable systems, ID3 tags in web based devices supporting hypertext transfer protocol live streaming (HLS), and satellite network iTV protocols.

38. The non-transitory computer readable media according to claim 27, wherein the assignment of the one or more interactive event identifiers to the different sets of video fingerprints is based on a time corresponding to each identified interactive event.

39. The non-transitory computer readable media according to claim 27, wherein each of the plurality of fingerprint servers is operable to communicate its corresponding set of video fingerprints and the assigned one or more interactive event identifiers to a corresponding fingerprint match system, the fingerprint match system being operable to communicate with a plurality of devices that utilize the same video fingerprint technology utilized by the fingerprint server that corresponds to that fingerprint match system.

40. A method, comprising:

in a real-time event manager;

receiving one or more interactive event identifiers from one or more sources;

triggering the assignment to different sets of video fingerprints the one or more interactive event identifiers, wherein;

a different set of video fingerprints is generated from a network television feed by a respective one of a plurality of fingerprint servers; and

each of the plurality of fingerprint servers utilize a video fingerprint technology that is different from a video fingerprint technology utilized by any of the other fingerprint servers;

generating in a monitor application module communicatively coupled to the real-time event manager, a set of video fingerprints in real time from the network television feed;

comparing the generated set of video fingerprints to a set of video fingerprints generated from pre-recorded content;

determining at least a portion of the interactive event identifiers based on the comparison;

generating a timeline from the pre-recorded content, the timeline comprising a plurality of interactive event identifiers, each of which corresponds to a particular time in the timeline; and

determining from the plurality of interactive event identifiers in the timeline, and based on the comparison, at least a portion of the one or more interactive event identifiers assigned to the different sets of video fingerprints.

41. The method according to claim 40, comprising:

determining a version of the pre-recorded content; and

generating the timeline of the pre-recorded content based on one or more of the version of the pre-recoded content, a set of video fingerprints generated from a master of the pre-recorded content, and a timeline generated from the master of the pre-recorded content.

42. A method, comprising:

in a real-time event manager;

receiving one or more interactive event identifiers from one or more sources;

triggering the assignment to different sets of video fingerprints the one or more interactive event identifiers, wherein;

a different set of video fingerprints is generated from a network television feed by a respective one of a plurality of fingerprint servers; and

each of the plurality of fingerprint servers utilize a video fingerprint technology that is different from a video fingerprint technology utilized by any of the other fingerprint servers; and

communicating to one or both of a television system and a web system the one or more interactive event identifiers, wherein;

the one or more interactive event identifiers are utilized by one or both of the television system and the web system to trigger interactive events; and

the communication is accomplished via one or more of an enhanced television binary interchange format (EBIF) triggering format for legacy cable systems, ID3 tags in web based devices supporting hypertext transfer protocol live streaming (HLS), and satellite network iTV protocols.

43. A system, comprising:

a real-time event manager that is operable to;

receive one or more interactive event identifiers from one or more sources;

trigger the assignment to different sets of video fingerprints the one or more interactive event identifiers, wherein;

a different set of video fingerprints is generated from a network television feed by a respective one of a plurality of fingerprint servers; and

each of the plurality of fingerprint servers utilize a video fingerprint technology that is different from a video fingerprint technology utilized by any of the other fingerprint servers;

a monitor application module that is communicatively coupled to the real-time event manager, the monitor application module being operable to;

generate in a monitor application module communicatively coupled to the real-time event manager, a set of video fingerprints in real time from the network television feed;

compare the generated set of video fingerprints to a set of video fingerprints generated from pre-recorded content;

determine at least a portion of the interactive event identifiers based on the comparison;

generate a timeline from the pre-recorded content, the timeline comprising a plurality of interactive event identifiers, each of which corresponds to a particular time in the timeline; and

determine from the plurality of interactive event identifiers in the timeline, and based on the comparison, at least a portion of the one or more interactive event identifiers assigned to the different sets of video fingerprints.

44. The system according to claim 43, wherein the monitor application is operable to:

determine a version of the pre-recorded content; and

generate the timeline of the pre-recorded content based on one or more of the version of the pre-recoded content, a set of video fingerprints generated from a master of the pre-recorded content, and a timeline generated from the master of the pre-recorded content.

45. A system, comprising:

a real-time event manager that is operable to;

receive one or more interactive event identifiers from one or more sources;

trigger the assignment to different sets of video fingerprints the one or more interactive event identifiers, wherein;

a different set of video fingerprints is generated from a network television feed by a respective one of a plurality of fingerprint servers; and

each of the plurality of fingerprint servers utilize a video fingerprint technology that is different from a video fingerprint technology utilized by any of the other fingerprint servers; and

communicate to one or both of a television system and a web system the one or more interactive event identifiers, wherein;

the one or more interactive event identifiers are utilized by one or both of the television system and the web system to trigger interactive events; and

the communication is accomplished via one or more of an enhanced television binary interchange format (EBIF) triggering format for legacy cable systems, ID3 tags in web based devices supporting hypertext transfer protocol live streaming (HLS), and satellite network iTV protocols.

46. A non-transitory computer readable media having stored thereon a computer program having at least one code section for processing data, the at least one code section being executable by a computer to cause the computer to perform steps comprising:

in a real-time event manager;

receiving one or more interactive event identifiers from one or more sources; and

triggering the assignment to different sets of video fingerprints the one or more interactive event identifiers, wherein;

a different sets of video fingerprints is generated from a network television feed by a respective one of a plurality of fingerprint servers; and

each of the plurality of fingerprint servers being operable to utilize a video fingerprint technology that is different from a video fingerprint technology utilized by any of the other fingerprint servers;

generating in a monitor application module communicatively coupled to the real-time event manager, a set of video fingerprints in real time from the network television feed;

comparing the generated set of video fingerprints to a set of video fingerprints generated from pre-recorded content;

determining at least a portion of the interactive event identifiers based on the comparison;

generating a timeline from the pre-recorded content, the timeline comprising a plurality of interactive event identifiers, each of which corresponds to a particular time in the timeline; and

determining from the plurality of interactive event identifiers in the timeline, and based on the comparison, at least a portion of the one or more interactive event identifiers assigned to the different sets of video fingerprints.

47. The non-transitory computer readable media according to claim 46, wherein the at least one code section comprises code for:

determining a version of the pre-recorded content; and

generating the timeline of the pre-recorded content based on one or more of the version of the pre-recoded content, a set of video fingerprints generated from a master of the pre-recorded content, and a timeline generated from the master of the pre-recorded content.

48. A non-transitory computer readable media having stored thereon a computer program having at least one code section for processing data, the at least one code section being executable by a computer to cause the computer to perform steps comprising:

in a real-time event manager;

receiving one or more interactive event identifiers from one or more sources; and

triggering the assignment to different sets of video fingerprints the one or more interactive event identifiers, wherein;

a different sets of video fingerprints is generated from a network television feed by a respective one of a plurality of fingerprint servers; and

each of the plurality of fingerprint servers being operable to utilize a video fingerprint technology that is different from a video fingerprint technology utilized by any of the other fingerprint servers; and

communicating to one or both of a television system and a web system the one or more interactive event identifiers, wherein;

the one or more interactive event identifiers are utilized by one or both of the television system and the web system to trigger interactive events; and

the communication is accomplished via one or more of an enhanced television binary interchange format (EBIF) triggering mechanism for legacy cable systems, ID3 tags in web based devices supporting hypertext transfer protocol live streaming (HLS), and satellite network iTV protocols.