Methods for identifying video segments and displaying contextual targeted content on a connected television

• US 8,595,781 B2
• Filed: 05/27/2010
• Issued: 11/26/2013
• Est. Priority Date: 05/29/2009
• Status: Active Grant

• Current Assignee: Inscape Data, Inc. (VIZIO, Inc.)
• Original Assignee: Cognitive Media Networks, Inc.
• Inventors: Neumeier, Zeev, Liberty, Edo
• Primary Examiner(s): Goodarzi, Nasser
• Assistant Examiner(s): GRAHAM, PAUL J
• Application Number:
  US12/788,721
• Publication Number:
  US 20100306808A1
• Time in Patent Office: 1,279 Days
• Field of Search: 725/105, 725106-117, 707/780, 707/E17.014
• US Class Current: 725/105
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44204 Monitoring of content usage, e.g. the number of times a movie has been viewed, copied or the amount which has been watched monitoring of user activities for profile generation for accessing a video database G06F16/739; protecting generic digital content where the protection is independent of the precise nature of the content G06F21/10; arrangements for monitoring the use made of the broadcast services in broadcast systems H04H60/31
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/462 Content or additional data management, e.g. creating a master electronic program guide from data received from the Internet and a Head-end, controlling the complexity of a video stream by scaling the resolution or bit-rate based on the client capabilities
• CPC Class Codes: H04N 21/4667 Processing of monitored end-user data, e.g. trend analysis based on the log file of viewer selections
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 21/478 Supplemental services, e.g. displaying phone caller identification, shopping application
• CPC Class Codes: H04N 21/8126 involving additional data, e.g. news, sports, stocks, weather forecasts

Abstract

Systems and methods for identifying which video segment is being displayed on a screen of a television system. The video segment is identified by deriving data from the television signals, the derived data being indicative of the video segment being displayed on the screen. This feature can be used to extract a viewer'"'"'s reaction (such as changing the channel) to a specific video segment (such as an advertisement) and reporting the extracted information as metrics. The systems and methods may further provide contextually targeted content to the television system. The contextual targeting is based on not only identification of the video segment being displayed, but also a determination concerning the playing time or offset time of the particular portion of the video segment being currently displayed.

Claims

1. A method, performed by a centrally located computer system, for automatically providing contextually targeted content to a remotely located television system comprising a video screen, while a video segment is being displayed on the video screen, said method comprising the following steps:

(a) storing in a database a respective reference cue data set for each of a multiplicity of reference video segments, each data set comprising;

(i) a respective sample set of reference data points derived from television signals for respective portions of a respective reference video segment having respective offset times relative to a starting point of said respective reference video segment; and

(ii) a respective set of associated location data, each set of associated location data comprising segment identity data identifying said respective reference video segment and offset time data comprising a respective offset time of a respective portion relative to a starting point of said respective reference video segment;

(b) receiving cue points and respective time stamps from the television system during display of a video segment on the video screen, each cue point comprising a respective sample set of data points derived from television signals for a respective portion of said video segment being displayed on the video screen;

(c) for each received cue point, searching the database for sample sets of reference data points which are a sufficiently close match to the received cue point using a path pursuit algorithm;

(d) storing suspect location data in a history data structure, said suspect location data comprising segment identity data and time offset data associated with the sufficiently close sample sets of reference data points found in step (c), and said data history structure comprising respective bins associated with respective reference video segments corresponding to said suspect location data;

(e) calculating a respective certainty probability for each of said suspect locations, each certainty probability representing a degree of certainty that a respective portion of a reference video segment corresponding to a respective suspect location is the same as location of the portion of said video segment that was displayed on the video screen;

(f) summing storing said certainty probabilities in said bins;

(g) determining whether the sum of said certainty probabilities in any bin exceeds a preset probability threshold;

(h) in response to a determination in step (g) that the sum of said certainty probabilities in any bin exceeds said preset probability threshold, retrieving from said database content associated with the portion of the reference video segment corresponding to said suspect location having said certainty probability in excess of said preset probability threshold; and

(i) sending said retrieved content to the television system while said video segment is still being displayed on the video screen.

2. The method as recited in claim 1, wherein said step of calculating respective certainty probabilities comprises exponential discount with time for past cue points.

3. The method as recited in claim 1, wherein step (a) comprises indexing said data sets for said reference video segments using locality-sensitive hashing, and step (c) comprises locality-sensitive hashing of each received cue point to determine a sector of said database to search in.

4. The method as recited in claim 1, further comprising removing suspect locations which are sufficiently unlikely from said history data structure.

5. A system for providing contextually targeted content to a television system comprising a video screen while a video segment is being displayed on the video screen, said system comprising a database and a processing system,

wherein said database contains a respective reference cue data set for each of a multiplicity of reference video segments, each data set comprising:

(i) a respective sample set of reference data points derived from television signals for respective portions of a respective reference video segment having respective offset times relative to a starting point of said respective reference video segment; and

(ii) a respective set of associated location data, each set of associated location data comprising segment identity data identifying said respective reference video segment and offset time data comprising a respective offset time of a respective portion relative to a starting point of said respective reference video segment; and

wherein said processing system is programmed to perform the following operations while the video segment is being displayed on the video screen;

(a) receiving cue points and respective time stamps from the television system during display of a video segment on the video screen, each cue point comprising a respective sample set of data points derived from television signals for a respective portion of said video segment being displayed on said video screen;

(b) for each received cue point, searching the database for sample sets of reference data points which are a sufficiently close match to the received cue point using a path pursuit algorithm;

(c) storing suspect location data in a history data structure, said suspect location data comprising segment identity data and time offset data associated with the sufficiently close sample sets of reference data points found by operation (b), and said data history structure comprising respective bins associated with respective reference video segments identified by said suspect location data;

(d) calculating a respective certainty probability for each of said suspect locations, each certainty probability representing a degree of certainty that a respective portion of a reference video segment corresponding to a respective suspect location is the same as the portion of said video segment that was displayed on said video screen;

(e) summing said certainty probabilities in said bins;

(f) determining whether the sum of said certainty probabilities in any bin exceeds a preset probability threshold;

(g) in response to a determination by operation (f) that the sum of said certainty probabilities in any bin exceeds said preset probability threshold, retrieving from said database content associated with the portion of the reference video segment corresponding to said suspect location having said certainty probability in excess of said preset probability threshold; and

(h) sending said retrieved content to the television system while said video segment is still being displayed on said video screen.

6. The system as recited in claim 5, wherein said path pursuit algorithm comprises an algorithm for performing probabilistic point location in equal balls.

7. The system as recited in claim 5, wherein said processing system is further programmed to remove suspect locations which are sufficiently unlikely from said history data structure.

8. A method, performed by a centrally located computer system, for automatically identifying which video segment of a multiplicity of video segments is being displayed on a video screen of a remotely located television system, comprising the following steps:

(a) storing in a database a respective reference cue data set for each of a multiplicity of reference video segments, each data set comprising;

(i) a respective sample set of reference data points derived from television signals for respective portions of a respective reference video segment having respective offset times relative to a starting point of said respective reference video segment; and

(ii) a respective set of associated location data, each set of associated location data comprising segment identity data identifying said respective reference video segment and offset time data comprising a respective offset time of a respective portion relative to a starting point of said respective reference video segment;

(b) receiving cue points and respective time stamps from the television system during display of a video segment on the video screen, each cue point comprising a respective sample set of data points derived from television signals for a respective portion of said video segment being displayed on the video screen;

(c) for each received cue point, searching the database for sample sets of reference data points which are a sufficiently close match to the received cue point using a path pursuit algorithm;

(d) storing suspect location data in a history data structure, said suspect location data comprising segment identity data and time offset data associated with the sufficiently close sample sets of reference data points found in step (c), and said data history structure comprising respective bins associated with respective reference video segments corresponding to said suspect location data;

(e) calculating a respective certainty probability for each of said suspect locations, each certainty probability representing a degree of certainty that a respective portion of a reference video segment corresponding to a respective suspect location is the same as location of the portion of said video segment that was ˜

displayed on the video screen;

(f) summing said certainty probabilities in said bins;

(g) determining whether the sum of said certainty probabilities in any bin exceeds a preset probability threshold;

(h) in response to a determination in step (g) that the sum of said certainty probabilities in any bin exceeds said preset probability threshold, retrieving from said database

segment identity data identifying the portion of the reference video segment corresponding to the suspect location having said certainty probability in excess of said preset probability threshold.

9. The method as recited in claim 8, wherein said step of calculating respective certainty probabilities comprises exponential discount with time for past cue points.

10. The method as recited in claim 8, further comprising the step of generating a report including said segment identity data identifying the portion of the reference video segment corresponding to the suspect location having said certainty probability in excess of said preset probability threshold.

11. A method, performed by a centrally located computer system, for automatically identifying a video segment being displayed on a video screen of a remotely located television system, comprising the following steps:

(a) storing in a database a respective reference cue data set for each of a multiplicity of reference video segments, each data set comprising;

(i) a respective sample set of reference data points derived from television signals for respective portions of a respective reference video segment having respective

offset times relative to a starting point of said respective reference video segment; and

(ii) a respective set of associated location data, each set of associated location data comprising segment identity data identifying said respective reference video segment and offset time data comprising a respective offset time of a respective portion relative to a starting point of said respective reference video segment;

(b) receiving cue points and respective time stamps from the television system during display of a video segment on the video screen, each cue point comprising a respective sample set of data points derived from television signals for a respective portion of said video segment being displayed on the video screen;

(c) for each received cue point, searching the database for sample sets of reference data points which are a sufficiently close match to the received cue point using a path pursuit algorithm;

(d) storing suspect location data in a history data structure, said suspect location data comprising segment identity data and time offset data associated with the sufficiently close sample sets of reference data points found in step (c), and said data history structure comprising respective bins associated with respective reference video segments corresponding to said suspect location data;

(e) calculating a respective certainty probability for each of said suspect locations, each certainty probability representing a degree of certainty that a respective portion of a reference video segment corresponding to a respective suspect location is the same as the portion of said video segment that was displayed on the video screen;

(f) retrieving from said database segment identity data identifying the portion of the reference video segment corresponding to the suspect location having a certainty probability in excess of a preset probability threshold; and

(g) generating a report including said segment identity data for said video segment displayed on said video screen.

12. The method as recited in claim 11, wherein said step of calculating respective certainty probabilities comprises exponential discount with time for past cue points.

13. The method as recited in claim 11, wherein said cue points are pixel cue points or audio cue points.

14. A system for automatically identifying a video segment being displayed on a video screen of a remotely located television system, comprising:

a database storing a respective reference cue data set for each of a multiplicity of reference video segments, each data set comprising;

(i) a respective sample set of reference data points derived from television signals for respective portions of a respective reference video segment having respective offset times relative to a starting point of said respective reference video segment; and

(ii) a respective set of associated location data, each set of associated location data comprising segment identity data identifying said respective reference video segment and offset time data comprising a respective offset time of a respective portion relative to a starting point of said respective reference video segment; and

a processor system programmed to perform the following steps;

(a) receiving cue points and respective time stamps from the television system during display of a video segment on the video screen, each cue point comprising a respective sample set of data points derived from television signals for a respective portion of said video segment being displayed on the video screen;

(b) for each received cue point, searching the database for sample sets of reference data points which are a sufficiently close match to the received cue point using a path pursuit algorithm;

(c) storing suspect location data in a history data structure, said suspect location data comprising segment identity data and time offset data associated with the sufficiently close sample sets of reference data points found by step (b), and said data history structure comprising respective bins associated with respective reference video segments identified by said suspect location data;

(d) calculating a respective certainty probability for each of said suspect locations, each certainty probability representing a degree of certainty that a respective portion of a reference video segment corresponding to a respective suspect location is the same as the portion of said video segment that was displayed on the video screen;

(e) retrieving from said database segment identity data identifying the portion of the reference video segment corresponding to the suspect location having a certainty probability in excess of a preset probability threshold; and

(f) generating a report including said segment identity data for said video segment displayed on said video screen.

15. The system as recited in claim 14, wherein said cue points are pixel cue points, each of said pixel cue points being derived by extracting a multiplicity of patches of pixel data, each of said patches comprising a two-dimensional array of sets of pixel values, and computing values which are a function of respective sets of pixel values for each array, said computed values for each patch forming a respective pixel cue point.

16. The system as recited in claim 14, wherein said data points are audio cue points, each of said audio cue points being derived by extracting a multiplicity of sets of audio data samples, and computing values which are a function of respective sets of audio data samples, said computed values for each set of audio data samples forming a respective audio cue point.

17. A web-based server system for automatically providing contextually targeted video or audio content for display on a video screen of a client television system, comprising:

one or more databases for storing a reference cue data set and associated content for each of a multiplicity of reference video segments;

a reference cue data generation computer system programmed to act on a multiplicity of television program sources for generating a multiplicity of reference cue data sets, each reference cue data set comprising;

(i) a sample set of reference data points derived from a respective reference video segment, (ii) a time code derived from the respective broadcast time of said respective video segment, and (iii) a respective set of associated metadata, each set of associated metadata comprising segment identity data identifying said respective reference video segment;

a remotely located client television system comprising means for generating cue data sets during display of a video

segment on said video screen of said client television system, each cue data set comprising a respective sample set of data points derived from television signals for a respective portion of said video segment being displayed on said video screen in a manner consistent with the reference cue data generation computer system and associated time codes;

a video matching computer system programmed to match cue data sets from unknown video segments from said client television system against said reference cue data sets using a path pursuit algorithm comprising searching the database for reference cue data sets which are a sufficiently close match to the received cue data sets using locality sensitive hashing and a probabilistic point location in equal balls algorithm; and

a content retrieval computer system programmed to retrieve content associated with the reference cue data sets which are a closest match to the received cue data sets, and send said retrieved content to said client television system while said video segment is still being displayed on said video screen of said client television system,

wherein said video matching computer system is programmed to determine a closest match by executing the following operations;

(a) store suspect location data in a history data structure, said suspect location data comprising segment identity data and time offset data respectively associated with reference cue data sets which are a sufficiently close match to the received cue data sets, and said data history structure comprising respective bins respectively associated with said suspect location data;

(b) calculate a respective certainty probability for each suspect location identified by said suspect location data, each certainty probability representing a degree of certainty that a respective portion of a reference video segment identified by a respective suspect location is the same as the portion of said video segment that was displayed on said screen of said client television system;

(c) sum said certainty probabilities in said respective bins; and

(d) determine whether the sum of said certainty probabilities in any bin exceeds a preset probability threshold.

18. The web-based server system as recited in claim 17, wherein operation (b) comprises exponential discounting with time for past suspect location data in order to remove from consideration said suspect location data when time distance renders said suspect location data an unlikely match.