Systems and methods for real-time television ad detection using an automated content recognition database

• US 8,930,980 B2
• Filed: 03/17/2014
• Issued: 01/06/2015
• Est. Priority Date: 05/27/2010
• Status: Active Grant

• Current Assignee: Inscape Data, Inc. (VIZIO, Inc.)
• Original Assignee: Cognitive Networks, Inc. (VIZIO, Inc.)
• Inventors: Neumeier, Zeev, Reed, Brian
• Primary Examiner(s): Goodarzi, Nasser
• Assistant Examiner(s): NGUYEN, AN V
• Application Number:
  US14/217,094
• Publication Number:
  US 20140282673A1
• Time in Patent Office: 295 Days
• Field of Search: 725/19, 725/22, 725/20, 725/90, 725/9, 348/772, 348/553
• US Class Current: 725/19
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/442 Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk arrangements for monitoring broadcast services or broadcast-related services H04H60/29; arrangements for identifying or recognising characteristics with a direct linkage to broadcast information H04H60/35; monitoring of user activities for profile generation for accessing a video database G06F16/739; monitoring in wireless networks H04W24/00
• CPC Class Codes: H04N 21/44204 Monitoring of content usage, e.g. the number of times a movie has been viewed, copied or the amount which has been watched monitoring of user activities for profile generation for accessing a video database G06F16/739; protecting generic digital content where the protection is independent of the precise nature of the content G06F21/10; arrangements for monitoring the use made of the broadcast services in broadcast systems H04H60/31
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/462 Content or additional data management, e.g. creating a master electronic program guide from data received from the Internet and a Head-end, controlling the complexity of a video stream by scaling the resolution or bit-rate based on the client capabilities
• CPC Class Codes: H04N 21/4667 Processing of monitored end-user data, e.g. trend analysis based on the log file of viewer selections
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 21/478 Supplemental services, e.g. displaying phone caller identification, shopping application
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/8126 involving additional data, e.g. news, sports, stocks, weather forecasts

Abstract

A system and method for the automated real-time detection and processing of commercial messages, public service announcements or similar short-duration, repeated TV programming segments occurring in one or more broadcast video steams is described. A process is utilized that identifies possible commercial segments by identifying discrete video segments that have specific short durations among other attributes. Video segments that appear to have these characteristics are considered likely to be television commercials, promotions, or public service announcements and are stored in a TV Ad database which is separate from the primary television content database. Incoming video from a plurality of television programming sources is process into fingerprints and placed in a master TV content database. At the same time, said incoming programming is tested for matches in a TV ad database. If a match is found, then the presumed TV ad is removed from the master content database.

Claims

1. A method, comprising:

ingesting at least one audiovisual stream, including at least storing at least some data associated with one or more frames of the audiovisual stream into a data store of program content;

determining a candidate segment, the candidate segment including at least one or more sequential frames from the at least one ingested audiovisual stream, including at least;

adjusting at least one threshold associated with matching, the adjusted at least one threshold associated with a lower returned number of suspects;

matching the candidate segment with one or more contiguous sequences of frames previously stored in the data store based at least partially on the adjusted at least one threshold; and

if the candidate segment matches the one or more contiguous sequences of frames previously stored in the data store based at least partially on the adjusted at least one threshold, determining starting and ending points of the candidate segment, including at least;

adjusting at least one threshold associated with matching, the adjusted at least one threshold associated with a higher returned number of suspects;

comparing one or more indications associated with each frame in the candidate segment with one or more indications associated with a corresponding frame in the one or more contiguous sequences of frames previously stored in the data store, the comparing based at least partially on the adjusted at least one threshold; and

testing for inconsistencies between each frame in the candidate segment and a corresponding frame in the one or more contiguous sequences of frames previously stored in the data store, based at least partially on the comparing; and

testing the determined candidate segment for at least one match in a data store of known advertisements and, if testing the candidate segment returns at least one match in the data store of known advertisements, at least removing at least some data associated with the candidate segment from the data store of program content,

wherein at least one of the ingesting, determining, or testing is at least partially implemented using at least one processing device.

2. The method of claim 1, further comprising:

testing the determined candidate segment for at least one match in the data store of program content and, if testing the candidate segment returns at least one match in the data store of program content, at least removing at least some data associated with the candidate segment from the data store of program content and storing at least some data associated with the candidate segment in the data store of known advertisements.

3. The method of claim 1, wherein determining a candidate segment, the candidate segment including at least one or more sequential frames from the at least one ingested audiovisual stream comprises:

receiving one or more indications of a frame of the ingested audiovisual data stream;

analyzing the one or more indications of a frame, including at least determining whether the one or more indications could be a starting frame of a candidate segment;

if a starting frame is determined, analyzing one or more frames received subsequent to the starting frame, including at least determining whether one of the one or more frames received subsequent to the starting frame could be an ending frame; and

if an ending frame is determined, establishing (i) the starting frame, (ii) the one or more frames received subsequent to the starting frame and previous to the ending frame, and (iii) the ending frame as the candidate segment.

4. The method of claim 3, wherein analyzing the one or more indications of a frame, including at least determining whether the one or more indications could be a starting frame of a candidate segment and analyzing one or more frames received subsequent to the starting frame, including at least determining whether a frame received subsequent to the starting frame could be an ending frame comprise:

determining whether a frame is at least one of a monochromatic frame or a frame having one or more substantially different fingerprint values from an immediately-previous frame and, if a frame is at least one of a monochromatic frame or a frame having one or more substantially different fingerprint values from an immediately-previous frame, establishing the frame as at least one of a starting frame or an ending frame.

5. The method of claim 1, wherein determining a candidate segment, the candidate segment including at least one or more sequential frames from the at least one ingested audiovisual stream comprises:

receiving one or more indications of a frame of the ingested audiovisual data stream;

comparing data associated with a contiguous sequence of ingested frames ending with the most recently received frame with data associated with one or more contiguous sequences of frames previously stored in the data store of program content; and

providing one or more indications of one or more contiguous sequences of frames previously stored in the data store of program content based at least partially on the comparing data.

6. The method of claim 5, further comprising:

for each of the one or more indicated contiguous sequences of frames, determining whether the contiguous sequence of frames is an advertisement.

7. The method of claim 6, wherein determining whether the contiguous sequence of frames is an advertisement comprises:

evaluating at least one of a length in seconds or a count in frames associated with the contiguous sequence of frames and, if the at least one of a length in seconds or a count in frames is substantially similar to a standard advertisement length, establishing the contiguous sequence of frames as a candidate segment.

8. The method of claim 1, wherein adjusting at least one threshold associated with matching, the adjusted at least one threshold associated with a lower returned number of suspects comprises:

adjusting at least one radius associated with a path pursuit algorithm, the adjusted at least one radius associated with a lower returned number of suspects.

9. The method of claim 1, wherein adjusting at least one threshold associated with matching, the adjusted at least one threshold associated with a lower returned number of suspects comprises:

adjusting at least one duration related to at least one bin associated with a path pursuit algorithm, the adjusted at least one radius associated with a lower returned number of suspects.

10. The method of claim 1, wherein testing for inconsistencies between each frame in the candidate segment and a corresponding frame in the one or more contiguous sequences of frames previously stored in the data store, based at least partially on the comparing comprises:

if any frame of the candidate segment does not have a corresponding match in the one or more contiguous sequences of frames previously stored in the data store, dropping the frame of the candidate segment without a corresponding match from the one or more contiguous sequences of frames previously stored in the data store.

11. The method of claim 1, wherein testing for inconsistencies between each frame in the candidate segment and a corresponding frame in the one or more contiguous sequences of frames previously stored in the data store, based at least partially on the comparing comprises:

if any frame of the one or more contiguous sequences of frames previously stored in the data store does not have a corresponding match in the candidate segment, adding the frame of the one or more contiguous sequences of frames previously stored in the data store to the candidate segment.

12. The method of claim 1, wherein testing for inconsistencies between each frame in the candidate segment and a corresponding frame in the one or more contiguous sequences of frames previously stored in the data store, based at least partially on the comparing comprises:

testing for inconsistencies between each frame in the candidate segment and a corresponding frame in the one or more contiguous sequences of frames previously stored in the data store, based at least partially on the comparing; and

remediating any inconsistencies found via testing, the remediating operable to identify borders of the candidates.

13. The method of claim 1, wherein ingesting at least one audiovisual stream, including at least storing at least some data associated with one or more frames of the audiovisual stream into a data store of program content and testing the determined candidate segment for at least one match in a data store of known advertisements and, if testing the candidate segment returns at least one match in the data store of known advertisements, at least removing at least some data associated with the candidate segment from the data store of program content comprise:

ingesting at least two audiovisual streams, including at least storing at least some data associated with one or more frames of at least one of the at least two audiovisual streams into a data store of program content, the at least some data including at least one indication of the audiovisual stream from which the one or more frames were received, and testing the determined candidate segment for at least one match in a data store of known advertisements and, if testing the candidate segment returns at least one match in the data store of known advertisements, the at least one match related to any of the at least two audiovisual streams, at least removing at least some data associated with the candidate segment from the data store of program content.

14. The method of claim 1, wherein determining a candidate segment, the candidate segment including at least one or more sequential frames from the at least one ingested audiovisual stream comprises:

confirming a candidate segment having a low threshold match at least partially based on at least one duration of the candidate segment; and

confirming the candidate segment having a high threshold match at least partially based on a match of one or more fingerprints of the candidate segment with a known advertisement.

15. The method of claim 1, wherein determining a candidate segment, the candidate segment including at least one or more sequential frames from the at least one ingested audiovisual stream comprises:

determining a segment which may be at least one of an advertisement, a commercial, a public service announcement, a promotion, at least a portion of an ad break, at least a portion of an ad pod, or an upcoming television programming promotion.

16. The method of claim 1, wherein determining a candidate segment, the candidate segment including at least one or more sequential frames from the at least one ingested audiovisual stream comprises:

determining a candidate segment, the candidate segment including at least one or more sequential video frames from the at least one ingested audiovisual stream.

17. The method of claim 1, wherein ingesting at least one audiovisual stream, including at least storing at least some data associated with one or more frames of the audiovisual stream into a data store of program content comprises:

ingesting at least one audiovisual stream associated with at least one television program source, including at least one of a satellite ingest, a fiber optic ingest, a microwave ingest, a cable ingest, a telephonic ingest, an ISDN ingest, an Internet ingest, an over-the-air ingest, or a recorded medium ingest.

18. The method of claim 1, wherein ingesting at least one audiovisual stream, including at least storing at least some data associated with one or more frames of the audiovisual stream into a data store of program content comprises:

ingesting at least one audiovisual stream associated with at least one television program source at least partially received via at least one satellite dish, at least one fiber optic connection, at least one microwave receiver, at least one cable television connection, at least one telephone line, at least one ISDN line, at least one network connection, at least one over-the-air antenna, or at least one recorded medium.

19. A computer program product, comprising:

at least one non-transitory computer-readable medium including at least;

one or more instructions for ingesting at least one audiovisual stream, including at least storing at least some data associated with one or more frames of the audiovisual stream into a data store of program content;

one or more instructions for determining a candidate segment, the candidate segment including at least one or more sequential frames from at least one ingested audiovisual stream, including at least;

one or more instructions for adjusting at least one threshold associated with matching, the adjusted at least one threshold associated with a lower returned number of suspects;

one or more instructions for matching the candidate segment with one or more contiguous sequences of frames previously stored in the data store based at least partially on the adjusted at least one threshold; and

one or more instructions for if the candidate segment matches the one or more contiguous sequences of frames previously stored in the data store based at least partially on the adjusted at least one threshold, determining starting and ending points of the candidate segment, including at least;

one or more instructions for adjusting at least one threshold associated with matching, the adjusted at least one threshold associated with a higher returned number of suspects;

one or more instructions for comparing one or more indications associated with each frame in the candidate segment with one or more indications associated with a corresponding frame in the one or more contiguous sequences of frames previously stored in the data store, the comparing based at least partially on the adjusted at least one threshold; and

one or more instructions for testing for inconsistencies between each frame in the candidate segment and a corresponding frame in the one or more contiguous sequences of frames previously stored in the data store, based at least partially on the comparing; and

one or more instructions for testing a determined candidate segment for at least one match in a data store of known advertisements and, if testing the candidate segment returns at least one match in the data store of known advertisements, at least removing at least some data associated with the candidate segment from the data store of program content.

20. The computer program product of claim 19, wherein at least some of the one or more instructions for adjusting at least one threshold associated with matching, the adjusted at least one threshold associated with a lower returned number of suspects comprises:

one or more instructions for adjusting at least one duration related to at least one bin associated with a path pursuit algorithm, the adjusted at least one radius associated with a lower returned number of suspects.

21. A system, comprising:

circuitry for ingesting at least one audiovisual stream, including at least storing at least some data associated with one or more frames of the audiovisual stream into a data store of program content;

circuitry for determining a candidate segment, the candidate segment including at least one or more sequential frames from at least one ingested audiovisual stream, including at least;

circuitry for adjusting at least one threshold associated with matching, the adjusted at least one threshold associated with a lower returned number of suspects;

circuitry for matching the candidate segment with one or more contiguous sequences of frames previously stored in the data store based at least partially on the adjusted at least one threshold; and

circuitry for, if the candidate segment matches the one or more contiguous sequences of frames previously stored in the data store based at least partially on the adjusted at least one threshold, determining starting and ending points of the candidate segment, including at least;

circuitry for adjusting at least one threshold associated with matching, the adjusted at least one threshold associated with a higher returned number of suspects;

circuitry for comparing one or more indications associated with each frame in the candidate segment with one or more indications associated with a corresponding frame in the one or more contiguous sequences of frames previously stored in the data store, the comparing based at least partially on the adjusted at least one threshold; and

circuitry for testing for inconsistencies between each frame in the candidate segment and a corresponding frame in the one or more contiguous sequences of frames previously stored in the data store, based at least partially on the comparing; and

circuitry for testing a determined candidate segment for at least one match in a data store of known advertisements and, if testing the candidate segment returns at least one match in the data store of known advertisements, at least removing at least some data associated with the candidate segment from the data store of program content.

22. The system of claim 21, wherein the circuitry for adjusting at least one threshold associated with matching, the adjusted at least one threshold associated with a lower returned number of suspects comprises:

circuitry for adjusting at least one duration related to at least one bin associated with a path pursuit algorithm, the adjusted at least one radius associated with a lower returned number of suspects.

23. A system, comprising:

means for ingesting at least one audiovisual stream, including at least storing at least some data associated with one or more frames of the audiovisual stream into a data store of program content;

means for determining a candidate segment, the candidate segment including at least one or more sequential frames from at least one ingested audiovisual stream, including at least;

means for adjusting at least one threshold associated with matching, the adjusted at least one threshold associated with a lower returned number of suspects;

means for matching the candidate segment with one or more contiguous sequences of frames previously stored in the data store based at least partially on the adjusted at least one threshold; and

means for, if the candidate segment matches the one or more contiguous sequences of frames previously stored in the data store based at least partially on the adjusted at least one threshold, determining starting and ending points of the candidate segment, including at least;

means for adjusting at least one threshold associated with matching, the adjusted at least one threshold associated with a higher returned number of suspects;

means for comparing one or more indications associated with each frame in the candidate segment with one or more indications associated with a corresponding frame in the one or more contiguous sequences of frames previously stored in the data store, the comparing based at least partially on the adjusted at least one threshold; and

means for testing for inconsistencies between each frame in the candidate segment and a corresponding frame in the one or more contiguous sequences of frames previously stored in the data store, based at least partially on the comparing; and

means for testing a determined candidate segment for at least one match in a data store of known advertisements and, if testing the candidate segment returns at least one match in the data store of known advertisements, at least removing at least some data associated with the candidate segment from the data store of program content,

wherein at least one of the means for ingesting, means for determining, or means for testing is at least partially implemented using computing hardware.

24. The system of claim 23, wherein the means for adjusting at least one threshold associated with matching, the adjusted at least one threshold associated with a lower returned number of suspects comprises:

means for adjusting at least one duration related to at least one bin associated with a path pursuit algorithm, the adjusted at least one radius associated with a lower returned number of suspects.

25. A system, comprising:

one or more computing devices; and

at least some instructions which executed on at least one of the one or more computing devices, cause at least one of the one or more computing devices to at least;

ingest at least one audiovisual stream, including at least storing at least some data associated with one or more frames of the audiovisual stream into a data store of program content;

determine a candidate segment, the candidate segment including at least one or more sequential frames from at least one ingested audiovisual stream, including at least;

adjusting at least one threshold associated with matching, the adjusted at least one threshold associated with a lower returned number of suspects;

matching the candidate segment with one or more contiguous sequences of frames previously stored in the data store based at least partially on the adjusted at least one threshold; and

if the candidate segment matches the one or more contiguous sequences of frames previously stored in the data store based at least partially on the adjusted at least one threshold, determining starting and ending points of the candidate segment, including at least;

adjusting at least one threshold associated with matching, the adjusted at least one threshold associated with a higher returned number of suspects;

comparing one or more indications associated with each frame in the candidate segment with one or more indications associated with a corresponding frame in the one or more contiguous sequences of frames previously stored in the data store, the comparing based at least partially on the adjusted at least one threshold; and

testing for inconsistencies between each frame in the candidate segment and a corresponding frame in the one or more contiguous sequences of frames previously stored in the data store, based at least partially on the comparing; and

test a determined candidate segment for at least one match in a data store of known advertisements and, if testing the candidate segment returns at least one match in the data store of known advertisements, at least removing at least some data associated with the candidate segment from the data store of program content.

26. The system of claim 25, wherein one or more of the at least some instructions which, when executed on at least one of the one or more computing devices, cause at least one of the one or more computing devices to adjust at least one threshold associated with matching, the adjusted at least one threshold associated with a lower returned number of suspects, cause at least one of the computing devices to at least:

adjust at least one duration related to at least one bin associated with a path pursuit algorithm, the adjusted at least one radius associated with a lower returned number of suspects.