Dynamic video overlays

• US 10,412,447 B2
• Filed: 12/16/2016
• Issued: 09/10/2019
• Est. Priority Date: 12/16/2015
• Status: Active Grant

• Current Assignee: Roku, Inc.
• Original Assignee: Gracenote, Inc. (RR Donnelley & Sons Company)
• Inventors: Cremer, Markus K., Dunker, Peter, Scherf, Steven D., Pearce, David Henry, Li, Sherman Ling Fung, Harron, Wilson, Dimitriou, Konstantinos Antonios, Lee, Dewey Ho
• Primary Examiner(s): Corbo, Nicholas T
• Application Number:
  US15/381,513
• Publication Number:
  US 20170180793A1
• Time in Patent Office: 998 Days
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/431 Generation of visual interfaces for content selection or interaction; Content or additional data rendering
• CPC Class Codes: H04N 21/4312 involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/4314 for fitting data in a restricted space on the screen, e.g. EPG data in a rectangular grid
• CPC Class Codes: H04N 21/4318 by altering the content in the rendering process, e.g. blanking, blurring or masking an image region image enhancement or restoration in general G06T5/00
• CPC Class Codes: H04N 21/4325 by playing back content from the storage medium reproduction of recorded television signals H04N5/76; reproduction of recorded television signals H04N9/79
• CPC Class Codes: H04N 21/4333 Processing operations in response to a pause request
• CPC Class Codes: H04N 21/4383 Accessing a communication channel
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44016 involving splicing one content stream with another content stream, e.g. for substituting a video clip
• CPC Class Codes: H04N 21/458 Scheduling content for creating a personalised stream, e.g. by combining a locally stored advertisement with an incoming stream; Updating operations, e.g. for OS modules ; time-related management operations arrangements for replacing or switching information during the broadcast or during the distribution H04H20/10
• CPC Class Codes: H04N 21/47217 for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks specific graphical features in visual interfaces H04N21/4312
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02

Abstract

A client device accesses a video input stream from an intermediate device for display. The client device analyzes the video input stream to determine that the video input stream matches a template indicating a semi-transparent overlay. Based on the video input stream matching the template, a video output stream is generated and caused to be presented on a display. In some example embodiments, the analysis is performed while the client device is replacing video content received from a content source via the intermediate device. For example, commercials transmitted from a national content provider to a smart TV via a set-top box may be replaced with targeted commercials. During the replacement, semi-transparent menus generated by the set-top box may be detected and the replacement video altered by the smart TV to include the menus.

Claims

1. A system comprising:

a memory having instructions embodied thereon; and

one or more processors configured by the instructions to perform operations comprising;

accessing, from an intermediate device upstream of the one or more processors, a video input stream that includes first content and a semi-transparent overlay overlaid at least partially on the first content in an overlay zone, wherein the first content is from a content provider and the semi-transparent overlay is overlaid on the first content by the intermediate device;

accessing a template that includes an indication of the overlay zone in the video input stream;

analyzing, using the template, the video input stream to determine that the video input stream includes the semi-transparent overlay in the overlay zone;

accessing replacement content for replacing the first content outside of the overlay zone;

while accessing the video input stream, generating a video output stream that includes a modified semi-transparent overlay overlaid on the replacement content in the overlay zone; and

causing the video output stream, including the replacement content outside of the overlay zone and the modified semi-transparent overlay in the overlay zone, to be presented on a display.

2. The system of claim 1, wherein the operations further comprise:

modifying the semi-transparent overlay to cancel out the first content presented in the overlay zone, using operations comprising;

computing a difference between the first content and the video input stream for each pixel in the overlay zone; and

replacing each pixel in the overlay zone based on the difference and a threshold.

3. The system of claim 2, wherein the operations further comprise:

generating a fingerprint of the first content; and

identifying the first content based on the fingerprint.

4. The system of claim 2, wherein:

the template indicates a degree of transparency; and

the replacing of each pixel in the overlay zone is further based on the degree of transparency.

5. The system of claim 4, wherein:

the replacing of each pixel in the overlay zone is further based on a corresponding pixel of the first content.

6. The system of claim 1, wherein:

the indication of the overlay zone in the video input stream comprises coordinates of boundaries of the overlay zone.

7. The system of claim 1, wherein:

the indication of the overlay zone in the video input stream comprises a bitmap.

8. The system of claim 1, wherein the operations further comprise:

modifying the semi-transparent overlay to cancel out the first content presented in the overlay zone, using operations comprising;

identifying an average color of a background of the overlay zone in the video input stream;

comparing a color of each pixel of a frame of the first content lacking the semi-transparent overlay to the average color of the background; and

for each of the pixels having a color within a predetermined range of the average color of the background, changing the color of the pixel to the average color of the background.

9. The system of claim 1, wherein the operations further comprise:

identifying a text portion of the semi-transparent overlay; and

modifying the semi-transparent overlay to cancel out the first content while maintaining the text portion of the semi-transparent overlay without change.

10. The system of claim 9, wherein the identifying of the text portion of the semi-transparent overlay comprises:

identifying, based on the template, a region that may contain text; and

determining, based on color values of pixels within the region and a threshold, which pixels within the region are included in the text portion of the semi-transparent overlay.

11. The system of claim 1, wherein:

the template indicates a color; and

the generating of the video output stream that includes the modified semi-transparent overlay overlaid on the replacement content comprises semi-transparently applying the color to at least a portion of the overlay zone.

12. The system of claim 11, wherein:

the template indicates an information type of text to be presented; and

the generating of the video output stream that includes the modified semi-transparent overlay overlaid on the replacement content further comprises adding text to at least a portion of the overlay zone, the added text based on the information type indicated in the template.

13. A method comprising:

accessing, by one or more processors, a video input stream that includes first content and a semi-transparent overlay overlaid at least partially on the first content in an overlay zone, wherein the first content is from a content provider and the semi-transparent overlay is overlaid on the first content by an intermediate device that is upstream of the one or more processors;

accessing, from a database that stores a plurality of templates, a template that includes an indication of the overlay zone in the video input stream;

analyzing, using the template, the video input stream to determine that the video input stream includes the semi-transparent overlay in the overlay zone;

accessing replacement content for replacing the first content outside of the overlay zone;

while accessing the video input stream, generating, by one or more processors, a video output stream that includes a modified semi-transparent overlay overlaid on the replacement content in the overlay zone; and

causing, by the one or more processors, the video output stream, including the replacement content outside of the overlay zone and the modified semi-transparent overlay in the overlay zone, to be presented on a display.

14. The method of claim 13, further comprising:

modifying the semi-transparent overlay to cancel out the first content presented in the overlay zone, using operations comprising;

computing a difference between the first content and the video input stream for each pixel in the overlay zone; and

replacing each pixel in the overlay zone based on the difference and a threshold.

15. The method of claim 14, further comprising:

generating a fingerprint of the first content; and

identifying the first content based on the fingerprint.

16. The method of claim 14, wherein:

the template indicates a degree of transparency; and

the replacing of each pixel in the overlay zone is further based on the degree of transparency.

17. The method of claim 16, wherein:

the replacing of each pixel in the overlay zone is further based on a corresponding pixel of the first content.

18. The method of claim 13, further comprising:

modifying the semi-transparent overlay to cancel out the first content presented in the overlay zone, using operations comprising;

identifying an average color of a background of the overlay zone in the video input stream;

comparing a color of each pixel of a frame of the first content lacking the semi-transparent overlay to the average color of the background; and

for each of the pixels having a color within a predetermined range of the average color of the background, changing the color of the pixel to the average color of the background.

19. The method of claim 13, further comprising:

identifying a text portion of the semi-transparent overlay; and

modifying the semi-transparent overlay to cancel out the first content while maintaining the text portion of the semi-transparent overlay without change.

20. A non-transitory machine-readable medium that stores instructions which, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

accessing, from an intermediate device upstream of the one or more processors, a video input stream that includes first content and a semi-transparent overlay overlaid at least partially on the first content in an overlay zone, wherein the first content is from a content provider and the semi-transparent overlay is overlaid on the first content by the intermediate device;

accessing a template that includes an indication of the overlay zone in the video input stream;

analyzing, using the template, the video input stream to determine that the video input stream includes the semi-transparent overlay in the overlay zone;

accessing replacement content for replacing the first content outside of the overlay zone;

while accessing the video input stream, generating a video output stream that includes a modified semi-transparent overlay overlaid on the replacement content in the overlay zone; and

causing the video output stream, including the replacement content outside of the overlay zone and the modified semi-transparent overlay in the overlay zone, to be presented on a display.