Dynamic video overlays

• US 10,142,680 B2
• Filed: 12/16/2016
• Issued: 11/27/2018
• Est. Priority Date: 12/16/2015
• Status: Active Grant

• Current Assignee: Roku, Inc.
• Original Assignee: Gracenote, Inc. (RR Donnelley & Sons Company)
• Inventors: Cremer, Markus K., Dunker, Peter, Scherf, Steven D., Pearce, David Henry, Li, Sherman Ling Fung, Harron, Wilson, Dimitriou, Konstantinos Antonios, Lee, Dewey Ho
• Primary Examiner(s): HUYNH, AN SON PHI
• Application Number:
  US15/381,496
• Publication Number:
  US 20170150213A1
• Time in Patent Office: 711 Days
• Field of Search: None
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/431 Generation of visual interfaces for content selection or interaction; Content or additional data rendering
• CPC Class Codes: H04N 21/4312 involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/4314 for fitting data in a restricted space on the screen, e.g. EPG data in a rectangular grid
• CPC Class Codes: H04N 21/4318 by altering the content in the rendering process, e.g. blanking, blurring or masking an image region image enhancement or restoration in general G06T5/00
• CPC Class Codes: H04N 21/4325 by playing back content from the storage medium reproduction of recorded television signals H04N5/76; reproduction of recorded television signals H04N9/79
• CPC Class Codes: H04N 21/4333 Processing operations in response to a pause request
• CPC Class Codes: H04N 21/4383 Accessing a communication channel
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44016 involving splicing one content stream with another content stream, e.g. for substituting a video clip
• CPC Class Codes: H04N 21/458 Scheduling content for creating a personalised stream, e.g. by combining a locally stored advertisement with an incoming stream; Updating operations, e.g. for OS modules ; time-related management operations arrangements for replacing or switching information during the broadcast or during the distribution H04H20/10
• CPC Class Codes: H04N 21/47217 for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks specific graphical features in visual interfaces H04N21/4312
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02

Abstract

A client device accesses a video input stream from an intermediate device for display. The client device analyzes the video input stream to determine that the video input stream matches a template indicating a change of speed. In some example embodiments, the analysis is performed while the client device is generating a video output stream that replaces video content received from a content source via the intermediate device. Based on the video input stream matching the template, the client device modifies the video output stream to show the change of speed. For example, commercials transmitted from a national content provider to a smart TV via a set-top box may be replaced with targeted commercials. During the replacement, fast-forwarding of the commercial may be detected by matching a template with the video input stream. While the video input stream is being fast-forwarded, the smart TV fast-forwards the targeted commercial.

Claims

1. A method comprising:

causing, by a processor of a client device, a video output stream to be presented on a display of the client device; and

while causing the video output stream to be presented on the display;

accessing, by the processor, a video input stream comprising first video content from a content provider and second video content generated by a set-top box upstream of the processor, wherein, in a frame of the video input stream, the first video content corresponds to a first screen portion of the display and the second video content corresponds to a second screen portion of the display, wherein the video output stream comprises third video content from a replacement content source, wherein the third video content differs from the first video content and is accessed separately from the video input stream, wherein the third video content is selected in response to a video identification system receiving a query fingerprint from a query fingerprint generator of the client device and the video identification system identifying a match between the query fingerprint and at least one reference fingerprint generated by a reference fingerprint generator, wherein the query fingerprint is generated based on a frame of the first video content, and wherein the video identification system and the reference fingerprint generator are remote from the client device;

accessing, from a database, a template corresponding to the second screen portion of the display;

analyzing, by the processor, the video input stream to determine that at least a portion of the second video content of the frame of the video input stream matches the template;

based on the at least the portion of the second video content of the frame of the video input stream matching the template, determining, by the processor, that a speed of play of the video input stream has been modified from a default speed;

responsive to determining that the speed of play of the video input stream has been modified, modifying, by the processor, the video output stream by modifying a speed of play of the video output stream to generate a modified video output stream that mimics the speed of play of the video input stream; and

responsive to modifying the video output stream, causing, by the processor, the modified video output stream to be presented on the display.

2. The method of claim 1, further comprising:

based on the at least the portion of the second video content of the frame of the video input stream matching the template, modifying the video output stream to include the second video content generated by the set-top box.

3. The method of claim 1, further comprising:

analyzing the video input stream to determine the speed of play of the video input stream; and

wherein;

the modifying of the speed of play of the video output stream includes setting the speed of play of the video output stream to the speed of play of the video input stream.

4. The method of claim 3, further comprising:

generating fingerprints for a plurality of frames of the video input stream; and

wherein;

the analyzing of the video input stream to determine the speed of play of the video input stream is based on the fingerprints for the plurality of frames of the video input stream.

5. The method of claim 1, further comprising:

identifying a replaceable segment of the video input stream; and

while the video input stream provides the identified replaceable segment, causing the video output stream to switch from the video input stream to an alternative video segment comprising the third video content from the replacement content source; and

wherein;

the analyzing of the video input stream to determine that at least the portion of the second video content of the frame of the video input stream matches the template is performed while the video output stream provides the alternative video segment.

6. The method of claim 5, further comprising:

based on the at least the portion of the second video content of the frame of the video input stream matching the template, generating a modified frame of the video output stream that is based on both the alternative video segment and the video input stream.

7. The method of claim 6, wherein:

the template defines a region of a frame; and

the generating of the modified frame of the video output stream comprises basing a region of the modified frame of the video output stream that corresponds to the region defined by the template on the video input stream and basing a remaining region of the modified frame of the video output stream on the alternative video segment.

8. A system comprising:

a memory that stores instructions; and

one or more processors configured by the instructions to perform operations comprising;

causing a video output stream to be presented on a display of a client device;

while causing the video output stream to be presented on the display;

accessing a video input stream comprising first video content from a content provider and second video content generated by a set-top box upstream of the one or more processors, wherein, in a frame of the video input stream, the first video content corresponds to a first screen portion of the display and the second video content corresponds to a second screen portion of the display, wherein the video output stream comprises third video content from a replacement content source, wherein the third video content differs from the first video content and is accessed separately from the video input stream, wherein the third video content is selected in response to a video identification system receiving a query fingerprint from a query fingerprint generator of the client device and the video identification system identifying a match between the query fingerprint and at least one reference fingerprint generated by a reference fingerprint generator, wherein the query fingerprint is generated based on a frame of the first video content, and wherein the video identification system and the reference fingerprint generator are remote from the client device;

accessing a template corresponding to a screen portion of the display;

analyzing the video input stream to determine that at least a portion of the second video content of the frame of the video input stream matches the template;

based on the at least the portion of the second video content of the frame of the video input stream matching the template, determining that a speed of play of the video input stream has been modified from a default speed; and

responsive to determining that the speed of play of the video input stream has been modified, modifying the video output stream by modifying a speed of play of the video output stream to generate a modified video output stream that mimics the speed of play of the video input stream; and

responsive to modifying the video output stream, causing the modified video output stream to be presented on the display.

9. The system of claim 8, wherein the operations further comprise:

based on the at least the portion of the second video content of the frame of the video input stream matching the template, modifying the video output stream to include the second video content generated by the set-top box.

10. The system of claim 8, wherein:

the operations further comprise;

analyzing the video input stream to determine the speed of play of the video input stream; and

the modifying of the speed of play of the video output stream includes setting the speed of play of the video output stream to the speed of play of the video input stream.

11. The system of claim 10, wherein:

the operations further comprise;

generating fingerprints for a plurality of frames of the video input stream; and

the analyzing of the video input stream to determine the speed of play of the video input stream is based on the fingerprints for the plurality of frames of the video input stream.

12. The system of claim 8, wherein:

the operations further comprise;

identifying a replaceable segment of the video input stream; and

while the video input stream provides the identified replaceable segment, causing the video output stream to switch from the video input stream to an alternative video segment comprising the third video content from the replacement content source; and

the analyzing of the video input stream to determine that at least the portion of the second video content of the frame of the video input stream matches the template is performed while the video output stream provides the alternative video segment.

13. The system of claim 12, wherein the operations further comprise:

based on the at least the portion of the second video content of the frame of the video input stream matching the template, generating a modified frame of the video output stream that is based on both the alternative video segment and the video input stream.

14. The system of claim 13, wherein:

the template defines a region of a frame; and

the generating of the modified frame of the video output stream comprises basing a region of the modified frame of the video output stream that corresponds to the region defined by the template on the video input stream and basing a remaining region of the modified frame of the video output stream on the alternative video segment.

15. A non-transitory machine-readable medium that stores instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

causing a video output stream to be presented on a display of a client device;

while causing the video output stream to be presented on the display;

accessing a video input stream comprising first video content from a content provider and second video content generated by a set-top box upstream of the one or more processors, wherein, in a frame of the video input stream, the first video content corresponds to a first screen portion of the display and the second video content corresponds to a second screen portion of the display, wherein the video output stream comprises third video content from a replacement content source, wherein the third video content differs from the first video content and is accessed separately from the video input stream, wherein the third video content is selected in response to a video identification system receiving a query fingerprint from a query fingerprint generator of the client device and the video identification system identifying a match between the query fingerprint and at least one reference fingerprint generated by a reference fingerprint generator, wherein the query fingerprint is generated based on a frame of the first video content, and wherein the video identification system and the reference fingerprint generator are remote from the client device;

accessing a template corresponding to a screen portion of the display;

analyzing the video input stream to determine that at least a portion of the second video content of the frame of the video input stream matches the template;

based on the at least the portion of the second video content of the frame of the video input stream matching the template, determining that a speed of play of the video input stream has been modified from a default speed; and

responsive to determining that the speed of play of the video input stream has been modified, modifying the video output stream by modifying a speed of play of the video output stream to generate a modified video output stream that mimics the speed of play of the video input stream; and

responsive to modifying the video output stream, causing the modified video output stream to be presented on the display.

16. The non-transitory machine-readable medium of claim 15, wherein the operations further comprise:

based on the at least the portion of the second video content of the frame of the video input stream matching the template, modifying the video output stream to include the second video content generated by the set-top box.

17. The non-transitory machine-readable medium of claim 15, wherein:

the operations further comprise;

analyzing the video input stream to determine the speed of play of the video input stream; and

the modifying of the speed of play of the video output stream includes setting the speed of play of the video output stream to the speed of play of the video input stream.

18. The non-transitory machine-readable medium of claim 17, wherein:

the operations further comprise;

generating fingerprints for a plurality of frames of the video input stream; and

the analyzing of the video input stream to determine the speed of play of the video input stream is based on the fingerprints for the plurality of frames of the video input stream.

19. The non-transitory machine-readable medium of claim 15, wherein:

the operations further comprise;

identifying a replaceable segment of the video input stream; and

while the video input stream provides the identified replaceable segment, causing the video output stream to switch from the video input stream to an alternative video segment comprising the third video content from the replacement content source; and

the analyzing of the video input stream to determine that at least the portion of the second video content of the frame of the video input stream matches the template is performed while the video output stream provides the alternative video segment.

20. The non-transitory machine-readable medium of claim 19, wherein the operations further comprise:

based on the at least the portion of the second video content of the frame of the video input stream matching the template, generating a modified frame of the video output stream that is based on both the alternative video segment and the video input stream.