Placing unobtrusive overlays in video content

• US 9,467,750 B2
• Filed: 05/31/2013
• Issued: 10/11/2016
• Est. Priority Date: 05/31/2013
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Banica, Dan, Cucu, Andrian, Chirita, Paul-Alexandru
• Primary Examiner(s): SHEPARD, JUSTIN E
• Application Number:
  US13/906,838
• Publication Number:
  US 20140359656A1
• Time in Patent Office: 1,229 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: H04N 21/23 Processing of content or additional data; Elementary server operations; Server middleware
• CPC Class Codes: H04N 21/234 Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/2547 Third Party Billing, e.g. billing of advertiser
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/44 Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
• CPC Class Codes: H04N 21/44016 involving splicing one content stream with another content stream, e.g. for substituting a video clip
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/85 Assembly of content; Generation of multimedia applications
• CPC Class Codes: H04N 21/8545 for generating interactive applications

Abstract

Methods and systems for placing an overlay in video content are provided. A method receives video content and input indicating an overlay to be placed in the video. The method determines, based on overlay and video properties, locations where the overlay can be placed. The method presents suggested locations for the overlay and receives a selection of a suggested location. The overlay is placed in the selected location. A system includes memory with instructions for inserting an overlay into video content. The system receives an indication of an overlay to be placed in the video, performs attention modeling on the video to identify zones likely to be of interest to a viewer. The system presents locations within the identified zones where the overlay can be inserted and receives a selection of a location. The system inserts the overlay into the selected location and renders the video with the inserted overlay.

Claims

1. A computer-implemented method for placing an overlay in video content, the method comprising:

receiving, at an overlay server, video content;

receiving, at the overlay server, an indication of an overlay to be placed in the received video content, the overlay being an item of electronic content including one or more of text, image, and multimedia content;

generating, by an attention modeling module of the overlay server, saliency maps indicating relative salience of regions within frames of the video content based upon feature values in a region differing from other feature values in regions adjacent to the region;

computing an auxiliary matrix by cumulating saliency values from the saliency maps;

identifying, based on the saliency maps and the properties of the overlay, a plurality of unobtrusive locations where the overlay can be placed within the frames of the video content, wherein the unobtrusive locations are in lower salience regions;

calculating unobtrusiveness values of the plurality of unobtrusive locations, the calculating being based at least in part on the auxiliary matrix;

determining, by the overlay server, based on properties of the overlay, the unobtrusiveness values, and properties of the video content, an optimal location from amongst the plurality of unobtrusive locations where the overlay can be placed within the video content; and

placing the overlay in the optimal location.

2. The method of claim 1, further comprising, prior to the placing:

presenting the plurality of unobtrusive locations in a user interface on a display of the computing device as suggested locations for the overlay;

receiving a selection of one of the suggested locations;

identifying, based at least in part on the selected location and properties of the overlay, one or more overlay transformations; and

applying, by the overlay server, the identified one or more overlay transformations to the overlay to produce a transformed overlay,

wherein the placing comprises placing the transformed overlay in the selected location.

3. The method of claim 2, wherein the one or more overlay transformations comprise one or more of a spatial transformation, a color transformation, and a translucency transformation.

4. The method of claim 1, further comprising:

storing, in a data store, the video content with the placed overlay as edited video content.

5. The method of claim 1, wherein the received video content is received as a complete video.

6. The method of claim 1, wherein the received video content is a video stream.

7. The method of claim 1 wherein:

the received video content comprises three dimensional (3D) video content;

the properties of the overlay include a depth dimension of the overlay; and

the depth dimension is in terms of pixels.

8. The method of claim 1, wherein the properties of the overlay include:

size;

a color palette or scheme; and

duration.

9. The method of claim 8 wherein the size property of the overlay includes dimensions of the overlay in pixels, and wherein the duration includes an amount of frames, a number of seconds, or fractions thereof.

10. The method of claim 1, wherein the overlay is an advertisement, and wherein the properties of the advertisement include:

monetary cost per placement;

duration; and

product or service category.

11. The method of claim 1, wherein the determining comprises calculating a salience cost for each of the plurality of unobtrusive locations, the salience cost being based at least in part on the properties of the overlay.

12. The method of claim 1, wherein the unobtrusiveness of locations in the video content is based in part on the saliency maps generated by the attention modeling module for the frames of the video content.

13. The method of claim 1, wherein:

higher salience regions have been predicted by the attention modeling module to be more likely to be considered interesting by a viewer of the video content as compared to lower salience regions; and

the feature values include one or more of;

color contrast;

pixels on a horizon line in the region;

intensity;

brightness; and

features of an object in the region, wherein the features of the object comprise a local orientation of the object, a presence of text in the object, and a presence of a face in the object.

14. The method of claim 1,

wherein calculating the unobtrusiveness values comprises using the auxiliary matrix to rapidly calculate the unobtrusiveness values of the plurality of unobtrusive locations within the frames of the video content.

15. The method of claim 1, further comprising identifying one or more overlay transformations including at least a spatial transformation, wherein the identifying one or more overlay transformations comprises scanning the frames of the video content to detect substantially straight lines proximate to lower salience regions, and wherein the spatial transformation includes reshaping the overlay so that one or more sides of the overlay are substantially parallel with at least one of the lines.

16. A system comprising:

an input device;

a display device;

a processor; and

a memory having instructions stored thereon that, if executed by the processor, cause the processor to perform operations for inserting an overlay into video content, the overlay being an item of electronic content including one or more of text, image, and multimedia content, the operations comprising;

receiving an indication of an overlay to be placed in the video content;

performing attention modeling on frames of the video content to identify zones in the video content and to generate saliency maps having saliency values indicating relative salience of regions within the frames of the video content based upon feature values in a region differing from feature values in surrounding regions;

determining, based on the saliency maps and the properties of the overlay, a plurality of unobtrusive locations within the identified zones at which the overlay can be inserted without overlapping with one or more objects, wherein the unobtrusive locations are in lower salience regions;

identifying one or more overlay transformations including at least a spatial transformation, the identifying comprising scanning the frames of the video content to detect substantially straight lines proximate to lower salience regions, wherein the spatial transformation includes reshaping the overlay so that one or more sides of the overlay are substantially parallel with at least one of the lines;

identifying, based on the attention modeling, an optimal location from amongst the plurality of unobtrusive locations;

inserting the overlay into the optimal location; and

rendering, on the display device, the video content with the inserted overlay.

17. The system of claim 16, the operations further comprising, prior to the rendering:

presenting, in a user interface on the display device, at least some of the plurality of unobtrusive locations within the identified zones where the overlay can be inserted into;

receiving, via the input device, a selection of a location within one of the identified zones; and

applying the one or more overlay transformations to the overlay, wherein the one or more overlay transformations further include one or more of;

a color transformation to change an edge of the overlay to substantially match colors of pixels of the video content adjacent to the selected location;

a spatial transformation to reshape or resize the overlay to substantially conform to an object in the video content proximate to the selected location; and

a translucency transformation to gradually increase translucency of the overlay relative to distance from a center of the overlay to edges of the overlay so that the edges of the overlay are substantially transparent.

18. The system of claim 17, the operations further comprising:

calculating respective salience costs for inserting the overlay into each of the presented locations;

receiving, via the input device, a request to move the overlay from the optimal location to another one of the plurality of unobtrusive locations;

inserting the overlay into the another one of the plurality of unobtrusive locations; and

rendering, on the display device, the video content with the inserted overlay.

19. The system of claim 16, wherein the performing the attention modeling comprises identifying visually unobtrusive locations within the identified zones where the overlay can be inserted into, the visually unobtrusive locations being locations not overlapping with relatively highly salient regions within the frames of the video content.

20. The system of claim 16, the operations further comprising:

receiving, via the input device, a request to resize the overlay from an initial size to a selected size;

resizing the overlay from the initial size to the selected size to produce a resized overlay;

inserting the resized overlay into the optimal location; and

rendering, on the display device, the video content with the inserted, resized overlay.

21. A computer readable storage medium having executable instructions stored thereon, that, if executed by a computing device, cause the computing device to perform operations for placing an overlay in video content, the instructions comprising:

instructions for receiving video content;

instructions for receiving an indication of an overlay to be placed in the received video content, the overlay being an item of electronic content including one or more of text, image, and multimedia content;

instructions for generating saliency maps indicating relative salience of regions within frames of the video content based upon feature values in a region differing from other feature values in regions adjacent to the region;

instructions for computing an auxiliary matrix by cumulating saliency values from the saliency maps;

instructions for identifying, based on the saliency maps and the properties of the overlay, a plurality of unobtrusive locations where the overlay can be placed within the frames of the video content, wherein the unobtrusive locations are in lower salience regions;

instructions for calculating unobtrusiveness values of the plurality of unobtrusive locations, the calculating being based at least in part on the auxiliary matrix;

instructions for determining, based on properties of the overlay, the unobtrusiveness values, and properties of the video content, an optimal location from amongst the plurality of unobtrusive locations where the overlay can be placed within the video content; and

instructions for placing the overlay in the optimal location.