Descriptive metadata extraction and linkage with editorial content

• US 10,264,329 B2
• Filed: 10/28/2014
• Issued: 04/16/2019
• Est. Priority Date: 10/28/2014
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Accardo, Anthony M., Hedinsson, Skarphedinn S., Smolic, Aljosa Aleksej Andrej, Farre Guiu, Miquel A., Beeler, Thabo D.
• Primary Examiner(s): Alcon, Fernando
• Application Number:
  US14/526,096
• Publication Number:
  US 20160119691A1
• Time in Patent Office: 1,631 Days
• Field of Search: None
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/234 Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content
• CPC Class Codes: H04N 21/4725 using interactive regions of the image, e.g. hot spots details of information retrieval from the Internet by using URLs G06F16/955; processing chained hypermedia data for information retrieval G06F16/94
• CPC Class Codes: H04N 21/6581 Reference data, e.g. a movie identifier for ordering a movie or a product identifier in a home shopping application
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program
• CPC Class Codes: H04N 21/8352 involving content or source identification data, e.g. Unique Material Identifier [UMID]
• CPC Class Codes: H04N 21/8358 involving watermark protecting executable software by watermarking G06F21/16; image watermarking in general G06T1/0021; watermarks inserted in still images for transmission purposes H04N1/32144; inserting watermarks during video coding H04N19/467
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73

Abstract

Embodiments provide techniques for distributing supplemental content based on content entities within video content. Embodiments include receiving video data containing an embedded watermark at a first position within the video data. The embedded watermark is detected at the first position within the video data. Embodiments also include transmitting, to a remote content server, a message specifying a time stamp corresponding to the first position within the video data. In response to transmitting the message, supplemental content corresponding to a content entity depicted within the video content at the first position within the video content is received from the remote content server. Embodiments also include outputting the video data for display together with at least an indication of the supplemental content.

Claims

1. A computer-implemented method, comprising:

receiving, at a client device, video data for a first instance of video content and containing an indication that supplemental content is available for a character in a scene depicted within frames of the video content at a first position within the video data, the indication comprising a watermark embedded at the first position within the video data, wherein the character in the scene was automatically detected via at least one of a temporal consistency enhancement operation and an object/character segmentation refinement operation;

responsive to the client device detecting the embedded watermark at the first position within the video data, dynamically determining, during playback of the video data and without requiring user action, a timestamp that identifies the first position within the video data where the embedded watermark was detected;

transmitting, to a remote content server, a message specifying (i) a content identifier uniquely identifying the first instance of video content, (ii) the timestamp that identifies the first position within the video data at which the embedded watermark was detected and (iii) a client identifier that uniquely identifies the client device;

in response to transmitting the message, receiving, from the remote content server, supplemental content corresponding to the character, wherein the remote content server is configured to;

determine the character by accessing a mapping data structure that contains a mapping between a data value pair and the character, wherein the data value pair specifies (i) the content identifier that uniquely identifies the first instance of video content and (ii) the timestamp corresponding to the first position within the first instance of video content;

determine the supplemental content to transmit to the client device, by selecting a portion of supplemental content having a predefined association with the character, wherein the selection is made based on a device profile corresponding to the client identifier for the client device; and

transmit the supplemental content to the client device; and

outputting the video data for display together with at least an indication of the supplemental content.

2. The computer-implemented method of claim 1, wherein the embedded watermark comprises an audio watermark within an audio stream of the video data, and wherein the audio watermark is embedded at a position within the audio stream corresponding to the timestamp.

3. The computer-implemented method of claim 1, wherein playback of the video data is output in a playback region of a user interface, and wherein outputting the video data for display together with at least an indication of the supplemental content further comprises:

outputting the at least an indication of the supplemental content within the user interface in a region adjacent to the playback region.

4. The computer-implemented method of claim 1, wherein playback of the video data is output in a playback region of a user interface, and wherein outputting the video data for display together with at least an indication of the supplemental content further comprises:

outputting the at least an indication of the supplemental content within the user interface as an overlay on top of the playback region.

5. The computer-implemented method of claim 1, wherein outputting the video data for display together with at least an indication of the supplemental content further comprises:

outputting the at least an indication of the supplemental content as one or more links, such that an instance of supplemental content is loaded within a user interface responsive to a corresponding one of the one or more links being selected.

15. The computer-implemented method of claim 1, wherein the character in the scene was automatically detected by the remote content server, wherein the indication was automatically generated by the remote content server;

wherein the remote content server comprises a plurality of components including a video processing component, a content repository component, and a content selection component;

wherein the character in the scene was detected and the indication generated based on a predefined processing workflow that includes in input stage, an identification stage, a refinement stage, an linkage stage, a watermarking stage, and an output stage.

16. The computer-implemented method of claim 15, wherein the input stage includes a selection of the video content for processing by the video processing component;

wherein the identification stage includes an analysis of the first instance of the video content in order to automatically identify the character in the scene, based on a determination that a region of pixels in the frames of the scene substantially match a predefined region of pixels in a predefined scene other than the scene, the predefined region of pixels pertaining to the character.

17. The computer-implemented method of claim 16, wherein the refinement stage includes the temporal consistency enhancement operation and the object/character segmentation refinement operation;

wherein the temporal consistency enhancement operation is performed in order to increase temporal consistency across the frames of the scene;

wherein the object/character segmentation refinement operation is performed in order to normalize the region of pixels in order to increase character segmentation across the frames of the scene;

wherein the linkage stage includes identification of the supplemental content as being associated with the character and further includes creation of a link between the supplemental content and the frames;

wherein the watermarking stage includes an embedment of the watermark into the video data at the first position, wherein the output stage includes an output of the video content containing the embedded watermark and further includes an output of a summary of (i) all characters within the video content, (ii) all scenes within the video content, and (iii) for each character, all scenes within the video content in which the respective character appears.

18. The computer-implemented method of claim 17, wherein the embedded watermark comprises an audio watermark within an audio stream of the video data, wherein the audio watermark is embedded at a position within the audio stream corresponding to the timestamp;

wherein playback of the video data is output in a playback region of a user interface, wherein outputting the video data for display together with at least an indication of the supplemental content further comprises, in respective instances;

(i) outputting the at least an indication of the supplemental content within the user interface in a region adjacent to the playback region;

(ii) outputting the at least an indication of the supplemental content within the user interface as an overlay on top of the playback region; and

(iii) outputting the at least an indication of the supplemental content as one or more links, such that an instance of supplemental content is loaded within the user interface responsive to a corresponding one of the one or more links being selected.

19. The computer-implemented method of claim 18, wherein the video data is received from the remote content server, which generates the video containing the embedded watermark at the first position within the video data by automatically:

analyzing the video data in order to identify the character within two or more frames of the video data;

performing the temporal consistency enhancement operation to increase temporal consistency across the frames of the scene in order to optimize the identification of the character within the two or more frames;

determining, for each of the two or more frames, the region of pixels within the respective frame that correspond to the character;

performing the object/character segmentation refinement operation to normalize the region of pixels in order to increase character segmentation across the frames of the scene and optimize the region of pixels corresponding to the character with respect to both space and time;

determining the supplemental content corresponding to the character; and

embedding the watermark at the first position within the video data, such that the watermark corresponds to an identifier associated with the determined supplemental content.

20. The computer-implemented method of claim 19, wherein the message specifying the timestamp corresponding to the first position within the video data further specifies a client device identifier, wherein the remote content server is configured to select the supplemental content based at least in part on the client device identifier;

wherein the supplemental content comprises (i) image data, (ii) audio data, (iii) video data, (iv) bibliographical data, and (v) textual data;

wherein the character comprises, in respective instances, an actor and a cartoon character;

wherein the supplemental content to transmit to the client device is determined by selecting a portion of supplemental content having a predefined association with the character, wherein the selection is made by, in respective instances;

(i) determining one or more types of content that have been indicated as preferred content types when historically played on the client device, wherein a preference is given to the preferred content types in making the selection; and

(ii) determining, based on the client identifier, a plurality of instances of supplemental content that the client device is authorized to access, wherein the selection is made from only the plurality of instances of supplemental content, and wherein at least one instance of supplemental content that the client device is not authorized to access is not considered as part of the selection.

6. A non-transitory computer-readable medium containing program code that, when executed by operation of one or more computer processors, performs an operation comprising:

receiving, at a client device, video data for a first instance of video content and containing an indication that supplemental content is available for a character in a scene depicted within frames of the video content at a first position within the video data, the indication comprising a watermark embedded at the first position within the video data, wherein the character in the scene was automatically detected via at least one of a temporal consistency enhancement operation and an object/character segmentation refinement operation;

responsive to the client device detecting the embedded watermark at the first position within the video data, dynamically determining, during playback of the video data and without requiring user action, a timestamp that identifies the first position within the video data where the embedded watermark was detected;

transmitting, to a remote content server, a message specifying (i) a content identifier uniquely identifying the first instance of video content, (ii) the timestamp that identifies the first position within the video data at which the embedded watermark was detected and (iii) a client identifier that uniquely identifies the client device;

in response to transmitting the message, receiving, from the remote content server, supplemental content corresponding to the character, wherein the remote content server is configured to;

determine the character by accessing a mapping data structure that contains a mapping between a data value pair and the character, wherein the data value pair specifies (i) the content identifier that uniquely identifies the first instance of video content and (ii) the timestamp corresponding to the first position within the first instance of video content;

determine the supplemental content to transmit to the client device, by selecting a portion of supplemental content having a predefined association with the character, wherein the selection is made based on a device profile corresponding to the client identifier for the client device; and

transmit the supplemental content to the client device; and

outputting the video data for display together with at least an indication of the supplemental content.

7. The non-transitory computer-readable medium of claim 6, wherein the embedded watermark comprises an audio watermark within an audio stream of the video data, and wherein the audio watermark is embedded at a position within the audio stream corresponding to the timestamp.

8. The non-transitory computer-readable medium of claim 6, wherein playback of the video data is output in a playback region of a user interface, and wherein outputting the video data for display together with at least an indication of the supplemental content further comprises:

outputting the at least an indication of the supplemental content within the user interface in a region adjacent to the playback region.

9. The non-transitory computer-readable medium of claim 6, wherein playback of the video data is output in a playback region of a user interface, and wherein outputting the video data for display together with at least an indication of the supplemental content further comprises:

outputting the at least an indication of the supplemental content within the user interface as an overlay on top of the playback region.

10. The non-transitory computer-readable medium of claim 6, wherein outputting the video data for display together with at least an indication of the supplemental content further comprises:

outputting the at least an indication of the supplemental content as one or more links, such that an instance of supplemental content is loaded within a user interface responsive to a corresponding one of the one or more links being selected.

11. A system, comprising:

a computer processor; and

a memory containing a program that, when executed by the computer processor, performs an operation comprising;

receiving, at a client device, video data for a first instance of video content and containing an indication that supplemental content is available for a character in a scene depicted within frames of the video content at a first position within the video data, the indication comprising a watermark embedded at the first position within the video data, wherein the character in the scene was automatically detected via at least one of a temporal consistency enhancement operation and an object/character segmentation refinement operation;

responsive to the client device detecting the embedded watermark at the first position within the video data, dynamically determining, during playback of the video data and without requiring user action, a timestamp that identifies the first position within the video data where the embedded watermark was detected;

transmitting, to a remote content server, a message specifying (i) a content identifier uniquely identifying the first instance of video content, (ii) the timestamp that identifies the first position within the video data at which the embedded watermark was detected and (iii) a client identifier that uniquely identifies the client device;

in response to transmitting the message, receiving, from the remote content server, supplemental content corresponding to the character, wherein the remote content server is configured to;

determine the character by accessing a mapping data structure that contains a mapping between a data value pair and the character, wherein the data value pair specifies (i) the content identifier that uniquely identifies the first instance of video content and (ii) the timestamp corresponding to the first position within the first instance of video content;

determine the supplemental content to transmit to the client device, by selecting a portion of supplemental content having a predefined association with the character, wherein the selection is made based on a device profile corresponding to the client identifier for the client device; and

transmit the supplemental content to the client device; and

outputting the video data for display together with at least an indication of the supplemental content.

12. The system of claim 11, wherein the embedded watermark comprises an audio watermark within an audio stream of the video data, and wherein the audio watermark is embedded at a position within the audio stream corresponding to the timestamp.

13. The system of claim 11, wherein playback of the video data is output in a playback region of a user interface, and wherein outputting the video data for display together with at least an indication of the supplemental content further comprises:

outputting the at least an indication of the supplemental content within the user interface in a region adjacent to the playback region.

14. The system of claim 11, wherein playback of the video data is output in a playback region of a user interface, and wherein outputting the video data for display together with at least an indication of the supplemental content further comprises:

outputting the at least an indication of the supplemental content within the user interface as an overlay on top of the playback region.