Descriptive metadata extraction and linkage with editorial content

• US 10,299,012 B2
• Filed: 10/28/2014
• Issued: 05/21/2019
• Est. Priority Date: 10/28/2014
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Accardo, Anthony M., Hedinsson, Skarphedinn S., Smolic, Aljosa Aleksej Andrej, Farre Guiu, Miquel A., Beeler, Thabo D.
• Primary Examiner(s): Alcon, Fernando
• Application Number:
  US14/526,071
• Publication Number:
  US 20160119655A1
• Time in Patent Office: 1,666 Days
• Field of Search: None
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/6581 Reference data, e.g. a movie identifier for ordering a movie or a product identifier in a home shopping application
• CPC Class Codes: H04N 21/8133 specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program
• CPC Class Codes: H04N 21/8358 involving watermark protecting executable software by watermarking G06F21/16; image watermarking in general G06T1/0021; watermarks inserted in still images for transmission purposes H04N1/32144; inserting watermarks during video coding H04N19/467

Abstract

Embodiments provide techniques for distributing supplemental content based on content entities within video content. Embodiments include analyzing video data to identify a known content entity within two or more frames of the video data. For each of the two or more frames, a region of pixels within the respective frame is determined that corresponds to the known content entity. Embodiments further include determining supplemental content corresponding to the known content entity. A watermark is embedded at a first position within the video data, such that the watermark corresponds to an identifier associated with the determined supplemental content. Upon receiving a message specifying the identifier, embodiments include transmitting the supplement content to a client device for output together with the video data.

Claims

1. A computer-implemented method, comprising:

analyzing video data for a first instance of video content in order to identify a character in a scene depicted within two or more of a plurality of frames of the video data, including performing a temporal consistency enhancement operation across the plurality frames of the video data;

determining, for each of the two or more frames, a region of pixels within the respective frame that correspond to the identified character, including performing an object/character segmentation refinement operation across the two or more frames of the video data;

generating a mapping data structure that contains a mapping between a data value pair and the identified character, wherein the data value pair specifies (i) a content identifier that uniquely identifies the first instance of video content and (ii) a timestamp corresponding to a first position within the first instance of video content where the character was identified;

embedding a watermark at the first position within the video data; and

upon receiving, from a remote client device, a message specifying (i) the content identifier that uniquely identifies the first instance of video content (ii) the timestamp corresponding to the first position within the first instance of video content and (iii) a client identifier that uniquely identifies the remote client device, wherein the timestamp was determined by the remote client device based on the remote client device detecting the embedded watermark at the first position within the video data and without requiring user action;

determining the character by accessing the mapping data structure using the timestamp and the content identifier specified within the received message;

determining supplemental content to transmit to the remote client device, by selecting a portion of supplemental content having a predefined correspondence with the character, wherein the selection is made based on a device profile corresponding to the client identifier for the remote client device; and

transmitting the supplemental content corresponding to the character to the remote client device, for output together with the video data.

2. The computer-implemented method of claim 1, wherein embedding a watermark at a first position within the video data further comprises:

embedding an audio watermark within an audio stream of the video data, at a position corresponding to one of the two or more frames.

3. The computer-implemented method of claim 1, wherein analyzing the video data to identify the character within two or more frames of the video data further comprises:

receiving a user input specifying a portion of at least one of the two or more frames of video data that corresponds to the character, and

wherein determining the region of pixels for each of the two or more frames is further based on the received user input.

4. The computer-implemented method of claim 1, wherein the supplemental content comprises at least one of:

(i) image data, (ii) audio data, (iii) video data, (iv) bibliographical data, and (v) textual data.

5. The computer-implemented method of claim 1, wherein the temporal consistency enhancement operation is performed across the plurality of frames of the video data in order to optimize the identification of the character within the two or more frames.

6. The computer-implemented method of claim 1, wherein the object/character segmentation refinement operation is performed across the two or more frames of the video data in order to optimize the region of pixels corresponding to the character with respect to both space and time.

7. The computer-implemented method of claim 1, wherein the character comprises at least one of (i) an actor, (ii) a fictional character, (iii) a non-fictional character, and (iv) a geographic location.

8. The computer-implemented method of claim 1, wherein determining the supplemental content further comprises generating a query adapted to return the supplemental content corresponding to the character, wherein the query is generated based on a client identifier corresponding to a client device from which the message specifying the identifier is received.

15. The computer-implemented method of claim 1, wherein the computer-implemented method is performed by a remote content server operatively connected to the remote client device via a network, the remote content server including a plurality of components comprising a video processing component, a content repository component, and a content selection component;

wherein the character in the scene was identified based on a predefined processing workflow that includes in input stage, an identification stage, a refinement stage, an linkage stage, a watermarking stage, and an output stage.

16. The computer-implemented method of claim 15, wherein the input stage includes a selection of the video content for processing by the video processing component;

wherein the identification stage includes an analysis of the first instance of the video content in order to automatically identify the character in the scene, based on a determination that a region of pixels in the plurality of frames of the scene substantially match a predefined region of pixels in a predefined scene other than the scene, the predefined region of pixels pertaining to the character;

wherein the refinement stage includes the temporal consistency enhancement operation and the object/character segmentation refinement operation;

wherein the temporal consistency enhancement operation is performed in order to increase temporal consistency across the plurality of frames of the scene;

wherein the object/character segmentation refinement operation is performed in order to normalize the region of pixels in order to increase character segmentation across the plurality of frames of the scene.

17. The computer-implemented method of claim 16, wherein the linkage stage includes identification of the supplemental content as being associated with the character and further includes creation of a link between the supplemental content and at least the two or more frames;

wherein the watermarking stage includes an embedment of the watermark into the video data at the first position, wherein the output stage includes an output of the video content containing the embedded watermark and further includes an output of a summary of (i) all characters within the video content, (ii) all scenes within the video content, and (iii) for each character, all scenes within the video content in which the respective character appears.

18. The computer-implemented method of claim 17, wherein the supplemental content to transmit to the remote client device is determined by selecting a portion of supplemental content having a predefined correspondence with the character, wherein the selection is made by, in respective instances:

(i) determining one or more types of content that have been indicated as preferred content types when historically played on the remote client device, wherein a preference is given to the preferred content types in making the selection; and

(ii) determining, based on the client identifier, a plurality of instances of supplemental content that the remote client device is authorized to access, wherein the selection is made from only the plurality of instances of supplemental content, and wherein at least one instance of supplemental content that the remote client device is not authorized to access is not considered as part of the selection.

19. The computer-implemented method of claim 18, wherein analyzing the video data to identify the character within two or more frames of the video data further comprises:

receiving a user input specifying a portion of at least one of the two or more frames of video data that corresponds to the character; and

wherein determining the region of pixels for each of the two or more frames is further based on the received user input;

wherein determining supplemental content corresponding to the character further comprises;

generating a query adapted to return the supplemental content corresponding to the character, wherein query is generated based on the client identifier corresponding to the client device from which the message specifying the identifier is received; and

responsive to submitting the query to a database system for execution, receiving at least an indication of the supplemental content from the database system.

20. The computer-implemented method of claim 19, wherein embedding a watermark at a first position within the video data comprises embedding an audio watermark within an audio stream of the video data, wherein the audio watermark is embedded at a position within the audio stream corresponding to the timestamp;

wherein playback of the video data is output in a playback region of a user interface, wherein outputting the video data for display together with at least an indication of the supplemental content further comprises, in respective instances;

(i) outputting the at least an indication of the supplemental content within the user interface in a region adjacent to the playback region;

(ii) outputting the at least an indication of the supplemental content within the user interface as an overlay on top of the playback region; and

(iii) outputting the at least an indication of the supplemental content as one or more links, such that an instance of supplemental content is loaded within the user interface responsive to the corresponding one of the one or more links being selected;

wherein the message specifying the timestamp corresponding to the first position within the video data further specifies a client device identifier, wherein the remote content server is configured to select the supplemental content based at least in part on the client device identifier;

wherein the supplemental content comprises (i) image data, (ii) audio data, (iii) video data, (iv) bibliographical data, and (v) textual data;

wherein the character comprises, in respective instances, an actor and a cartoon character.

9. A non-transitory computer-readable medium containing program code that, when executed by operation of one or more computer processors, performs an operation comprising:

analyzing video data for a first instance of video content to identify a character in a scene depicted within two or more of a plurality of frames of the video data, including performing a temporal consistency enhancement operation across the plurality frames of the video data;

determining, for each of the two or more frames, a region of pixels within the respective frame that correspond to the identified character, including performing an object/character segmentation refinement operation across the two or more frames of the video data;

generating a mapping data structure that contains a mapping between a data value pair and the identified character, wherein the data value pair specifies (i) a content identifier that uniquely identifies the first instance of video content and (ii) a timestamp corresponding to a first position within the first instance of video content where the character was identified;

embedding a watermark at the first position within the video data; and

upon receiving, from a remote client device, a message specifying (i) a content identifier that uniquely identifies the first instance of video content (ii) the timestamp corresponding to the first position within the first instance of video content and (iii) a client identifier that uniquely identifies the remote client device, wherein the timestamp was determined by the remote client device based on the remote client device detecting the embedded watermark at the first position within the video data and without requiring user action;

determining the character by accessing the mapping data structure using the timestamp and the content identifier specified within the received message;

determining supplemental content to transmit to the remote client device, by selecting a portion of supplemental content having a predefined association correspondence with the character, wherein the selection is made based on a device profile corresponding to the client identifier for the remote client device; and

transmitting the supplemental content corresponding to the character to the remote client device, for output together with the video data.

10. The non-transitory computer-readable medium of claim 9, wherein embedding a watermark at a first position within the video data further comprises:

embedding an audio watermark within an audio stream of the video data, at a position corresponding to one of the two or more frames.

11. The non-transitory computer-readable medium of claim 9, wherein analyzing the video data to identify the character within two or more frames of the video data further comprises:

receiving a user input specifying a portion of at least one of the two or more frames of video data that corresponds to the character, and

wherein determining the region of pixels for each of the two or more frames is further based on the received user input.

12. The non-transitory computer-readable medium of claim 9, wherein the supplemental content comprises at least one of:

(i) image data, (ii) audio data, (iii) video data, (iv) bibliographical data, and (v) textual data.

13. The non-transitory computer-readable medium of claim 9, wherein the temporal consistency enhancement operation is performed across the plurality of frames of the video data in order to optimize the identification of the character within the two or more frames.

14. A system, comprising:

one or more computer processors; and

a memory containing a program that, when executed by the one or more computer processors, performs an operation comprising;

analyzing video data for a first instance of video content to identify a character in a scene depicted within two or more of a plurality of frames of the video data, including performing a temporal consistency enhancement operation across the plurality frames of the video data;

determining, for each of the two or more frames, a region of pixels within the respective frame that correspond to the identified character, including performing an object/character segmentation refinement operation across the two or more frames of the video data;

generating a mapping data structure that contains a mapping between a data value pair and the identified character, wherein the data value pair specifies (i) a content identifier that uniquely identifies the first instance of video content and (ii) a timestamp corresponding to a first position within the first instance of video content where the character was identified;

embedding a watermark at the first position within the video data; and

upon receiving, from a remote client device, a message specifying (i) a content identifier that uniquely identifies the first instance of video content (ii) the timestamp corresponding to the first position within the first instance of video content and (iii) a client identifier that uniquely identifies the remote client device, wherein the timestamp was determined by the remote client device based on the remote client device detecting the embedded watermark at the first position within the video data and without requiring user action;

determining the character by accessing the mapping data structure using the timestamp and the content identifier specified within the received message;

determining supplemental content to transmit to the remote client device, by selecting a portion of supplemental content having a predefined correspondence with the character, wherein the selection is made based on a device profile corresponding to the client identifier for the remote client device; and

transmitting the supplemental content corresponding to the character to the remote client device, for output together with the video data.