Apparatus, systems, and methods for integrating digital media content

• US 10,522,186 B2
• Filed: 07/30/2018
• Issued: 12/31/2019
• Est. Priority Date: 07/28/2017
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Inc.
• Inventors: Attorre, Brunno Fidel Maciel, Xue, Xiaozhen, Marzban, Shabbir, Thien, Nicolas Huynh, Marino, William L.
• Primary Examiner(s): Chevalier, Robert
• Application Number:
  US16/049,690
• Publication Number:
  US 20190035431A1
• Time in Patent Office: 519 Days
• Field of Search: 386278, 386281, 386290, 386326, 386239, 386241
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: G10L 21/007 characterised by the process used
• CPC Class Codes: G10L 25/09 the extracted parameters being zero crossing rates
• CPC Class Codes: G10L 25/21 the extracted parameters being power information
• CPC Class Codes: G10L 25/24 the extracted parameters being the cepstrum
• CPC Class Codes: G10L 25/30 using neural networks
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: G11B 27/3081 used signal is a video-frame or a video-field (P.I.P)
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/23424 involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02

Abstract

Disclosed herein are techniques for digital content integration. A computer-implemented method includes receiving a target digital content item that includes a plurality of frames, identifying a set of candidate host frames for inserting source digital content items from the plurality of frames based on one or more attributes of the target digital content item, determining a candidate score for each respective candidate host frame of the candidate host frames, and generating host time defining data including identifications and the candidate scores of the candidate host frames, where the candidate score indicates a degree of transition of the target digital content item at the candidate host frame. One or more candidate host frames are then selected based on the candidate scores for inserting one or more source digital content items into the target digital content item.

Claims

1. A computer-implemented method, comprising, by one or more processing devices:

receiving a target digital content item, the target digital content item comprising a plurality of frames;

identifying, from the plurality of frames and based on one or more attributes of the target digital content item, a set of candidate host frames for inserting source digital content items;

determining a candidate score for each respective candidate host frame of the candidate host frames, wherein the candidate score indicates a degree of transition of the target digital content item at the candidate host frame; and

generating host time defining data including identifications and the candidate scores of the candidate host frames.

2. The method of claim 1, wherein the one or more attributes of the target digital content item include at least one of metadata associated with the target digital content item, pixel values or objects in visual content of the target digital content item, or a transcript, amplitude, zero crossing rate, energy, entropy of energy, spectral centroids, spectral spread, spectral entropy, spectral flux, spectral roll-off, MFCCs, chroma vector, or chroma deviation of audio content of the target digital content item.

3. The method of claim 1, further comprising:

ranking the candidate host frames identified in the host time defining data based on their respective candidate scores in the host time defining data;

selecting, based on the ranking, one or more candidate host frames for inserting one or more source digital content items into the target digital content item; and

inserting the one or more source digital content items into the target digital content item at the one or more candidate host frames.

4. The method of claim 1, wherein:

identifying the set of candidate host frames comprises, for each pair of consecutive frames in the plurality of frames;

determining a level of change of the one or more attributes between the pair of consecutive frames; and

selecting a later frame in the pair of consecutive frames as a candidate host frame based on the level of change being greater than a threshold value; and

determining the candidate score for each candidate host frame comprises determining the candidate score for the candidate host frame based on the level of change of the one or more attributes between the candidate host frame and a frame immediately before the candidate host frame.

5. The method of claim 1, wherein:

identifying the set of candidate host frames comprises, for each frame in an analyzing interval that includes a set of frames;

determining an attribute of the frame;

determining, as a threshold, a variance of the attribute among the frame and frames before the frame in the analyzing interval; and

identifying the frame as a candidate host frame based on the attribute of the frame being greater than the threshold.

6. The method of claim 1, wherein:

the one or more attributes comprise pixel values of the target digital content;

identifying the set of candidate host frames comprises, for each frame in the plurality of frames;

determining a level of change of the pixel values between the frame and a white frame or between the frame and a black frame; and

selecting the frame as a candidate host frame based on the level of change being greater than a threshold value; and

determining the candidate score for each candidate host frame comprises determining the candidate score for the candidate host frame based on the level of change of the pixel values between the frame and the white frame or between the frame and the black frame.

7. The method of claim 1, further comprising:

extracting, using one or more neural networks, a first set of one or more feature vectors from a frame in an analyzing interval before a candidate host frame;

combining the first set of one or more feature vectors into a first content vector;

extracting, using the one or more neural networks, a second set of one or more feature vectors from a frame in an analyzing interval after the candidate host frame;

combining the second set of one or more feature vectors into a second content vector;

determining a distance between the first content vector and the second content vector;

increasing the candidate score of the candidate host frame based on the distance being greater than a first threshold value or removing the candidate host frame from the set of candidate host frames based on the distance being less than a second threshold value.

8. The method of claim 1, wherein identifying the set of candidate host frames comprises, for each frame in the plurality of frames:

recognizing, using a neural network, objects in the frame;

comparing the objects recognized in the frame with objects recognized in an earlier frame;

identifying the frame as a candidate host frame based on the objects recognized in the frame different from the objects recognized in the earlier frame.

9. The method of claim 1, further comprising:

determining, for each of the candidate host frames, a difference of the one or more attributes between a frame immediately preceding the candidate host frame and a frame immediately following the candidate host frame; and

reducing the candidate score of the candidate host frame or removing the candidate host frame from the set of candidate host frames based on the difference being less than a threshold.

10. The method of claim 1, further comprising, for each candidate host frame in the candidate host frames:

determining, for a first analyzing interval immediately preceding the candidate host frame, a first motion vector for frames in the first analyzing interval using an optical flow method;

determining, for a second analyzing interval immediately following the candidate host frame, a second motion vector for frames in the second analyzing interval using the optical flow method;

determining a difference between the first motion vector and the second motion vector; and

decreasing the candidate score of the candidate host frame or removing the candidate host frame from the set of candidate host frames based on the difference being less than a threshold.

11. The method of claim 1, wherein:

the target digital content item comprises audio data;

identifying the set of candidate host frames comprises;

classifying frames or segments of the audio data, the classifying including determining a degree of confidence of the classification for each respective frame or segment of the audio data; and

identifying, as a candidate host frame, a frame immediately before a change of classification among the frames or segments of the audio data; and

determining the candidate score for the candidate host frame comprises determining the candidate score for the candidate host frame based on the degree of confidence associated with the candidate host frame.

12. The method of claim 1, wherein:

the target digital content item comprises audio data;

identifying the set of candidate host frames comprises;

classifying segments of the audio data;

identifying two or more segments of the audio data classified as human speech based on the classification; and

determining a change of speakers among the two or more segments based on an amplitude or Mel-Frequency Cepstral Coefficients (MFCCs) associated with each of the two or more segments, wherein determining the change of speakers includes determining a degree of confidence of determining the change of speakers; and

identifying, as a candidate host frame, a frame immediately before the change of speakers; and

determining the candidate score for the candidate host frame comprises determining the candidate score for the candidate host frame based on the degree of confidence of determining the change of speakers.

13. The method of claim 12, wherein identifying the set of candidate host frames further comprises,

determining that a same speaker is associated with two consecutive segments based on the amplitude or the Mel-Frequency Cepstral Coefficients (MFCCs) associated with each of the two consecutive segments;

determining a level of valence, arousal, or other emotional intensity for each of two consecutive segments;

identifying, as a candidate host frame, a frame immediately before a second segment of the two consecutive segments based on a difference between the levels of valence, arousal, or other emotional intensity for the two consecutive segments being greater than a threshold value; and

determining the candidate score for the candidate host frame comprises determining the candidate score for the candidate host frame based on the difference between the levels of valence, arousal, or other emotional intensity for the two consecutive segments.

14. The method of claim 1, wherein:

the target digital content item comprises audio data;

identifying the set of candidate host frames comprises;

converting the audio data into text using voice recognition; and

identifying, using natural language processing, the candidate host frames based on a pause or transition in the text.

15. The method of claim 1, wherein:

the target digital content item comprises visual data and audio data;

the method further comprises parsing the target digital content item into the visual data and audio data;

identifying the set of candidate host frames comprises;

identifying a preliminary set of candidate host frames based on one or more attributes associated with the visual data; and

identifying, from the preliminary set of candidate host frames, the set of candidate host frames based on one or more attributes of the audio data; and

determining the candidate score for the candidate host frame comprises determining the candidate score for the candidate host frame based on the one or more attributes associated with the visual data and the one or more attributes of the audio data.

16. The method of claim 1, further comprising adjusting the candidate scores of the set of candidate host frames based on:

a popularity of each portion of the target digital content item;

or

a proximity of each candidate host frame to a start of the target digital content item.

17. The method of claim 1, wherein identifying the set of candidate host frames comprises:

grouping the plurality of frames into scenes based on a level of camera motion in each frame, the camera motion including a translation or a rotation.

18. The method of claim 1, wherein the host time defining data further comprises:

a source digital content transformation object associated with each respective candidate host frame,

wherein the source digital content transformation object describes a transformation of an source digital content item before inserting the source digital content item into the target digital content item at the candidate host frame.

19. A system comprising:

a processing device; and

a non-transitory computer-readable medium communicatively coupled to the processing device, wherein the processing device is configured to execute program code stored in the non-transitory computer-readable medium and thereby perform operations comprising;

receiving a target digital content item, the target digital content item comprising a plurality of frames;

identifying, from the plurality of frames and based on one or more attributes of the target digital content item, a set of candidate host frames for inserting source digital content items;

determining a candidate score for each respective candidate host frame of the candidate host frames, wherein the candidate score indicates a degree of transition of the target digital content item at the candidate host frame; and

generating host time defining data including identifications and the candidate scores of the candidate host frames.

20. A system for integrating digital media content, the system comprising:

means for receiving a target digital content item, the target digital content item comprising a plurality of frames;

means for identifying, from the plurality of frames and based on one or more attributes of the target digital content item, a set of candidate host frames for inserting source digital content items;

means for determining a candidate score for each respective candidate host frame of the candidate host frames, wherein the candidate score indicates a degree of transition of the target digital content item at the candidate host frame; and

means for generating host time defining data including identifications and the candidate scores of the candidate host frames.