System and method for identifying and altering images in a digital video

• US 9,711,182 B2
• Filed: 06/07/2012
• Issued: 07/18/2017
• Est. Priority Date: 06/07/2011
• Status: Active Grant

• Current Assignee: In Situ Media Corporation
• Original Assignee: In Situ Media Corporation
• Inventors: Dharssi, Fatehali T., McKay, Ashley
• Primary Examiner(s): Dang, Hung
• Assistant Examiner(s): Park, Sunghyoun
• Application Number:
  US14/124,588
• Publication Number:
  US 20140363143A1
• Time in Patent Office: 1,867 Days
• CPC Class Codes: G06Q 30/0254 based on statistics
• CPC Class Codes: G06V 10/46 Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features colour feature extraction G06V10/56
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/768 using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G06V 20/68 Food, e.g. fruit or vegetables
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: H04N 21/23412 for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/854 Content authoring
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A method of identifying distinctive objects or images within a digital video comprises the use of one or more pixel (pel) based “pattern” or “feature” recognition protocols.

Claims

1. A method to automate post-production content modification, the method comprising:

performing image recognition on a plurality of frames of post-production video content to identify a plurality of objects in a scene by

identifying a first pixel of a selected color relating to a distinctive image for which the frames of post-production video content are being searched,

scanning a number of adjacent pixels, which are adjacent to the first pixel, to determine a shape of an image in each of at least two directions,

determining at least two outermost endpoints of the selected color relative to the first pixel in each of the at least two directions;

for each of the outermost end points, determining a respective distance between the first pixel and the respective outermost end point,

determining a ratio of the distance between the outermost end points; and

comparing the determined ratio to a known data for the distinctive image to determine a location of the distinctive image in the frames of the post-production video content,

calculating a probability that the distinctive image is present in at least some of the frames of post-production video content based in the ratio and the comparison, and

modifying only those frames of the post-production video content where the calculated probability exceeds a confidence level;

determining a context of the scene which at least some of the frames of video represent based at least in part on the objects recognized the frames of the post-production video content;

determining at least one post-production content modification to be made to at least some of the frames of the post-production video content based at least in part on the determined context of the scene which the frames of post-production video content represent; and

modifying at least some of the at least one of the frames of the postproduction video content based at least in part on determined at least one post-production content modification.

2. A method to automate post-production content modification, the method comprising:

performing image recognition on a plurality of frames of post-production video content to identify a plurality of objects in a scene by

identifying a first pixel of a selected color relating to a distinctive image for which the frames of post-production video content are being searched,

scanning a number of adjacent pixels, which are adjacent to the first pixel, to determine a shape of an image in each of at least two directions;

determining at least two outermost endpoints of the selected color relative to the first pixel in each of the at least two directions;

for each of the outermost end points, determining a respective distance between the first pixel and the respective outermost end point;

determining a ratio of the distance between the outermost end points;

comparing the determined ratio to a known data for the distinctive image to determine a location of the distinctive image in the frames of the post-production video content,

calculating a first probability that the distinctive image is present in at least some of the frames of post-production video content based in the ratio and the comparison, and

comparing a concentration of similar pixels that are similar to the first pixel and a shape of a cluster formed by the concentration of pixels to the set of known data for the distinctive image,

calculating a second probability that the distinctive image is present in the frames of post-production video content based on the concentration, shapes of the cluster, and the comparisons, and

locating the distinctive image in the at least some frames of post-production video content in response to either of the calculated first probability or the calculated second probability exceeding a confidence level;

determining a context of the scene which at least some of the frames of video represent based at least in part on the objects recognized the frames of the post-production video content;

determining at least one post-production content modification to be made to at least some of the frames of the post-production video content based at least in part on the determined context of the scene which the frames of post-production video content represent; and

modifying at least some of the at least one of the frames of the postproduction video content based at least in part on determined at least one post-production content modification.

3. The method of claim 2 wherein performing image recognition on a plurality of frames of post-production video content further includes:

in response to neither of the calculated first probability or the calculated second probability exceeding the confidence level, determining a composite probability; and

locating the distinctive image in the at least some frames of postproduction video content in response to the composite probability exceeding the confidence level; and

modifying only those frames of the post-production video content where the calculated composite probability exceeds a confidence level.

4. A system to automate post-production content modification, the system comprising:

at least one processor; and

at least one nontransitory processor-readable medium which stores at least one of processor-executable instructions or data, the at least one nontransitory processor-readable medium communicatively coupled to the at least one processor, and performing image recognition, by at least one processor, execution of which causes the at least one processor to;

perform image recognition on a plurality of frames of post-production video content to identify a plurality of objects in a scene by

identifying a first pixel of a selected color relating to a distinctive image for which the frames of post-production video content are being searched,

scanning a number of adjacent pixels, which are adjacent to the first pixel, to determine a shape of an image in each of at least two directions,

determining at least two outermost endpoints of the selected color relative to the first pixel in each of the at least two directions,

for each of the outermost end points, determining a respective distance between the first pixel and the respective outermost end point,

determining a ratio of the distance between the outermost end points,

comparing the determined ratio to a known data for the distinctive image to determine a location of the distinctive image in the frames of the post-production video content,

calculating a probability that the distinctive image is present in at least some of the frames of post-production video content based in the ratio and the comparison, and

modifying only those frames of the post-production video content where the calculated probability exceeds a confidence level;

determine a context of the scene which at least some of the frames of video represent based at least in part on the objects recognized the frames of the postproduction video content;

determine at least one post-production content modification to be made to at least some of the frames of the post-production video content based at least in part on the determined context of the scene which the frames of post-production video content represent; and

modify at least some of the at least one of the frames of the postproduction video content based at least in part on determined at least one post-production content modification.

5. A system to automate post-production content modification, the system comprising:

at least one processor; and

at least one nontransitory processor-readable medium which stores at least one of processor-executable instructions or data, the at least one nontransitory processor-readable medium communicatively coupled to the at least one processor, and performing image recognition, by at least one processor, execution of which causes the at least one processor to;

perform image recognition on a plurality of frames of post-production video content to identify a plurality of objects in a scene by

identifying a first pixel of a selected color relating to a distinctive image for which the frames of post-production video content are being searched,

scanning a number of adjacent pixels, which are adjacent to the first pixel, to determine a shape of an image in each of at least two directions,

determining at least two outermost endpoints of the selected color relative to the first pixel in each of the at least two directions,

for each of the outermost end points, determining a respective distance between the first pixel and the respective outermost end point,

determining a ratio of the distance between the outermost end points,

comparing the determined ratio to a known data for the distinctive image to determine a location of the distinctive image in the frames of the post-production video content,

calculating a first probability that the distinctive image is present in at least some of the frames of post-production video content based in the ratio and the comparison,

comparing a concentration of similar pixels that are similar to the first pixel and a shape of a cluster formed by the concentration of pixels to the set of known data for the distinctive image,

calculating a second probability that the distinctive image is present in the frames of post-production video content based on the concentration, shapes of the cluster, and the comparisons, and

locating the distinctive image in the at least some frames of post-production video content in response to either of the calculated first probability or the calculated second probability exceeding a confidence level;

determine a context of the scene which at least some of the frames of video represent based at least in part on the objects recognized the frames of the postproduction video content;

determine at least one post-production content modification to be made to at least some of the frames of the post-production video content based at least in part on the determined context of the scene which the frames of post-production video content represent; and

modify at least some of the at least one of the frames of the postproduction video content based at least in part on determined at least one post-production content modification.

6. The system of claim 5 wherein to perform image recognition on the plurality of frames of post-production video content, the at least one processor further:

in response to neither of the calculated first probability or the calculated second probability exceeding the confidence level, determines a composite probability; and

locates the distinctive image in the at least some frames of post-production video content in response to the composite probability exceeding the confidence level; and

modifies only those frames of the post-production video content where the calculated composite probability exceeds a confidence level.