Data Recognition in Content

• US 20130322765A1
• Filed: 06/04/2012
• Published: 12/05/2013
• Est. Priority Date: 06/04/2012
• Status: Active Grant

• Current Assignee: Comcast Cable Communications LLC (Comcast Corporation)
• Original Assignee: Comcast Cable Communications LLC (Comcast Corporation)
• Inventors: Neumann, Jan, Tzoukermann, Evelyne, Bagga, Amit, Jojic, Oliver, Shevade, Bageshree, Houghton, David, Farrell, Corey
• Granted Patent: US 8,849,041 B2
• Time in Patent Office: Days
• US Class Current: 382/197
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 40/263 Language identification
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 25/57 for processing of video signals

Abstract

The disclosure relates to recognizing data such as items or entities in content. In some aspects, content may be received and feature information, such as face recognition data and voice recognition data may be generated. Scene segmentation may also be performed on the content, grouping the various shots of the video content into one or more shot collections, such as scenes. For example, a decision lattice representative of possible scene segmentations may be determined and the most probable path through the decision lattice may be selected as the scene segmentation. Upon generating the feature information and performing the scene segmentation, one or more items or entities that are present in the scene may be identified.

Claims

1. A method, comprising:

performing, by a computing device, scene segmentation on video content, resulting in data that defines a plurality of scenes in the video content;

identifying a set of entities in the video content;

for a first scene in the plurality of scenes, identifying a plurality of confidence value vectors representative of features of the first scene, wherein at least one vector of the plurality of confidence value vectors includes a confidence value for at least one entity in the set of entities; and

for the first scene, determining a vector of presence identifiers based at least in part on the plurality of confidence value vectors, wherein at least one identifier in the vector of presence identifiers defines whether an entity in the set of entities is present in the first scene.

2. The method of claim 1, wherein performing scene segmentation includes:

identifying a plurality of shots in the video content;

creating a lattice of nodes that includes one or more nodes for each shot in the plurality of shots, wherein the lattice of nodes defines one or more paths beginning at a first shot of the plurality of shots and ending at a last shot of the plurality of shots;

ranking the one or more paths through the lattice of nodes; and

selecting the plurality of scenes based on the ranking.

3. The method of claim 2, wherein creating the lattice of nodes includes:

calculating a probability that a current shot is a scene boundary;

calculating a probability that the current shot is a non-scene boundary; and

inserting one or more nodes for the current shot into the lattice based on the probability that the current shot is a scene boundary and the probability that the current shot is a non-scene boundary.

4. The method of claim 1, wherein a first vector of the plurality of confidence value vectors is from a face recognition process, and a second vector of the plurality of confidence value vectors is from a voice recognition process, wherein at least one confidence value of the first vector defines a probability that a face of an entity in the set of entities is present, and wherein at least one confidence value of the second vector defines a probability that a category of phone is being uttered by an entity in the set of entities.

5. The method of claim 1, further comprising:

calculating acoustic features from audio of the video content;

detecting an occurrence of a phone based on the acoustic features;

determine a plurality of confidence values for each entity in the set of entities, wherein at least one value in the plurality of confidence values defines a probability that the phone belongs to one of a plurality of phone categories; and

determining the second vector from the plurality of confidence values for each entity in the set of entities, wherein a first value of the second vector is selected from a first plurality of confidence values for a first entity in the set of entities.

6. The method of claim 5, wherein determining a plurality of confidence values for each entity in the set of entities includes calculating the first plurality of confidence values for the first entity using a set of mixture models, each model in the set of mixture models calculates a probability that an input phone belongs to a phone category pronounced by the first entity, and wherein each model in the set of mixture models corresponds to a different phone category from the plurality of phone categories.

7. The method of claim 1, further comprising:

for the first scene, determining a plurality of salience measurements, wherein each of the plurality of salience measurements corresponds to a different entity in the set of entities, and a first measurement in the plurality of salience measurements numerically indicates importance of the first entity to the first scene;

determining that the first measurement satisfies a salience threshold; and

inserting an identifier of the first entity into a listing of entities that are present and salient to the first scene.

8. An apparatus, comprising:

one or more processors;

memory storing computer-executable instructions configured to, with the one or more processors, cause the apparatus to;

perform scene segmentation on video content, resulting in data that defines a plurality of scenes in the video content;

identify a set of entities in the video content;

for a first scene in the plurality of scenes, identify a plurality of confidence value vectors representative of features of the first scene, wherein at least one vector of the plurality of confidence value vectors includes a confidence value for at least one entity in the set of entities; and

for the first scene, determine a vector of presence identifiers based at least in part on the plurality of confidence value vectors, wherein at least one identifier in the vector of presence identifiers defines whether an entity in the set of entities is present in the first scene.

9. The apparatus of claim 8, wherein performing scene segmentation includes:

identifying a plurality of shots in the video content;

creating a lattice of nodes that includes one or more nodes for each shot in the plurality of shots, wherein the lattice of nodes defines one or more paths beginning at a first shot of the plurality of shots and ending at a last shot of the plurality of shots;

ranking the one or more paths through the lattice of nodes; and

selecting the plurality of scenes based on the ranking.

10. The apparatus of claim 9, wherein creating the lattice of nodes includes:

calculating a probability that a current shot is a scene boundary;

calculating a probability that the current shot is a non-scene boundary; and

inserting one or more nodes for the current shot into the lattice based on the probability that the current shot is a scene boundary and the probability that the current shot is a non-scene boundary.

11. The apparatus of claim 8, wherein a first vector of the plurality of confidence value vectors is from a face recognition process, and a second vector of the plurality of confidence value vectors is from a voice recognition process, wherein at least one confidence value of the first vector defines a probability that a face of an entity in the set of entities is present, and wherein at least one confidence value of the second vector defines a probability that a category of phone is being uttered by an entity in the set of entities.

12. The apparatus of claim 8, wherein the memory further stores computer-executable instructions configured to, with the one or more processors, cause the apparatus to:

calculate acoustic features from audio of the video content;

detect an occurrence of a phone based on the acoustic features;

determine a plurality of confidence values for each entity in the set of entities, wherein at least one value in the plurality of confidence values defines a probability that the phone belongs to one of a plurality of phone categories; and

determine the second vector from the plurality of confidence values for each entity in the set of entities, wherein a first value of the second vector is selected from a first plurality of confidence values for a first entity in the set of entities.

13. The apparatus of claim 12, wherein determining a plurality of confidence values for each entity in the set of entities includes calculating the first plurality of confidence values for the first entity using a set of mixture models, each model in the set of mixture models calculates a probability that an input phone belongs to a phone category pronounced by the first entity, and wherein each model in the set of mixture models corresponds to a different phone category from the plurality of phone categories.

14. The apparatus of claim 8, wherein the memory further stores computer-executable instructions configured to, with the one or more processors, cause the apparatus to:

for the first scene, determine a plurality of salience measurements, wherein each of the plurality of salience measurements corresponds to a different entity in the set of entities, and a first measurement in the plurality of salience measurements numerically indicates importance of the first entity to the first scene;

determine that the first measurement satisfies a salience threshold; and

insert an identifier of the first entity into a listing of entities that are present and salient to the first scene.

15. A method comprising:

performing feature recognition on video content using a plurality of feature recognition techniques, resulting in feature information for the video content;

performing, by a computing device, scene segmentation on the video content, wherein the scene segmentation is based on one or more probabilities that the one or more identified shots in the video content are a scene boundary or a non-scene boundary, and wherein performing the scene segmentation results in data that defines one or more scenes in the video content;

identify, from the feature information, a set of confidence value vectors for a first scene of the one or more scenes, wherein a first vector of the set of confidence value vectors is from a first of the plurality of feature recognition techniques, and a second vector of the set of confidence value vectors is from a second of the plurality of feature recognition techniques; and

identify one or more items present in the first scene based on the set of confidence value vectors.

16. The method of claim 15, wherein performing scene segmentation includes:

creating a lattice of nodes that includes one or more nodes for each shot in the one or more identified shots, wherein the lattice of nodes defines one or more possible segmentation paths;

ranking the one or more possible segmentation paths; and

selecting the one or more scenes based on the ranking.

17. The method of claim 15, wherein the plurality of feature recognition techniques includes a face recognition process and a voice recognition process, wherein at least one confidence value of the first vector defines a probability that a face is present, and wherein at least one confidence value of the second vector defines a probability that a category of phone is being uttered.

18. The method of claim 17, wherein the voice recognition process comprises:

calculating acoustic features from audio of the video content;

detecting an occurrence of a phone based on the acoustic features;

determining a plurality of confidence values, wherein at least one value in the plurality of confidence values defines a probability that the phone belongs to one of a plurality of phone categories; and

determining the second vector from the plurality of confidence values.

19. The method of claim 18, wherein determining a plurality of confidence values includes calculating the first plurality of confidence values using a set of mixture models, each model in the set of mixture models calculates a probability that an input phone belongs to a phone category, and wherein each model in the set of mixture models corresponds to a different phone category from the plurality of phone categories.

20. The method of claim 15, further comprising:

determining a plurality of salience measurements, wherein a first measurement in the plurality of salience measurements numerically indicates importance of a particular item of the one or more items to the first scene;

determining that the first measurement satisfies a salience threshold; and

storing an identifier of the first item into a listing of item identifiers that are present and salient to the first scene.