EXTENDED VIDEOLENS MEDIA ENGINE FOR AUDIO RECOGNITION

• US 20130006625A1
• Filed: 06/28/2011
• Published: 01/03/2013
• Est. Priority Date: 06/28/2011
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Gunatilake, Priyan, Nguyen, Djung, Patil, Abhishek, Saha, Dipendu
• Granted Patent: US 8,938,393 B2
• Time in Patent Office: Days
• US Class Current: 704/235
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 17/08 Use of distortion metrics or a particular distance between probe pattern and reference templates
• CPC Class Codes: G10L 25/81 for discriminating voice from music
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73

Abstract

A system, method, and computer program product for automatically analyzing multimedia data audio content are disclosed. Embodiments receive multimedia data, detect portions having specified audio features, and output a corresponding subset of the multimedia data and generated metadata. Audio content features including voices, non-voice sounds, and closed captioning, from downloaded or streaming movies or video clips are identified as a human probably would do, but in essentially real time. Particular speakers and the most meaningful content sounds and words and corresponding time-stamps are recognized via database comparison, and may be presented in order of match probability. Embodiments responsively pre-fetch related data, recognize locations, and provide related advertisements. The content features may be also sent to search engines so that further related content may be identified. User feedback and verification may improve the embodiments over time.

Claims

1. A method for automated analysis of multimedia data, comprising:

receiving multimedia data;

identifying at least one multimedia data portion having a target audio content feature;

recognizing at least one aspect of the identified feature; and

responsively outputting an indication of the aspect.

2. The method of claim 1 wherein the analysis occurs in real time.

3. The method of claim 1 wherein a resource-limited consumer application performs the analysis.

4. The method of claim 1 wherein the analysis operates on audio features including at least one of RMS power of a sampled audio frame and short-term MFC power spectrum representation on a nonlinear Mel scale frequency.

5. The method of claim 1 wherein the multimedia data includes at least one of audio content, video content, compressed multimedia data, uncompressed multimedia data, downloaded data, streaming data, online data, and free data.

6. The method of claim 1 wherein the identified portion is at least one of an audio clip, a video image, and a video clip.

7. The method of claim 1 further comprising generating metadata corresponding to the analysis.

8. The method of claim 1 wherein the target audio content feature is at least one of human dialogue, laughter, animal sounds, and location specific sounds.

9. The method of claim 1 wherein the target audio content feature is human dialogue and the aspect is at least one of the name of a speaking actor and the name of a speaking character.

10. The method of claim 9 further comprising recognizing the aspect by:

comparing the identified portion with a set of voice samples to find at least one best matching sample; and

outputting at least one of a speaker name and a character name.

11. The method of claim 10 wherein the comparing comprises:

representing a voice sample as one of a MFC coefficient vector and a feature vector;

computing at least one of a Euclidean distance and a correlation measure from the representation; and

outputting at least one best matching sample in probability order having at least one of a low Euclidean distance and a high correlation measure.

12. The method of claim 11 further comprising applying a minimum difference threshold to reject non-matching samples.

13. The method of claim 1 wherein the target audio content feature is human dialogue and the aspect includes a word and corresponding time-stamp.

14. The method of claim 13 further comprising recognizing the aspect by:

selectively converting speech to text if the word is not in text form;

selecting the most meaningful content words; and

outputting the selected words and their corresponding time-stamps.

15. The method of claim 1 further comprising:

finding additional multimedia data related to the indication with a directed search.

16. The method of claim 15 further comprising pre-fetching the additional multimedia data in anticipation of user interest.

17. The method of claim 15 wherein the additional multimedia data includes at least one of related movies, current movies, movie previews, current events, a biography, a filmography, related advertisements, information on recognized locations, and a scene characterization.

18. The method of claim 1 further comprising:

storing user verification of the indication; and

improving recognition accuracy by incorporating prior recognitions.

19. A system for automatically analyzing multimedia data, comprising:

a processor configured to;

receive multimedia data;

identify at least one multimedia data portion having a target audio content feature;

recognize at least one aspect of the identified feature; and

responsively output an indication of the aspect.

20. A computer program product comprising a machine-readable medium tangibly embodying non-transitory program instructions thereon that, when executed by a computer, cause the computer to:

receive multimedia data;

identify at least one multimedia data portion having a target audio content feature;

recognize at least one aspect of the identified feature; and

responsively output an indication of the aspect.