VIDEOLENS MEDIA ENGINE

• US 20120114233A1
• Filed: 06/28/2011
• Published: 05/10/2012
• Est. Priority Date: 11/08/2010
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Gunatilake, Priyan
• Granted Patent: US 8,971,651 B2
• Time in Patent Office: Days
• US Class Current: 382/165
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2025/783 based on threshold decision
• CPC Class Codes: H04N 5/91 Television signal processing therefor
• CPC Class Codes: Y10S 707/914 Video

Abstract

A system, method, and computer program product for automatically analyzing multimedia data are disclosed. Embodiments receive multimedia data, detect portions having specified features, and output a corresponding subset of the multimedia data. Content features from downloaded or streaming movies or video clips are identified as a human probably would do, but in essentially real time. Embodiments then generate an index or menu based on individual consumer preferences. Consumers can peruse the index, or produce customized trailers, or edit and tag content with metadata as desired. The tool can categorize and cluster content by feature, to assemble a library of scenes or scene clusters according to user-selected criteria.

Claims

1. A method for automated analysis of multimedia data, comprising:

receiving multimedia data;

identifying at least one multimedia data portion having specified content features; and

responsively outputting at least one identified multimedia data portion.

2. The method of claim 1 wherein the analysis occurs in real time.

3. The method of claim 1 wherein a resource-limited consumer application performs the analysis.

4. The method of claim 1 wherein the analysis includes performing heuristics based on human perception.

5. The method of claim 1 wherein the analysis operates on a subset of frames based on at least one of a time division and a group-of-pictures boundary.

6. The method of claim 1 wherein the analysis includes at least one of converting speech to text and performing facial recognition.

7. The method of claim 1 wherein the analysis operates on video features including at least one of (a) compressed domain motion vectors and location distributions, (b) compressed domain macro block sizes and corresponding distributions and block types and corresponding locations, (c) compressed domain frame types, and (d) raw color histograms and associated statistics.

8. The method of claim 1 wherein the analysis operates on audio features including at least one of (a) RMS power of a sampled audio frame, (b) spectral centroid calculated via a short term Fourier transform, and (c) short-term MFC power spectrum representation on a nonlinear Mel scale frequency.

9. The method of claim 1 wherein the analysis identifies action scenes as multimedia data portions having at least one of elevated audio levels and elevated motion vector magnitudes.

10. The method of claim 9 wherein action scene identification further comprises:

selecting a subset of P frames having elevated motion vector magnitudes;

for each selected P frame, decoding the corresponding audio frame, computing an RMS power value for the corresponding audio frame; and

selecting audio frames having the highest RMS power value.

11. The method of claim 1 wherein the analysis identifies low-motion scenes as multimedia data portions having both largest adjacent macro blocks and smallest motion vector magnitudes.

12. The method of claim 11 wherein the low-motion scene identification further comprises:

selecting a subset of P frames;

for each selected P frame, extracting macro blocks and motion vectors, determining macro block sizes and motion vector magnitudes; and

selecting P frames having the largest adjacent macro blocks and the smallest motion vector magnitudes.

13. The method of claim 1 wherein the analysis identifies preview points as multimedia data portions having wider color distributions.

14. The method of claim 13 wherein the preview point identification further comprises:

dividing the multimedia data into non-overlapping sections;

for each section, extracting three equally spaced candidate frames, computing a color histogram for each candidate frame; and

selecting the candidate frame with the largest standard deviation.

15. The method of claim 1 wherein the analysis treats MFCCs as feature vectors and responsively identifies multimedia data portions having at least one of human dialog and music.

16. The method of claim 1 wherein the multimedia data includes at least one of audio content, video content, compressed multimedia data, uncompressed multimedia data, downloaded data, streaming data, online data, and free data.

17. The method of claim 1 wherein the identified portion is at least one of an audio clip, a video image, and a video clip.

18. The method of claim 1 further comprising generating metadata corresponding to the analysis.

19. A system for automatically analyzing multimedia data, comprising:

a processor configured to;

receive multimedia data;

identify at least one multimedia data portion having specified content features; and

responsively output at least one identified multimedia data portion.

20. A computer program product comprising a machine-readable medium tangibly embodying non-transitory program instructions thereon that, when executed by a computer, cause the computer to:

receive multimedia data;

identify at least one multimedia data portion having specified content features; and

responsively output at least one identified multimedia data portion.