Context-based analysis of multimedia content items using signatures of multimedia elements and matching concepts

• US 9,639,532 B2
• Filed: 12/04/2013
• Issued: 05/02/2017
• Est. Priority Date: 10/26/2005
• Status: Active Grant

• Current Assignee: Cortica Ltd.
• Original Assignee: Cortica Ltd.
• Inventors: Raichelgauz, Igal, Odinaev, Karina, Zeevi, Yehoshua Y.
• Primary Examiner(s): Hill, Stanley K
• Assistant Examiner(s): Bejcek, II, Robert H
• Application Number:
  US14/096,901
• Publication Number:
  US 20140095536A1
• Time in Patent Office: 1,245 Days
• Field of Search: None
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: H04H 60/37 for identifying segments of broadcast information, e.g. scenes or extracting programme ID
• CPC Class Codes: H04H 60/58 of audio determination or detection of speech characteristics in general G10L25/00; speech recognition in general G10L15/00
• CPC Class Codes: H04H 60/59 of video recognising characters or patterns in general G06F18/00, G06V20/00
• CPC Class Codes: H04H 60/64 for providing detail information
• CPC Class Codes: H04H 60/73 using meta-information
• CPC Class Codes: H04N 21/25891 being end-user preferences retrieval of video data in a video database based on user preferences G06F16/739; arrangements for recognizing users' preferences H04H60/46; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/2668 Creating a channel for a dedicated end-user group, e.g. insertion of targeted commercials based on end-user profiles information retrieval from the Internet by querying with filtering and personalisation G06F16/9535; arrangements for replacing or switching information during the broadcast H04H20/10; push services over packet-switching network H04L12/1859; adaptation of message content in packet-switching networks H04L51/063
• CPC Class Codes: H04N 21/466 Learning process for intelligent management, e.g. learning user preferences for recommending movies services using the results of monitoring in broadcast systems H04H60/61
• CPC Class Codes: H04N 21/8106 involving special audio data, e.g. different tracks for different languages
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests

Abstract

A method and server for analyzing a multimedia content item are provided. The method comprises receiving a multimedia content item; extracting from the multimedia content item a plurality of multimedia elements; generating at least one signature for each of the plurality of multimedia elements; for each of the plurality of multimedia elements, querying a deep-content-classification (DCC) system to identify at least one concept that matches one of the plurality of multimedia elements, wherein querying is performed using the at least one signature generated for the multimedia elements and wherein an unidentified multimedia content element does not have a matching concept; generating a context for the multimedia content item using matching concepts; and characterizing each unidentified multimedia element using the generating context and signatures of the matching concepts.

Claims

1. A method for analyzing multimedia content items, comprising:

receiving a multimedia content item;

extracting from the multimedia content item a plurality of multimedia elements;

generating at least one signature for each of the plurality of multimedia elements;

querying a deep-content-classification (DCC) system to identify at least one concept that matches at least one of the plurality of multimedia elements, wherein querying is performed using the at least one signature generated for the multimedia elements and wherein a multimedia element is unidentified when the multimedia element does not have a matching concept identified via the querying;

generating a context for the multimedia content item using matching concepts; and

characterizing each unidentified multimedia element using the generated context and signatures of the identified at least one concept.

2. The method of claim 1, wherein the multimedia content item is at least one of:

an image, a graphic, a video stream, a video clip, an audio stream, an audio clip, a video frame, a photograph, and images of signals.

3. The method of claim 2, wherein the images of signals are one of:

medical signals, geophysical signals, subsonic signals, supersonic signals, electromagnetic signals, infrared signals, an audio signal, a video signal, coordinates, and a sonography signal.

4. The method of claim 1, wherein the at least one signature generated for each of the plurality of multimedia elements is robust to noise and distortions.

5. The method of claim 1, wherein a matching concept is a collection of signatures representing a multimedia element and metadata describing the concept, wherein the collection is a signature reduced cluster generated by inter-matching signatures generated for the plurality of multimedia elements, wherein the matching concept is represented using at least one signature.

6. The method of claim 1, wherein a concept is determined to match a multimedia element when the at least one signature of the concept matches at least one signature generated for the multimedia element over a predefined threshold.

7. The method of claim 5, wherein upon identification of each matching concept the at least one signature representing the concept and metadata matching the concept are returned.

8. The method of claim 7, wherein characterization of each unidentified multimedia content element is performed in part using metadata of the matching concepts.

9. The method of claim 8, wherein the characterization of each unidentified multimedia element includes a correlation of the at least one signature of the unidentified multimedia element against signatures of the matching concepts and signatures of the generated context.

10. The method of claim 9, wherein the correlation of the at least one signature of the unidentified multimedia content element against signatures of the matching concepts further comprising:

identifying any one of a ratio of sizes, angles, scale, location and orientation between the at least one signature of the unidentified multimedia element and signatures of the matching concepts.

11. The method of claim 10, wherein the correlation of the at least one signature of the unidentified multimedia element against signatures of the matching concepts is performed by at least one probabilistic model.

12. A non-transitory computer readable medium having stored thereon instructions for causing one or more processing units to execute the method according to claim 1.

13. A server for analyzing a multimedia content item, comprising:

an interface to a network for receiving a multimedia content item;

a processor; and

a memory connected to the processor, the memory contains instructions that when executed by the processor, configure the server to;

extract from the multimedia content item a plurality of multimedia elements;

generate at least one signature for each of the plurality of multimedia elements;

query a deep-content-classification (DCC) system to identify at least one concept that matches at least one of the plurality of multimedia content elements, wherein querying is performed using the at least one signature generated for the multimedia elements and wherein a multimedia element is unidentified when the multimedia element does not have a matching concept identified via the query;

generate a context for the multimedia content item using the matching concepts; and

characterize each unidentified multimedia element using the generated context and signatures of the identified at least one concept.

14. The server of claim 13, wherein the multimedia content item is at least one of:

an image, a graphic, a video stream, a video clip, an audio stream, an audio clip, a video frame, a photograph, and images of signals.

15. The server of claim 14, wherein the images of signals are one of:

medical signals, geophysical signals, subsonic signals, supersonic signals, electromagnetic signals, infrared signals, an audio signal, a video signal, coordinates, and a sonography signal.

16. The server of claim 13, wherein a matching concept is a collection of signatures representing a multimedia element and metadata describing the concept, wherein the collection is a signature reduced cluster generated by inter-matching signatures generated for the plurality of multimedia elements, wherein the matching concept is represented using at least one signature.

17. The server of claim 13, wherein a concept is determined to match a multimedia element when the at least one signature of the concept matches at least one signature generated for the multimedia element over a predefined threshold.

18. The server of claim 17, wherein upon identification of each matching concept the at least one signature representing the concept and metadata matching the concept are returned.

19. The server of claim 18, wherein characterization of each unidentified multimedia element is performed in part using metadata of matching concepts.

20. The server of claim 19, wherein the characterization of each unidentified multimedia element includes a correlation of the at least one signature of the unidentified multimedia element against signatures of the matching concepts and signatures of the generated context.

21. The server of claim 20, wherein the correlation of the at least one signature of the unidentified multimedia element against signatures of the matching concepts further comprises identifying any one of a ratio of sizes, angles, scale, location and orientation between the at least one signature of the unidentified multimedia element and signatures of the matching concepts.

22. The server of claim 21, wherein the correlation of the at least one signature of the unidentified multimedia element against signatures of the matching concepts is performed by at least one probabilistic model.