Method and apparatus for content representation and retrieval in concept model space

• US 20040117367A1
• Filed: 12/13/2002
• Published: 06/17/2004
• Est. Priority Date: 12/13/2002
• Status: Active Grant

• Current Assignee: SINOEAST CONCEPT LIMITED (Tencent Holdings Limited)
• Original Assignee: International Business Machines Corporation
• Inventors: Smith, John R., Naphade, Milind, Natsev, Apostol
• Granted Patent: US 7,124,149 B2
• Time in Patent Office: Days
• US Class Current: 707/5
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/71 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

A method and apparatus for extracting a model vector representation from multimedia documents. A model vector provides a multidimensional representation of the confidence with which multimedia documents belong to a set of categories or with which a set of semantic concepts relate to the documents. A model vector can be associated with multimedia documents to provide an index of its content or categorization and can be used for comparing, searching, classifying, or clustering multimedia documents. A model vector can be used for purposes of information discovery, personalizing multimedia content, and querying a multimedia information repository.

Claims

1. A method for generating at least one model vector for representing a multimedia document comprising the steps of:

applying a plurality of concept detectors to the multimedia document,

scoring said multimedia document with respect to each detector; and

mapping said scores to a multidimensional space to produce at least one vector representation.

2. The method of claim 1 wherein said plurality of concept detectors corresponds to a fixed lexicon of categories, objects, events, scenes, and people.

3. The method of claim 1 wherein said document comprises multiple modalities such as audio, visual, text and speech and wherein said concept detectors operate on single or multiple modalities present in said multimedia document.

4. The method of claim 1 wherein said concept detectors operate on content-based descriptors of document features extracted from said multimedia document.

5. The method of claim 1 wherein said concept detectors operate on metadata associated with said multimedia document.

6. The method of claim 1 wherein said concept detectors operate on contextual information associated with the multimedia document.

7. The method of claim 1 wherein said concept detectors operate on knowledge bases.

8. The method of claim 1 wherein said detectors correspond to trained model statistical classifiers.

9. The method of claim 1 wherein said scoring is based on at least one of the confidence of detecting a concept in the multimedia document, the relevance of a concept to the multimedia document, and the reliability of the detector with respect to the concept.

10. The method of claim 1 wherein said mapping corresponds to at least one of concatenating said scores, performing linear transforms, performing nonlinear transformations, performing quantization, and dimensionality reduction by thresholding.

11. The method of claim 10 wherein said mapping of said scores to model vector dimensions may be one-to-one, one-to-many, many-to-one, or many-to-many.

12. The method of claim 1 wherein said plurality of detectors is fixed for said indexing of multimedia documents;

13. The method of claim 1 wherein said scoring and mapping methods are fixed for said indexing of multimedia documents.

14. A method for indexing multimedia documents using model vectors comprising the steps of

generating one or more model vectors for each multimedia document;

associating said model vectors with corresponding multimedia documents; and

building an index for accessing said multimedia documents based on values of said associated model vectors.

15. The method of claim 14 wherein said generating at least one model vector for representing a multimedia document comprises the steps of:

applying a plurality of concept detectors to the multimedia document;

scoring said multimedia document with respect to each detector; and

mapping said scores to a multidimensional space to produce at least one vector representation.

16. The method of claim 15 wherein said plurality of detectors is fixed for said indexing of multimedia documents.

17. The method of claim 15 wherein said scoring and mapping methods are fixed for said indexing of multimedia documents.

18. The method of claim 14 wherein multiple model vectors are generated for each multimedia document based on each of multiple modalities, features, descriptors, or models.

19. The method of claim 14 wherein said associating is based on database key values, media locators, or other types of identifiers.

20. The method of claim 14 wherein said index allows similarity searching, nearest neighbor access, or range searching based on said model vector values.

21. A method for using model vectors in applications with multimedia documents comprising the steps of:

generating at least one model vector for representing each multimedia document by the steps of

applying a plurality of concept detectors to the multimedia document;

scoring said multimedia document with respect to each detector; and

mapping said scores to a multidimensional space to produce at least one vector representation; and

performing at least one operation on said multimedia documents on the basis of the values of said at least one vector representation.

22. The method of claim 21 wherein said at least one operation comprises using said model vectors for search and retrieval of documents from a multimedia information repository.

23. The method of claim 21 wherein said at least one operation comprises at least one of filtering, summarizing and personalizing multimedia information.

24. The method of claim 21 wherein said at least one operation comprises data mining.

25. The method of claim 21 wherein said at least one operation comprises clustering said documents.

26. The method of claim 21 wherein said at least one operation comprises classifying said documents.

27. A program storage device readable by machine tangibly embodying a program of instructions executable by the machine for performing a method for generating at least one model vector for representing a multimedia document, said method comprising the steps of:

applying a plurality of concept detectors to the multimedia document;

scoring said multimedia document with respect to each detector; and

mapping said scores to a multidimensional space to produce at least one vector representation.

28. A program storage device readable by machine tangibly embodying a program of instructions executable by the machine for performing a method for using model vectors in applications with multimedia documents wherein said method comprises the steps of:

generating at least one model vector for representing each multimedia document by the steps of;

applying a plurality of concept detectors to the multimedia document;

scoring said multimedia document with respect to each detector; and

mapping said scores to a multidimensional space to produce at least one vector representation; and

performing at least one operation on said multimedia documents on the basis of the values of said at least one vector representation.

29. A system for using model vectors in applications with multimedia documents comprising:

at least one model vector generation component for generating at least one model vector for representing each multimedia document; and

at least one document processing component for performing at least one operation on said multimedia documents on the basis of the values of said at least one vector representation.

30. The system of claim 29 wherein said at least one model vector generation component comprises:

at least one concept detector application component for applying a plurality of concept detectors to the multimedia document,

a scoring component for scoring said multimedia document with respect to each detector; and

a mapping component for mapping said scores to a multidimensional space to produce at least one vector representation.

performing at least one operation on said multimedia documents on the basis of the values of said at least one vector representation.