ENHANCED MAX MARGIN LEARNING ON MULTIMODAL DATA MINING IN A MULTIMEDIA DATABASE

• US 20130251248A1
• Filed: 05/28/2013
• Published: 09/26/2013
• Est. Priority Date: 08/08/2008
• Status: Active Grant

• Current Assignee: The Research Foundation for The State University of New York (State University of New York)
• Original Assignee: The Research Foundation for The State University of New York (State University of New York)
• Inventors: Guo, Zhen, Zhang, Zhongfei
• Granted Patent: US 8,923,630 B2
• Time in Patent Office: Days
• US Class Current: 382/159
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/45 Clustering; Classification
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 17/10 Complex mathematical operations function generation by table look-up G06F1/03; evaluation of elementary functions by calculation G06F7/544
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 18/253 of extracted features
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects

Abstract

Multimodal data mining in a multimedia database is addressed as a structured prediction problem, wherein mapping from input to the structured and interdependent output variables is learned. A system and method for multimodal data mining is provided, comprising defining a multimodal data set comprising image information; representing image information of a data object as a set of feature vectors in a feature space; clustering in the feature space to group similar features; associating a non-image representation with a respective image data object based on the clustering; determining a joint feature representation of a respective data object as a mathematical weighted combination of a set of components of the joint feature representation; optimizing a weighting for a plurality of components of the mathematical weighted combination with respect to a prediction error between a predicted classification and a training classification; and employing the mathematical weighted combination for automatically classifying a new data object.

Claims

1. A method for multimodal data mining, comprising:

defining a multimodal data set comprising image information;

representing image information of a data object as a set of feature vectors in a feature space;

clustering in the feature space to group similar features;

associating a non-image representation with a respective image data object based on the clustering;

determining a joint feature representation of a respective data object as a mathematical weighted combination of a set of components of the joint feature representation;

optimizing a weighting for a plurality of components of the mathematical weighted combination with respect to a prediction error between a predicted classification and a training classification; and

employing the mathematical weighted combination for automatically classifying a new data object.

2. The method according to claim 1, wherein the multimodal data set comprises image information and annotations of the image information.

3. A system for multimodal data mining, comprising:

an input adapted to receive a multimodal data set comprising image information;

an automated processor, adapted for;

representing image information of a data object as a set of feature vectors in a feature space;

clustering in the feature space to group similar features;

associating a non-image representation with a respective image data object based on the clustering;

determining a joint feature representation of a respective data object as a mathematical weighted combination of a set of components of the joint feature representation;

optimizing a weighting for a plurality of components of the mathematical weighted combination with respect to a prediction error between a predicted classification and a training classification; and

an output from the automated processor, adapted to communicate a classification of a new data object based on the mathematical weighted combination.

4. The system according to claim 3, wherein the multimodal data set comprises image information and annotations of the image information.

5. A data processing method, comprising:

(a) receiving a dual variable set μ

comprising labeled examples;

(b) decomposing μ

into two partitions, μ

_B and μ

_N;

(c) iteratively solving with an automated data processor, using the variables in μ

_B, for each member of the set;

min ½

μ

^TDμ

−

μ

^TS

s.t. Aμ

C

μ

_B′

0,μ

_N=0

while there exists μ

_{i, y}ε

μ

_B such that μ

_{i, y}=0, move that variable to partition μ

_N;

while there exists μ

_{i, y}ε

μ

_N satisfying the condition

∃

i, Σ

_yμ

_{i, y}<

C

∃

μ

_{i, y}ε

μ

_N α

^TΦ

_{i,yi, y}−

l(y,y_i)<

0′

move that variable to partition μ

_B and if no such μ

_{i, y}ε

μ

_N exists, ceasing an iteration; and

(d) outputting at least one of;

(a) the members of the partitions and (b) an identification of the members of the partitions.

6. The method according to claim 5, wherein the dual variable set comprises a semantic variable.

7. The method according to claim 5, wherein the dual variable set comprises linguistic data.

8. The method according to claim 5, wherein the dual variable set comprises image data.

9. The method according to claim 5, wherein the dual variable set comprises audio data.

10. The method according to claim 5, wherein the dual variable set comprises a semantic variable and an image variable.

11. The method according to claim 5, further comprising using the outputting to inference between two different examples in the set μ

.

12. The method according to claim 5, further comprising annotating at least one example based on the outputting.

13. The method according to claim 5, further comprising receiving a query representing one of the dual variables, and identifying the examples which correspond to the query.

14. The method according to claim 5, wherein one of the variables is a linguistic variable, and the other of the variables comprises image data, further comprising receiving at least one word as a query and responding to the query by retrieving examples having images which correspond to that word.

15. The method according to claim 5, wherein a first of the variables represents a descriptive annotation of the second of the variables.

16. The method according to claim 5, wherein at least one of the labeled examples comprises representations of a plurality of objects, and wherein a label variable comprises a structured output word space.