Content retrieval based on semantic association

• US 20040098376A1
• Filed: 11/15/2002
• Published: 05/20/2004
• Est. Priority Date: 11/15/2002
• Status: Active Grant

• Current Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Original Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Inventors: Dimitrova, Nevenka, Li, Dongge
• Granted Patent: US 7,120,626 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/437 Administration of user profiles, e.g. generation, initialisation, adaptation, distribution
• CPC Class Codes: G06F 16/64 Browsing; Visualisation therefor generation of a list or set of audio data G06F16/638
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/7847 using low-level visual features of the video content
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

A method and system which enable a user to query a multimedia archive in one media modality and automatically retrieve correlating data in another media modality without the need for manually associating the data items through a data structure. The correlation method finds the maximum correlation between the data items without being affected by the distribution of the data in the respective subspace of each modality. Once the direction of correlation is disclosed, extracted features can be transferred from one subspace to another.

Claims

1. A multimedia system comprising:

a query module capable of generating a query in a plurality of media modalities;

a database capable of storing data representing a plurality of media modalities;

an object detection module capable of extracting a first plurality of object features from the query and a second plurality of object features from the database wherein the first plurality of object features and the second plurality of object features are extracted from media representing different modalities;

a processor coupled to the object detection modules,

wherein the processor is arranged to determine a correlation between the first plurality of object features and the second plurality of object features and to retrieve those items from the database which have a correlation at least equal to a predetermined maximum degree of correlation.

2. The system as in claim 1, wherein prior to retrieval, the system is trained to correlate cross-modality media using sample data.

3. The system as in claim 1, wherein the correlation is calculated using a canonical correlation methodology.

4. The system as in claim 1, wherein the correlation is calculated using a latent semantic indexing methodology.

5. The system as in claim 2, wherein the training produces orthogonal matrices A=C_xx⁻

1/2U and B=C_yy<sup>−

1/2</sup>V wherein det(A)=det(B)=1 and C_xx=E{(X−

m_x) (X−

m_x)^T}, C_yy=E{(Y−

m_y) (Y−

m_y)^T}, C_xy=E{(X−

m_x) (Y−

m_y)^T}, K=C<sub>xx−

1/2</sub>·

C_xy·

C_yy<sup>−

1/2</sup>=U·

S·

V^T and the correlation between AX representing a first feature set in a first modality and BY representing a second feature set in a second modality is greatest, thereby enabling a transfer of features from the first modality to the second modality.

6. The system as in claim 5 wherein AX, the query, representing the first feature set can be identified given only BY, the result of the query, representing the second feature set, in that BY has the greatest correlation with AX.

7. A method of retrieving at least one item of interest to a user from a multimedia archive comprising the steps of:

generating a query;

extracting a first plurality of object features from the query, the object features representing a first modality;

extracting a second plurality of object features from items in the multimedia archive, the object features representing a second modality;

determining a correlation between the first plurality of object features and the second plurality of object features;

retrieving those items from the archive which have object features having a correlation with the object features in the query at least equal to a predetermined maximum degree of correlation.

8. The method as in claim 7, further comprising the step of using sample data to generate correlation matrices which are used to correlate cross-modality media.

9. The method as in claim 7, wherein the method of correlation is canonical correlation.

10. The method as in claim 7, wherein the method of correlation is latent semantic indexing.

11. The method as in claim 7, wherein the matrices generated are represented by A=C_xx⁻

1/2U and B=C_yy<sup>−

1/2</sup>V and wherein det(A)=det(B)=1 and C_xx=E{(X−

m_x) (X−

m_x)^T}C_yy=E{(Y−

m_y) (Y−

m_y)^T}, C_xy=E{(X−

m_x) (Y−

m_y)^T}, K=C_xx<sup>−

1/2</sup>·

C_xy·

C_yy<sup>−

1/2</sup>=U·

S·

V^T and the correlation between AX representing a first feature set in a first modality and BY representing a second feature set in a second modality is greatest, thereby enabling a transfer of features from the first modality to the second modality.

12. The method as in claim 11, wherein AX, the query, representing the first feature set can be identified given only BY, the result of the query, representing the second feature set, in that BY has the greatest correlation with AX.

13. Computer-executable process steps, the computer-executable process steps being stored on a computer-readable medium enabling a user to retrieve media of interest from a database of multimedia comprising:

a query generation step for obtaining a query from the user, the query being in a first media modality;

a first extracting step for extracting a first plurality of object features from the query;

a second extracting step for extracting a second plurality of object features from items in the multimedia archive, the object features representing a second media modality;

a correlation calculation step for determining a correlation between the first plurality of object features and the second plurality of object features;

a retrieval step for retrieving those items from the database which have object features having a correlation with the object features in the query at least equal to a predetermined maximum degree of correlation.

14. Means for retrieving at least one item of interest to a user from a multimedia archive comprising:

means for generating a query in a first media modality;

means for extracting a first plurality of object features from the query;

means for extracting a second plurality of object features from items in the multimedia archive;

means for determining a correlation between the first plurality of object features and the second plurality of object features, the second plurality of object features being extracted from a second media modality;

retrieving those items from the archive which have object features having a correlation with the object features in the query at least equal to a predetermined maximum degree of correlation.

15. A method for retrieving at least one video clip of a character from a multimedia archive, the method comprising the steps of:

generating a query comprising an audio clip of the character'"'"'s voice;

extracting a plurality of audio features from the query;

extracting a plurality of video features from each video clip in the multimedia archive;

calculating a correlation between the plurality of audio features and the plurality of video features; and

retrieving at least one video clip of the character speaking based upon maximizing the degree of correlation between the audio and the video.

16. A method for retrieving at least one picture of a person, stored within a multimedia archives the method comprising the steps of:

generating a query comprising a biometric feature of the person;

extracting a plurality of visual features from the query;

extracting a plurality of visual features from each picture in the multimedia archive;

calculating a correlation between the plurality of visual features from the archive and the plurality of visual features form the query; and

retrieving at least one picture of the person based upon maximizing the degree of correlation between the plurality of visual features extracted from the query and the plurality of visual features extracted from the multimedia archive.

17. The method as in claim 16, wherein the biometric feature is a retinal image.

18. A method for retrieving at least one item of information, stored within a multimedia archive, identifying an unknown liquid, the method comprising the steps of:

generating a query comprising an aroma;

extracting a plurality of chemical features, which are represented digitally, from the query;

extracting a plurality of textual features from each text item in the multimedia archive;

calculating a correlation between the plurality of chemical features extracted from the query and the plurality of text features extracted from the multimedia archive; and

retrieving at least one item of information identifying the unknown liquid, based upon maximizing the degree of correlation between the chemical features extracted from the query and the textual features extracted from the multimedia archive.

19. The method as in claim 18, wherein the unknown liquid is a beverage.

20. A method for retrieving from a multimedia archive a sound associated with an emotion, the emotion being chosen from the standard emoticons list, the method comprising the steps of:

generating a query comprising a word for the emotion;

extracting a plurality of textual features from the query;

extracting a plurality of audio features for each sound in the multimedia archive;

calculating a correlation between the plurality of textual features extracted from the query and the plurality of audio features extracted from the multimedia archive; and

retrieving at least one sound based upon maximizing the degree of correlation between the textual features extracted from the query and the audio features extracted from the multimedia archive.

21. A method for retrieving a query in a first media modality, when only a result of the query, in a second media modality, is initially known, comprising the steps of:

retrieving a stored matrix, B, for transforming features in the second modality into feature space that is correlated with the first modality, wherein the matrix B was produced during a training procedure to correlate items in the first modality A with items in the second modality B, and vice-versa, such that A=C_xx<sup>−

1/2</sup>U and B=C_yy<sup>−

1/2</sup>V wherein det (A)=det (B)=1 and C_xx=E{(X−

m_x) (X−

m_x)^T}, C_yy=E{(Y−

m_Y) (Y−

m_y)^T}, C_xy=E{(X−

m_x) (y−

m_y)^T}, K=C_xx−

1/2·

C_xy·

C_yy<sup>−

1/2</sup>=U·

S·

V^T and the correlation between AX representing a first feature set in the first modality and BY representing a second feature set in the second modality is greatest;

extracting object features from items in the second modality;

calculating AY for the second modality;

extracting object features from items in the first modality, stored in a multimedia database;

calculating AX for each of the items;

correlating AX and AY; and

retrieving the X having the greatest correlation between AX and BY.