Content retrieval based on semantic association

• US 7,120,626 B2
• Filed: 11/15/2002
• Issued: 10/10/2006
• Est. Priority Date: 11/15/2002
• Status: Expired due to Fees

• Current Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Original Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Inventors: Dimitrova, Nevenka, Li, Dongge
• Primary Examiner(s): Wong, Leslie
• Assistant Examiner(s): Rayyan, Susan F.
• Application Number:
  US10/295,668
• Publication Number:
  US 20040098376A1
• Time in Patent Office: 1,425 Days
• Field of Search: 707/1, 707 3- 4, 707/6, 707/10, 707/102, 707/104.1, 709/203, 705/26
• US Class Current: 1/1
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/437 Administration of user profiles, e.g. generation, initialisation, adaptation, distribution
• CPC Class Codes: G06F 16/64 Browsing; Visualisation therefor generation of a list or set of audio data G06F16/638
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/7847 using low-level visual features of the video content
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching

Abstract

A method and system which enable a user to query a multimedia archive in one media modality and automatically retrieve correlating data in another media modality without the need for manually associating the data items through a data structure. The correlation method finds the maximum correlation between the data items without being affected by the distribution of the data in the respective subspace of each modality. Once the direction of correlation is disclosed, extracted features can be transferred from one subspace to another.

Claims

1. A multimedia system comprising:

a query module generating a query in a plurality of media modalities;

a database including a plurality of matrices, each matrix corresponding to one of the media modalities, wherein each matrix builds a correlation between the corresponding media modality and another media modality;

an object detection module extracting a first plurality of object features from the query and a second plurality of object features from the database wherein the first plurality of object features and the second plurality of object features are extracted from media representing different modalities; and

a processor coupled to the object detection module, the processor being trained on the plurality of matrices of the database to maximize a bi-directional correlation of cross-modality media using sample data, the processor determining a correlation between the first plurality of object features and the second plurality of object features and to retrieve those items from the database which have a correlation at least equal to a predetermined maximum degree of correlation.

2. The system as in claim 1, wherein the correlation is calculated using a canonical correlation methodology.

3. The system as in claim 1, wherein the correlation is calculated using a latent semantic indexing methodology.

4. The system of claim 1, wherein one of the media modalities is a video clip of a character, and a further media modality is an audio clip of the character.

5. The system of claim 1, wherein one of the media modalities is a picture of a person, and a further media modality is a biometric feature of the person.

6. The system of claim 5, wherein the biometric feature is a retinal image.

7. The system of claim 1, wherein one of the media modalities is an unknown liquid, and a further media modality is an aroma having chemical features.

8. The system of claim 7, wherein the unknown liquid is a beverage.

9. The system of claim 1, wherein one of the media modalities is a sound associated with an emotion, the emotion being chosen from a standard list of emotions, and a further media modality is a word for the emotion having textual features.

21. The system as in claim 1, wherein an intermediate step of the training produces orthogonal matrices of modality pairs.

10. A multimedia system comprising:

a query module capable of generating a query in a plurality of media modalities;

a database capable of storing data representing a plurality of media modalities;

an object detection module capable of extracting a first plurality of object features from the query and a second plurality of object features from the database wherein the first plurality of object features and the second plurality of object features are extracted from media representing different modalities; and

a processor coupled to the object detection modules, wherein the processor is arranged to determine a correlation between the first plurality of object features and the second plurality of object features and to retrieve those items from the database which have a correlation at least equal to a predetermined maximum degree of correlation, wherein prior to retrieval, the system is trained to correlate cross-modality media using sample data, wherein the training produces orthogonal matrices A=C_xx<sup>−

1/2</sup>U and B=C_yy<sup>−

1/2</sup>V wherein det(A)=det(B)=1 and C_xx=E{(X−

m_x) (X−

m_x)^T}, C_yy=E{(Y−

m_y)(Y−

m_y)^T}, C_xy=E{(X−

m_x)(y−

m_y)^T}, K=C_xx−

½

·

C_xy.C_yy<sup>−

1/2</sup>=U·

S·

V^T and the correlation between AX representing a first feature set in a first modality and BY representing a second feature set in a second modality is greatest, thereby enabling a transfer of features from the first modality to the second modality, and wherein A and B are orthogonal matrices, X and Y are feature sets from different modalities, C_xx, C_yy, and C_xy are covariance matrices, m_x and m_y are mean vectors, and U, S, and V are obtained from singular value decomposition.

11. The system as in claim 10 wherein AX, the quety, representing the first feature set can be identified given only BY, the result of the query, representing the second feature set, in that BY has the greatest correlation with AX.

12. A method of retrieving at least one item of interest to a user from a multimedia archive comprising the steps of:

generating a query in a plurality of media modalities;

generating a plurality of matrices, each matrix corresponding to one of the media modalities, wherein each matrix builds a correlation between the corresponding media modality and another media modality;

training the plurality of matrices to maximize a bi-directional correlation of cross-modality media using a sample data;

extracting a first plurality of object features from the query, the object features representing a first modality;

extracting a second plurality of object features from items in the multimedia archive, the object features representing a second modality, the archive including the plurality of matrices;

determining a correlation between the first plurality of object features and the second plurality of object features using the plurality of matrices;

retrieving those items from the archive which have object features having a correlation with the object features in the query at least equal to a predetermined maximum degree of correlation.

13. The method as in claim 12, wherein the method of correlation is canonical correlation.

14. The method as in claim 12, wherein the method of correlation is latent semantic indexing.

15. The method as in claim 12, wherein the step of using sample data comprises producing orthogonal matrices of modality pairs as an intermediate step.

16. A method of retrieving at least one item of interest to a user from a multimedia archive comprising the steps of:

generating a query;

extracting a first plurality of object features from the query, the object features representing a first modality;

extracting a second plurality of object features from items in the multimedia archive, the object features representing a second modality;

determining a correlation between the first plurality of object features and the second plurality of object features; and

retrieving those items from the archive which have object features having a correlation with the object features in the query at least equal to a predetermined maximum degree of correlation using sample data to generate correlation matrices to correlate cross-modality media, wherein the matrices generated are represented by A=C_xx<sup>−

1/2</sup>U and B=C_yy<sup>−

1/2</sup>V and wherein det(A)=det(B)=1 and C_xx=E{(X−

m_x)(X−

m_x)^T}, C_yy=E{(Y−

m_y)(Y−

m_y)^T}, C_xy=E{(X−

m_x)(Y−

m_y)^T}, K=C_xx<sup>−

1/2</sup>·

C_xy·

C_yy<sup>−

1/2</sup>=U·

S·

V^T and the correlation between AX representing a first feature set in a first modality and BY representing a second feature set in a second modality is greatest, thereby enabling a transfer of features from the first modality to the second modality, and wherein A and B are orthogonal matrices, X and Y are feature sets from different modalities, C_xx, C_yy, and C_xy are covariance matrices, m_x and m_y are mean vectors, and U, S, and V are obtained from singular value decomposition.

17. The method as in claim 16, wherein AX, the query, representing the first feature set can be identified given only BY, the result of the query, representing the second feature set, in that BY has the greatest correlation with AX.

18. Computer-executable process steps, the computer-executable process steps being stored on a computer-readable medium enabling a user to retrieve media of interest from a database of multimedia comprising:

a query generation step for obtaining a query from the user, the query being in a first media modality;

a matrices generating step generating a plurality of matrices, each matrix corresponding to one of the media modalities, wherein each matrix builds a correlation between the corresponding media modality and another media modality;

a training step training the plurality of matrices to maximize a bi-directional correlation of cross-modality media using a sample data;

a first extracting step for extracting a first plurality of object features from the query;

a second extracting step for extracting a second plurality of object features from items in the multimedia archive, the object features representing a second media modality, the archive including the plurality of matrices;

a correlation calculation step for determining a correlation between the first plurality of object features and the second plurality of object features using the plurality of matrices; and

a retrieval step for retrieving those items from the database which have object features having a correlation with the object features in the query at least equal to a predetermined maximum degree of correlation.

19. A system for retrieving at least one item of interest to a user from a multimedia archive comprising:

means for generating a query in a first media modality;

means for generating a plurality of matrices, each matrix corresponding to one of the media modalities, wherein each matrix builds a correlation between the corresponding media modality and another media modality;

means for training the plurality of matrices to maximize a bi-directional correlation of cross-modality media using a sample data;

means for extracting a first plurality of object features from the query;

means for extracting a second plurality of object features from items in the multimedia archive, the archive including the plurality of matrices;

means for determining a correlation between the first plurality of object features and the second plurality of object features, the second plurality of object features being extracted from a second media modality using the plurality of matrices; and

means for retrieving those items from the archive which have object features having a correlation with the object features in the query at least equal to a predetermined maximum degree of correlation.

20. A method for retrieving a query in a first media modality, when only a result of the query, in a second media modality, is initially known, comprising the steps of:

retrieving a stored matrix, B, for transforming features in the second modality into feature space that is correlated with the first modality, wherein the matrix B was produced during a training procedure to correlate items in the first modality A with items in the second modality B, and vice-versa, such that A=C_xx<sup>−

1/2</sup>U and B=C_yy<sup>−

1/2</sup>V wherein det(A)=det(B)=1 and C_xx=E{(X−

m_x) (X−

m_x)^T}, C_yy=E{(Y−

m_y)(Y−

m_y)^T}, C_xy=E{(X−

m_x)(Y−

m_y)^T}, K=C_xx−

½

·

C_xy·

C_yy<sup>−

1/2</sup>=U·

S·

V^T and the correlation between AX representing a first feature set in the first modality and BY representing a second feature set in the second modality is greatest;

extracting object features from items in the second modality;

calculating AY for the second modality;

extracting object features from items in the first modality, stored in a multimedia database;

calculating AX for each of the items;

correlating AX and AY; and

retrieving the X having the greatest correlation between AX and BY, and wherein A and B are orthogonal matrices, X and Y are feature sets from different modalities, C_xx, C_yy, and C_xy are covariance matrices, m_x and m_y are mean vectors, and U, S, and V are obtained from singular value decomposition.