CROSS-MEDIA SIMILARITY MEASURES THROUGH TRANS-MEDIA PSEUDO-RELEVANCE FEEDBACK AND DOCUMENT RERANKING

• US 20100082615A1
• Filed: 09/19/2008
• Published: 04/01/2010
• Est. Priority Date: 09/19/2008
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Clinchant, Stephane, Renders, Jean-Michel
• Granted Patent: US 9,183,227 B2
• Time in Patent Office: Days
• US Class Current: 707/728
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour

Abstract

A multimedia information retrieval system includes a storage and an electronic processing device. The latter is configured to perform a process including: computing values of a pairwise similarity measure quantifying pairwise similarity of documents of a multimedia reference repository; storing the computed values in the storage; performing an initial information retrieval process respective to the multimedia reference repository to return a set of initial repository documents; and identifying a set of top ranked documents of the multimedia reference repository based at least on the stored computed values pertaining to the set of initial repository documents.

Claims

1. A multimedia information retrieval method performed by an electronic:

device, the method comprising;

performing an initial information retrieval process respective to a multimedia reference repository to return a set of initial repository documents;

computing values of at least one monomodal pairwise similarity measure for candidate documents of the multimedia reference repository respective to repository documents of the set of initial repository documents; and

identifying a set of top-ranked documents of the multimedia reference repository based at least in part on the values computed for the candidate documents.

2. The multimedia information retrieval method as set forth in claim 1, wherein the performing of an initial information retrieval process comprises:

performing a first monomodal information retrieval operation respective to the multimedia reference repository using a first monomodal query derived from a query document, the first monomodal query relating to a first media modality;

wherein the set of initial repository documents comprises at least results of the first monomodal information retrieval operation.

3. The multimedia information retrieval method as set forth in claim 2, wherein the performing of an initial information retrieval process further comprises:

performing a second monomodal information retrieval operation respective to the multimedia reference repository using a second monomodal query derived from the query document, the second monomodal query relating to a second media modality different from the first media modality;

wherein the set of initial repository documents comprises an aggregation of results of the first monomodal information retrieval operation and the second monomodal information retrieval operation.

4. The multimedia information retrieval method as set forth in claim 2, further comprising:

assigning a classification to the query document based on the set of top-ranked documents of the multimedia reference repository.

5. The multimedia information retrieval method as set forth in claim 1, wherein the identifying comprises ranking the candidate documents based at least in part on an aggregation for each candidate document of the values of the monomodal pairwise similarity measure aggregated over the set of initial repository documents.

6. The multimedia information retrieval method as set forth in claim 5, wherein the identifying further comprises ranking the candidate documents based at least in part on an aggregation for each candidate document of the values of at least two monomodal pairwise similarity measures relating to at least two different media modalities.

7. The multimedia information retrieval method as set forth in claim 6, wherein the at least two different media modalities include an image media modality and a text media modality.

8. The multimedia information retrieval method as set forth in claim 1, wherein the computing is performed before the performing of the initial information retrieval process, and the method further comprises:

storing the computed values in a storage of or accessible by the electronic device, the identifying comprising selecting and retrieving the values computed for the candidate documents from the storage.

9. The multimedia information retrieval method as set forth in claim 1, further comprising:

displaying the set of top-ranked documents of the multimedia reference repository or an indication of the set of top-ranked documents of the multimedia reference repository on a display of or accessible by the electronic device.

10. The multimedia information retrieval method as set forth in claim 1, wherein:

the performing an initial information retrieval process comprises performing a monomodal initial information retrieval process using a query comprising first media modality content, and

the computing values comprises computing values of at least one monomodal second media modality pairwise similarity measure relating to a second media modality different from the first media modality.

11. The multimedia information retrieval method as set forth in claim 10, wherein the identifying comprises:

identifying a set of top-ranked documents of the multimedia reference repository based on (i) similarity of first media modality content of the candidate documents respective to first media modality content of the query and (ii) the values computed for the candidate documents respective to repository documents of the set of initial repository documents.

12. A multimedia information retrieval method performed by an electronic device, the method comprising:

performing an initial monomodal information retrieval process operating on a first media modality to retrieve a set of initial repository documents from the multimedia reference repository; and

identifying a set of top-ranked documents of the multimedia reference repository based at least in part on pairwise similarity measure values respective to a second media modality different from the first media modality for document pairs that include documents of the set of initial repository documents.

13. The multimedia information retrieval method as set forth in claim 12, wherein the first media modality is text and the second media modality is images.

14. The multimedia information retrieval method as set forth in claim 12, wherein the first media modality is images and the second media modality is text.

15. The multimedia information retrieval method as set forth in claim 12, wherein the identifying is further based on pairwise similarity measure values respective to the first media modality for document pairs that include documents of the set of initial repository documents.

16. The multimedia information retrieval method as set forth in claim 12, further comprising:

pre-computing pairwise similarity measure values respective to the first and second media modalities; and

storing the pre-computed values in a storage of or accessible by the electronic device, the pre-computed values being used in the initial monomodal information retrieval process and in the identifying operation.

17. The multimedia information retrieval method as set forth in claim 12, wherein identifying is based solely on pairwise similarity measure values respective to the second media modality.

18. A multimedia information retrieval system comprising:

a storage; and

an electronic processing device configured to perform a process including;

performing an initial monomodal information retrieval process respective to a multimedia reference repository to return a set of initial repository documents, the monomodal information retrieval process operating on one member of the group consisting of text content and image content, and

identifying a set of top-ranked documents of the multimedia reference repository based at least on pairwise similarity measure values indicative of similarity with documents of the set of initial repository documents, the pairwise similarity measure values being indicative of similarity respective to the other member of the group consisting of text content and image content.

19. The multimedia information retrieval system as set forth in claim 18, wherein the monomodal information retrieval process is a text-only information retrieval process and the pairwise similarity measure values are indicative of image content similarity.

20. The multimedia information retrieval system as set forth in claim 18, wherein the monomodal information retrieval process is an image-only information retrieval process and the pairwise similarity measure values are indicative of text content similarity.