Cross-media similarity measures through trans-media pseudo-relevance feedback and document reranking

• US 9,183,227 B2
• Filed: 09/19/2008
• Issued: 11/10/2015
• Est. Priority Date: 09/19/2008
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Clinchant, Stephane, Renders, Jean-Michel
• Primary Examiner(s): KINSAUL, DANIEL W
• Application Number:
  US12/233,978
• Publication Number:
  US 20100082615A1
• Time in Patent Office: 2,608 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour

Abstract

A multimedia information retrieval system includes a storage and an electronic processing device. The latter is configured to perform a process including: computing values of a pairwise similarity measure quantifying pairwise similarity of documents of a multimedia reference repository; storing the computed values in the storage; performing an initial information retrieval process respective to the multimedia reference repository to return a set of initial repository documents; and identifying a set of top ranked documents of the multimedia reference repository based at least on the stored computed values pertaining to the set of initial repository documents.

Claims

1. A multimedia information retrieval method performed by an electronic device, the method comprising:

pre-computing values of at least one monomodal pairwise similarity measure for document pairs of a multimedia reference repository and storing the pre-computed values of at least one monomodal pairwise similarity measure for the document pairs of the multimedia reference repository in a storage of or accessible by the electronic device;

after storing the pre-computed values of at least one monomodal pairwise similarity measure for the document pairs of the multimedia reference repository, performing an initial information retrieval process respective to the multimedia reference repository to return a set of initial repository documents;

selecting and retrieving from the storage pre-computed values of the at least one monomodal pairwise similarity measure for candidate documents of the multimedia reference repository respective to repository documents of the set of initial repository documents; and

identifying a set of top-ranked documents of the multimedia reference repository based at least in part on the retrieved pre-computed values of the at least one monomodal pairwise similarity measure for the candidate documents of the multimedia reference repository respective to repository documents of the set of initial repository documents.

2. The multimedia information retrieval method as set forth in claim 1, wherein the performing of an initial information retrieval process comprises:

performing a first monomodal information retrieval operation respective to the multimedia reference repository using a first monomodal query derived from a query document, the first monomodal query relating to a first media modality;

wherein the set of initial repository documents comprises at least results of the first monomodal information retrieval operation.

3. The multimedia information retrieval method as set forth in claim 2, wherein the performing of an initial information retrieval process further comprises:

performing a second monomodal information retrieval operation respective to the multimedia reference repository using a second monomodal query derived from the query document, the second monomodal query relating to a second media modality different from the first media modality;

wherein the set of initial repository documents comprises an aggregation of results of the first monomodal information retrieval operation and the second monomodal information retrieval operation.

4. The multimedia information retrieval method as set forth in claim 2, further comprising:

assigning a classification to the query document based on the set of top-ranked documents of the multimedia reference repository.

5. The multimedia information retrieval method as set forth in claim 1, wherein the identifying comprises ranking the candidate documents based at least in part on an aggregation for each candidate document of the retrieved pre-computed values of the at least one monomodal pairwise similarity measure for the candidate document respective to repository documents of the set of initial repository documents aggregated over the set of initial repository documents.

6. The multimedia information retrieval method as set forth in claim 5, wherein the identifying further comprises ranking the candidate documents based at least in part on an aggregation for each candidate document of the retrieved pre-computed values of at least two monomodal pairwise similarity measures relating to at least two different media modalities.

7. The multimedia information retrieval method as set forth in claim 6, wherein the at least two different media modalities include an image media modality and a text media modality.

8. The multimedia information retrieval method as set forth in claim 1, further comprising:

displaying the set of top-ranked documents of the multimedia reference repository or an indication of the set of top-ranked documents of the multimedia reference repository on a display of or accessible by the electronic device.

9. The multimedia information retrieval method as set forth in claim 1, wherein:

the performing an initial information retrieval process comprises performing a monomodal initial information retrieval process using a query comprising first media modality content, and

the pre-computing values comprises pre-computing values of at least one monomodal second media modality pairwise similarity measure relating to a second media modality different from the first media modality.

10. The multimedia information retrieval method as set forth in claim 9, wherein the identifying comprises:

identifying a set of top-ranked documents of the multimedia reference repository based on (i) similarity of first media modality content of the candidate documents respective to first media modality content of the query and (ii) the retrieved pre-computed values of the at least one monomodal pairwise similarity measure for the candidate documents respective to repository documents of the set of initial repository documents.

11. A multimedia information retrieval method performed by an electronic device, the method comprising:

pre-computing pairwise similarity measure values respective to different first and second media modalities for all pairs of documents in a multimedia reference repository and storing the pre-computed values in a storage of or accessible by the electronic device;

performing an initial monomodal information retrieval process operating on the first media modality using pre-computed pairwise similarity measure values respective to the first media modality retrieved from the storage to retrieve a set of initial repository documents from the multimedia reference repository; and

identifying a set of top-ranked documents of the multimedia reference repository based at least in part on pre-computed pairwise similarity measure values respective to the second media modality retrieved from the storage for document pairs that include documents of the set of initial repository documents.

12. The multimedia information retrieval method as set forth in claim 11, wherein the first media modality is text and the second media modality is images.

13. The multimedia information retrieval method as set forth in claim 11, wherein the first media modality is images and the second media modality is text.

14. The multimedia information retrieval method as set forth in claim 11, wherein the identifying is further based on pairwise similarity measure values respective to the first media modality for document pairs that include documents of the set of initial repository documents.

15. The multimedia information retrieval method as set forth in claim 11, wherein identifying is based solely on the pre-computed pairwise similarity measure values respective to the second media modality retrieved from the storage for document pairs that include documents of the set of initial repository documents.

16. A multimedia information retrieval system comprising:

a storage that stores;

(i) a multimedia reference repository of multimedia documents and corresponding image and text content feature representations,

(ii) precomputed pairwise document text similarity measure values indicative of text content similarity for all pairs of documents in the multimedia reference repository, and

(iii) precomputed pairwise document image similarity measure values indicative of image content similarity for all pairs of documents in the multimedia reference repository; and

an electronic processing device configured to perform a process including;

performing an initial monomodal information retrieval process respective to the multimedia reference repository to return a set of initial repository documents, the monomodal information retrieval process operating on one member of the group consisting of text content and image content, and

identifying a set of top-ranked documents of the multimedia reference repository based at least on precomputed pairwise similarity measure values indicative of similarity with documents of the set of initial repository documents, the precomputed pairwise similarity measure values being indicative of similarity respective to the other member of the group consisting of text content and image content, the identifying operation retrieving the precomputed pairwise similarity measure values from the storage.

17. The multimedia information retrieval system as set forth in claim 16, wherein the monomodal information retrieval process is a text-only information retrieval process and the pairwise similarity measure values are indicative of image content similarity.

18. The multimedia information retrieval system as set forth in claim 16, wherein the monomodal information retrieval process is an image-only information retrieval process and the pairwise similarity measure values are indicative of text content similarity.