System and method for multimedia information retrieval

• US 8,447,767 B2
• Filed: 12/15/2010
• Issued: 05/21/2013
• Est. Priority Date: 12/15/2010
• Status: Expired due to Fees

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Clinchant, Stphane, Csurka, Gabriela, Ah-Pine, Julien
• Primary Examiner(s): Trujillo, James
• Assistant Examiner(s): CONYERS, DAWAUNE A
• Application Number:
  US12/968,796
• Publication Number:
  US 20120158739A1
• Time in Patent Office: 888 Days
• Field of Search: 707/748, 706/54
• US Class Current: 707/748
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 16/432 Query formulation
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition

Abstract

A system and method for information retrieval are disclosed. The method includes querying a multimedia collection with a first component of a multimedia query (e.g., a text-based part of the query) to generate first comparison measures between the first component of the query and respective objects in the collection for a first media type (e.g., text). The multimedia collection is queried with a second component of the multimedia query (e.g., an image-based part of the query) to generate second comparison measures between the second component of the query and respective objects in the collection for a second media type (e.g., visual). An aggregated score for each of a set of objects in the collection is computed, based on the first comparison measure and the second comparison measure for the object. This includes applying an aggregating function to the first and second comparison measures in which a first confidence weighting is applied to the first comparison measure and a second confidence weighting is applied to the second comparison measure. The first confidence weighting is independent of the second comparison measure. The second confidence weighting is dependent on the first comparison measure. Information based on the aggregated scores is output.

Claims

1. A method for multimedia information retrieval comprising:

querying a multimedia collection with a first component of a multimedia query to generate first comparison measures between the first query component and respective objects in the collection for a first media type;

querying at least a part of the multimedia collection with a second component of the multimedia query to generate second comparison measures between the second query component and respective objects in the collection for a second media type;

computing an aggregated score for each of a set of objects in the collection based on the first comparison measure and the second comparison measure for the object, the computing comprising applying an aggregating function to the first and second comparison measures in which a first confidence weighting is applied to the first comparison measure and a second confidence weighting is applied to the second comparison measure, the first confidence weighting is independent of the second comparison measure and the second confidence weighting is dependent on the first comparison measure; and

outputting information based on the aggregated scores,

wherein the aggregated score is computed using an aggregating function of general format;

s_cw(q,o)=A(N(s_a(q,o)ƒ

),N(s_b(q,o)g)) 

 

(3)

where S_cw, is the aggregated score;

A represents the aggregating function;

a represents the first media type;

b represents the second media type;

s_a(q,o) and s_b(q,o) are similarity scores between the query q and the object o for the first and second media types respectively;

r_a(o,q) and r_b(o,q) are rankings of the object o given by the respective similarity scores s_a(q,o) and s_b(q,o), with respect to other objects in the collection;

ƒ

is a function of at least one of r_a(o,q) and s_a(q,o) and optionally also of θ

_a ;

g is a function of at least one of r_a(o,q) and s_a(q,o) and at least one of s_b(q,o) and r_b(o,q) and optionally also of θ

_b;

θ

_a is a set of one or more parameters;

θ

_b is a set of one or more parameters; and

N represents an optional normalizing operator,

wherein at least one of;

2. The method of claim 1, wherein the computing of the aggregated score is performed with a computer processor.

3. The method of claim 1, wherein the first media type comprises text and the first component of the multimedia query is a text-based query component.

4. The method of claim 3, wherein the text-based query component comprises a representation of a text document based on word frequencies.

5. The method of claim 1, wherein the second media type comprises images and the second component of the multimedia query is an image-based query component.

6. The method of claim 5, wherein the image-based query component comprises a representation of an image which is based on extracted features.

7. The method of claim 1, wherein the aggregating function is a linear combination of the function of the confidence weighted first comparison measure and the function of the confidence weighted second comparison measure.

8. The method of claim 1, wherein:

ƒ

is a function of r_a(o,q); and

g is a function of r_a (o,q).

9. The method of claim 1, wherein x is at least 1, y is at least 0, and δ

is greater than 0.

10. The method of claim 1, wherein the aggregating function has the general format:

s_cw(q,o)=α

_aN(s_a(q,o)ƒ

(s_a(q,o),r_a(o,q),θ

_a))+α

_bN(s_b(q,o)g(s_a(q,o),r_a(o,q),s_b(q,o),r_b(o,q),θ

_b) 

 

(4)

where 0<

α

_a<

1 and 0<

α

_b<

1.

11. The method of claim 1, wherein the aggregating function comprises an average mean operator.

12. The method of claim 1, wherein the outputting information comprises outputting objects from the collection based on their aggregated scores.

13. A system for performing the method of claim 1, comprising non-transitory memory which stores instructions for performing the method and a processor in communication with the memory for executing the instructions.

15. A computer program product comprising a non-transitory recording medium encoding instructions, which when executed on a computer causes the computer to perform the method of claim 1.

14. A method for multimedia information retrieval comprising:

querying a multimedia collection with a first component of a multimedia query to generate first comparison measures between the first query component and respective objects in the collection for a first media type;

querying at least a part of the multimedia collection with a second component of the multimedia query to generate second comparison measures between the second query component and respective objects in the collection for a second media type;

computing an aggregated score for each of a set of objects in the collection based on the first comparison measure and the second comparison measure for the object, the computing comprising applying an aggregating function to the first and second comparison measures in which a first confidence weighting is applied to the first comparison measure and a second confidence weighting is applied to the second comparison measure, the first confidence weighting is independent of the second comparison measure and the second confidence weighting is dependent on the first comparison measure; and

outputting information based on the aggregated scores.

wherein the aggregated score is computed using an aggregating function of general format;

s_cw(q,o)=α

_aN(s_a(q,o)ƒ

(s_a(q,o),r_a(o,q),θ

_a))+α

_bN(s_b(q,o)g(s_a(q,o),r_a(o,q),s_b(q,o),r_b(o,q),θ

_b) 

 

(4)

wherein 0<

α

_a<

1, 0<

α

_b<

1, and α

_b=1−

α

_a;

S_cw is the aggregated score;

a represents the first media type;

b represents the second media type;

s_a(q,o) and s_b(q,o) are similarity scores between the query q and the object o for the first and second media types respectively;

r_a(o,q) and r_b(o,q) are rankings of the object o given by the respective similarity scores s_a(q,o) and s_b(q,o), with respect to other objects in the collection;

ƒ

is a function of at least one of r_a(o,q) and s_a(q,o) and optionally also of θ

_a;

g is a function of at least one of r_a(o,q) and s_a(q,o) and at least one of s_b(q,o) and r_b(o,q) and optionally also of θ

_b;

θ

_a is a set of one or more parameters;

θ

_b is a set of one or more parameters; and

N represents an optional normalizing operator.

16. A system for multimedia information retrieval comprising:

a search engine for querying an associated multimedia collection with a first component of a multimedia query and for querying at least a part of the queried multimedia collection with a second component of the multimedia query;

a first comparison component for generating a first comparison measure between the first query component and a respective object in the collection for a first media type;

a second comparison component for generating a second comparison measure between the second query component and a respective object in the collection for the second media type;

a multimedia scoring component for generating aggregated scores for each of a set of objects in the collection based on the first comparison measure and the second comparison measure for the respective object, the multimedia scoring component applying an aggregating function which aggregates the first and second comparison measure in which a first confidence weighting is applied to the first comparison measure which is independent of the second comparison measure and a second confidence weighting is applied to the second comparison measure which is dependent on the first comparison measure,

wherein the aqqreqatinq function is of general format;

s_cw(q,o)=α

_aN(s_a(q,o)ƒ

(s_a(q,o),r_a(o,q),θ

_a))+α

_bN(s_b(q,o)g(s_a(q,o),r_a(o,q),s_b(q,o),r_b(o,q),θ

_b)

wherein 0<

α

_a<

1, 0<

α

_b<

1 and α

_b=1−

α

_a;

S_cw is the aggregated score;

a represents the first media type;

b represents the second media type;

s_a(q,o) and s_b(q,o) are similarity scores between the query q and the object o for the first and second media types respectively;

r_a(o,q) and r_b(o,q) are rankinqs of the object o given by the respective similarity scores s_a(q,o) and s_b(q,o), with respect to other objects in the collection;

ƒ

is a function of at least one of r_a(o,q) and s_a(q,o) and optionally also of θ

_a ;

g is a function of at least one of r_a(o,q) and s_a(q,o) and at least one of s_b(q,o) and r_b(o,q) and optionally also of θ

_b;

θ

_a is a set of one or more parameters;

θ

_b is a set of one or more parameters; and

N represents an optional normalizing operator; and

a processor for implementing the search engine, first and second comparison components, and multimedia scoring component.

17. The system of claim 16, further comprising the collection of multimedia objects.

18. The system of claim 16, wherein the system is configured for outputting retrieved objects from the collection based on their aggregated scores.

19. The system of claim 16, wherein the system is configured for receiving a multimedia query from a user which includes a textual component and an image component.

20. An information retrieval method comprising:

retrieving a set of objects from a collection based on computed text-based scores for the objects;

computing image-based scores for the retrieved objects;

with a processor, computing an aggregated score for objects in the retrieved set of objects based on the respective text-based and image-based scores in which a first confidence weighting is applied to the text-based score which is independent of the image-based score and a second confidence weighting is applied to the image-based score which is dependent on both the text-based score and the image-based score; and

outputting information based on the aggregated scores,

wherein the aggregated score is computed using an aggregating function of general format;

s_cw(q,o)=α

_aN(s_a(q,o)ƒ

(s_a(q,o),r_a(o,q),θ

_a))+α

_bN(s_b(q,o)g(s_a(q,o),r_a(o,q),s_b(q,o),r_b(o,q),θ

_b)

wherein 0<

α

_a<

1, 0<

α

_b<

1, and α

_b=1−

α

_a;

S_cw is the aggregated score;

a represents a first media type, which comprises text;

b represents a second media type, which comprises images;

s_a(q,o) and s_b(q,o) are similarity scores between the query q and the object o for the first and second media types respectively;

r_a(o,q) and r_b(o,q) are rankings of the object o given by the respective similarity scores s_a(q,o) and s_b(q,o), with respect to other objects in the collection;

ƒ

is a function of at least one of r_a(o,q) and s_a(q,o) and optionally also of θ

_a ;

g is a function of at least one of r_a(o,q) and s_a(q,o) and at least one of S_{b(q,o) and rb}(o,q) and optionally also of θ

_b;

θ

_a is a set of one or more parameters;

θ

_b is a set of one or more parameters; and

N represents an optional normalizing operator.